From patchwork Mon Aug 9 21:23:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12427289 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59482C43214 for ; Mon, 9 Aug 2021 21:24:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3ED9661040 for ; Mon, 9 Aug 2021 21:24:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234208AbhHIVYa (ORCPT ); Mon, 9 Aug 2021 17:24:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55886 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230295AbhHIVY3 (ORCPT ); Mon, 9 Aug 2021 17:24:29 -0400 Received: from mail-pj1-x1036.google.com (mail-pj1-x1036.google.com [IPv6:2607:f8b0:4864:20::1036]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F226BC0613D3 for ; Mon, 9 Aug 2021 14:24:08 -0700 (PDT) Received: by mail-pj1-x1036.google.com with SMTP id t7-20020a17090a5d87b029017807007f23so1017760pji.5 for ; Mon, 09 Aug 2021 14:24:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=kwxhg+k9Vp47rFzfroR6CXyf8w3Mx34MsOmSxP+U8Vw=; b=o0BxthJJQLcYeaGAWebI8muOYZ65I+IpoOYzcZpo/MWRER4HE8/qhqZUUfAMYmgHV1 bp2tM82xSlFDLJ2gd2MIX+mZroUQ8T8jPNqOCPycXFqGIvs4SeWHyZd0EaUfWaGHj4iW qhFwjiSNMQU31SvnTkBMu5cHgBrbPNkL/jemya27jc6ng5kPBYmkee6GHY05sBuoQJlt uNtsnde6QvEs8JhpaM0p2UrKbt5q1lkBlRsR+ah39UmHjWViZhRmPWRpMSQEJrNsAKiX RM05khVZT2hmX63vQ4Z72utyySoCzR9j/hAC2MIL4bT8xBpHlJrhCQ3rufhwDr7MzpVo f1GA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kwxhg+k9Vp47rFzfroR6CXyf8w3Mx34MsOmSxP+U8Vw=; b=DOZO9NZm0gj2tWGdQFJtfkozuiHtd6wIArigFsKKWjqjkzo5z+xi9p4AKVtBdB9bSL FyX1C9DBaaQHt15g+SRDAYg7+930Lr1iND7VQdcSfIm/tcwIaHVMNW2ulB2P3gasKtkl mhj6hz/cE5isoeYcl66HggJOh2MKuu09UX/jINNQKl05sit2mtJ+g4Que8dpAgrNL7h4 Nz/1z4rbT/r19ta3ioE4PTSLQdCpZ+g/kSAxPhU5R6dpoyQqTbrTadTWUiZhzVH6QNPl kgxRAjqlb61ooBf+OTGwCwc5/9U4C2qPqZPXzE1ov73uCdmSCH+iK0ltFZ6T10sqRQmn PO8w== X-Gm-Message-State: AOAM532SfV1Oeb5MypOECqb/BCDxCkQbPG9mjioevxGPakyl7weG5dON 1jax3roBbSEEdHYqlLKAhmcSQcvoYBaEupKZ X-Google-Smtp-Source: ABdhPJyo5i6FEVjJLVnBQFVIuF8/3kg3pz2/BPNdtQySqRhvt007jmUZi3VbUEnhJ+Uq74mA3gcIXA== X-Received: by 2002:a17:902:dcd5:b029:12d:219f:6c04 with SMTP id t21-20020a170902dcd5b029012d219f6c04mr7028661pll.7.1628544248475; Mon, 09 Aug 2021 14:24:08 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.61]) by smtp.gmail.com with ESMTPSA id m16sm439885pjz.30.2021.08.09.14.24.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Aug 2021 14:24:08 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe Subject: [PATCH 1/4] bio: add allocation cache abstraction Date: Mon, 9 Aug 2021 15:23:58 -0600 Message-Id: <20210809212401.19807-2-axboe@kernel.dk> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210809212401.19807-1-axboe@kernel.dk> References: <20210809212401.19807-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Add a set of helpers that can encapsulate bio allocations, reusing them as needed. Caller must provide the necessary locking, if any is needed. The primary intended use case is polled IO from io_uring, which will not need any external locking. Very simple - keeps a count of bio's in the cache, and maintains a max of 512 with a slack of 64. If we get above max + slack, we drop slack number of bio's. The cache is intended to be per-task, and the user will need to supply the storage for it. As io_uring will be the only user right now, provide a hook that returns the cache there. Stub it out as NULL initially. Signed-off-by: Jens Axboe --- block/bio.c | 126 +++++++++++++++++++++++++++++++++++---- include/linux/bio.h | 24 ++++++-- include/linux/io_uring.h | 7 +++ 3 files changed, 141 insertions(+), 16 deletions(-) diff --git a/block/bio.c b/block/bio.c index 1fab762e079b..3bbda1be27be 100644 --- a/block/bio.c +++ b/block/bio.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include "blk.h" @@ -238,6 +239,35 @@ static void bio_free(struct bio *bio) } } +static inline void __bio_init(struct bio *bio) +{ + bio->bi_next = NULL; + bio->bi_bdev = NULL; + bio->bi_opf = 0; + bio->bi_flags = bio->bi_ioprio = bio->bi_write_hint = 0; + bio->bi_status = 0; + bio->bi_iter.bi_sector = 0; + bio->bi_iter.bi_size = 0; + bio->bi_iter.bi_idx = 0; + bio->bi_iter.bi_bvec_done = 0; + bio->bi_end_io = NULL; + bio->bi_private = NULL; +#ifdef CONFIG_BLK_CGROUP + bio->bi_blkg = NULL; + bio->bi_issue.value = 0; +#ifdef CONFIG_BLK_CGROUP_IOCOST + bio->bi_iocost_cost = 0; +#endif +#endif +#ifdef CONFIG_BLK_INLINE_ENCRYPTION + bio->bi_crypt_context = NULL; +#endif +#ifdef CONFIG_BLK_DEV_INTEGRITY + bio->bi_integrity = NULL; +#endif + bio->bi_vcnt = 0; +} + /* * Users of this function have their own bio allocation. Subsequently, * they must remember to pair any call to bio_init() with bio_uninit() @@ -246,7 +276,7 @@ static void bio_free(struct bio *bio) void bio_init(struct bio *bio, struct bio_vec *table, unsigned short max_vecs) { - memset(bio, 0, sizeof(*bio)); + __bio_init(bio); atomic_set(&bio->__bi_remaining, 1); atomic_set(&bio->__bi_cnt, 1); @@ -591,6 +621,19 @@ void guard_bio_eod(struct bio *bio) bio_truncate(bio, maxsector << 9); } +static bool __bio_put(struct bio *bio) +{ + if (!bio_flagged(bio, BIO_REFFED)) + return true; + + BIO_BUG_ON(!atomic_read(&bio->__bi_cnt)); + + /* + * last put frees it + */ + return atomic_dec_and_test(&bio->__bi_cnt); +} + /** * bio_put - release a reference to a bio * @bio: bio to release reference to @@ -601,17 +644,8 @@ void guard_bio_eod(struct bio *bio) **/ void bio_put(struct bio *bio) { - if (!bio_flagged(bio, BIO_REFFED)) + if (__bio_put(bio)) bio_free(bio); - else { - BIO_BUG_ON(!atomic_read(&bio->__bi_cnt)); - - /* - * last put frees it - */ - if (atomic_dec_and_test(&bio->__bi_cnt)) - bio_free(bio); - } } EXPORT_SYMBOL(bio_put); @@ -1595,6 +1629,76 @@ int bioset_init_from_src(struct bio_set *bs, struct bio_set *src) } EXPORT_SYMBOL(bioset_init_from_src); +void bio_alloc_cache_init(struct bio_alloc_cache *cache) +{ + bio_list_init(&cache->free_list); + cache->nr = 0; +} + +static void bio_alloc_cache_prune(struct bio_alloc_cache *cache, + unsigned int nr) +{ + struct bio *bio; + unsigned int i; + + i = 0; + while ((bio = bio_list_pop(&cache->free_list)) != NULL) { + cache->nr--; + bio_free(bio); + if (++i == nr) + break; + } +} + +void bio_alloc_cache_destroy(struct bio_alloc_cache *cache) +{ + bio_alloc_cache_prune(cache, -1U); +} + +struct bio *bio_cache_get(gfp_t gfp, unsigned short nr_vecs, struct bio_set *bs) +{ + struct bio_alloc_cache *cache = io_uring_bio_cache(); + struct bio *bio; + + if (!cache || nr_vecs > BIO_INLINE_VECS) + return NULL; + if (bio_list_empty(&cache->free_list)) { +alloc: + if (bs) + return bio_alloc_bioset(gfp, nr_vecs, bs); + else + return bio_alloc(gfp, nr_vecs); + } + + bio = bio_list_peek(&cache->free_list); + if (bs && bio->bi_pool != bs) + goto alloc; + bio_list_del_head(&cache->free_list, bio); + cache->nr--; + bio_init(bio, nr_vecs ? bio->bi_inline_vecs : NULL, nr_vecs); + return bio; +} + +#define ALLOC_CACHE_MAX 512 +#define ALLOC_CACHE_SLACK 64 + +void bio_cache_put(struct bio *bio) +{ + struct bio_alloc_cache *cache = io_uring_bio_cache(); + + if (unlikely(!__bio_put(bio))) + return; + if (cache) { + bio_uninit(bio); + bio_list_add_head(&cache->free_list, bio); + cache->nr++; + if (cache->nr > ALLOC_CACHE_MAX + ALLOC_CACHE_SLACK) + bio_alloc_cache_prune(cache, ALLOC_CACHE_SLACK); + } else { + bio_free(bio); + } +} + static int __init init_bio(void) { int i; diff --git a/include/linux/bio.h b/include/linux/bio.h index 2203b686e1f0..b70c72365fa2 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -652,18 +652,22 @@ static inline struct bio *bio_list_peek(struct bio_list *bl) return bl->head; } -static inline struct bio *bio_list_pop(struct bio_list *bl) +static inline void bio_list_del_head(struct bio_list *bl, struct bio *head) { - struct bio *bio = bl->head; - - if (bio) { + if (head) { bl->head = bl->head->bi_next; if (!bl->head) bl->tail = NULL; - bio->bi_next = NULL; + head->bi_next = NULL; } +} +static inline struct bio *bio_list_pop(struct bio_list *bl) +{ + struct bio *bio = bl->head; + + bio_list_del_head(bl, bio); return bio; } @@ -676,6 +680,16 @@ static inline struct bio *bio_list_get(struct bio_list *bl) return bio; } +struct bio_alloc_cache { + struct bio_list free_list; + unsigned int nr; +}; + +void bio_alloc_cache_init(struct bio_alloc_cache *); +void bio_alloc_cache_destroy(struct bio_alloc_cache *); +struct bio *bio_cache_get(gfp_t, unsigned short, struct bio_set *bs); +void bio_cache_put(struct bio *); + /* * Increment chain count for the bio. Make sure the CHAIN flag update * is visible before the raised count. diff --git a/include/linux/io_uring.h b/include/linux/io_uring.h index 04b650bcbbe5..2fb53047638e 100644 --- a/include/linux/io_uring.h +++ b/include/linux/io_uring.h @@ -5,6 +5,8 @@ #include #include +struct bio_alloc_cache; + #if defined(CONFIG_IO_URING) struct sock *io_uring_get_socket(struct file *file); void __io_uring_cancel(struct files_struct *files); @@ -40,4 +42,9 @@ static inline void io_uring_free(struct task_struct *tsk) } #endif +static inline struct bio_alloc_cache *io_uring_bio_cache(void) +{ + return NULL; +} + #endif From patchwork Mon Aug 9 21:23:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12427287 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30460C4320E for ; Mon, 9 Aug 2021 21:24:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 11F2E60EE7 for ; Mon, 9 Aug 2021 21:24:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234504AbhHIVYb (ORCPT ); Mon, 9 Aug 2021 17:24:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55894 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230295AbhHIVYa (ORCPT ); Mon, 9 Aug 2021 17:24:30 -0400 Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E6BB5C0613D3 for ; Mon, 9 Aug 2021 14:24:09 -0700 (PDT) Received: by mail-pj1-x102e.google.com with SMTP id oa17so5151760pjb.1 for ; Mon, 09 Aug 2021 14:24:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=PP1ZfPKJqEpeBtzueEvBPGx+FIq9NBPbqy/H3X4n8yQ=; b=QntNk8vasegeRQEgvpTa/22T3k76x4N6NKutp80kBdgcUVkChC0EIARzOBzwW2+pTF RWHOkH2JqcD39EZC7on8rjGqpdPX4n7U96tyDrhHJNdbW/UtbwTKsiphXId+RsSCBq2M V9pjzdJfNYNUpHDrY/I8j2i+vgeukTmTtd2V0MELzz5NmkFOzktEP2fvX3drNNTsgcGM MEhMgreQtzoezgOnVZLzJSKqeSn8Q7krYOUC+nkMCosaLumz3slpkhUjvQpRP9Q5Xk5F n9L6n/M8KArazCvF28frDqVv6u5OAIAKNul8Xe7zT2MxrsTXBGjDdxQvJBJ4wNdi9IPm 1Psg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=PP1ZfPKJqEpeBtzueEvBPGx+FIq9NBPbqy/H3X4n8yQ=; b=TB3QzHOHTE0onWsODM9uce+ssRfI+5DXrHkPzBlXDsc10HFFZVyjTz8BCoCvJqRUIZ 3QB9HSM3MKDQ5GBCo7Tcc8d1zpQt7lQzQsOuNrjyXd7a9MtEjgCQbAmDmLF27sJJhNWF DOjzcJVUhu3wMdpRPa+h5GK0CIIRH7l0Pc/zgqPUzmzTDHbJ0bE2afjLMZRqP5cg/vUw qNK9IVy1st61MO4hYF1yenyWoqjyGyXWAXzvPWwRhjD4E+ZxeZ6R/R4IuvMSaOgKsDnx uMtyyV7rOnHZvxb+Alt3wz17HVAIi0iT2aYguyFFZbk5FMHdVlZ2js1KJ6v1OrH/Z7oo 1mkg== X-Gm-Message-State: AOAM533q77dbgDuCJWYRMYWZaf9L9xk1AVWb2ZitD4mfKTVN2N8/jzxT b7hk9n3bEZoNxtX99HibXU7NoQ== X-Google-Smtp-Source: ABdhPJz2rpmsQOTo4zm6VYOkCVmmpOo82fVRnfQFtN14Gdp8qotDxvLXa/X8xd9Ldy6LwtBIipfg7A== X-Received: by 2002:a17:90a:6a86:: with SMTP id u6mr1099946pjj.207.1628544249468; Mon, 09 Aug 2021 14:24:09 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.61]) by smtp.gmail.com with ESMTPSA id m16sm439885pjz.30.2021.08.09.14.24.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Aug 2021 14:24:09 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe Subject: [PATCH 2/4] fs: add bio alloc cache kiocb flag Date: Mon, 9 Aug 2021 15:23:59 -0600 Message-Id: <20210809212401.19807-3-axboe@kernel.dk> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210809212401.19807-1-axboe@kernel.dk> References: <20210809212401.19807-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org We'll be using this to implement a recycling cache for the bio units used to do IO. Signed-off-by: Jens Axboe --- include/linux/fs.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/include/linux/fs.h b/include/linux/fs.h index 640574294216..2ac1b01a4902 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -319,6 +319,8 @@ enum rw_hint { /* iocb->ki_waitq is valid */ #define IOCB_WAITQ (1 << 19) #define IOCB_NOIO (1 << 20) +/* bio cache can be used */ +#define IOCB_ALLOC_CACHE (1 << 21) struct kiocb { struct file *ki_filp; From patchwork Mon Aug 9 21:24:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12427291 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9631AC4338F for ; Mon, 9 Aug 2021 21:24:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7493C60EE7 for ; Mon, 9 Aug 2021 21:24:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234650AbhHIVYc (ORCPT ); Mon, 9 Aug 2021 17:24:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55902 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234177AbhHIVYb (ORCPT ); Mon, 9 Aug 2021 17:24:31 -0400 Received: from mail-pl1-x633.google.com (mail-pl1-x633.google.com [IPv6:2607:f8b0:4864:20::633]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0860FC0613D3 for ; Mon, 9 Aug 2021 14:24:11 -0700 (PDT) Received: by mail-pl1-x633.google.com with SMTP id e19so4527108pla.10 for ; Mon, 09 Aug 2021 14:24:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5/20ck0XnHnMDNZglLbT8+iG6Jg9pQpAk9fG8nwb5/0=; b=viS7Rgv3M63OffnJNQ8/5rkvdk/yaJm0ueeRmIWPwuzMQe6YOB+AS3Vj3FjyK44r/U 8mZnoMhh2r525e3QcDuKc55s7s07tPBdJfSGa1Yo1bjm5KWeFyySaOpFbzy14aYmtvt9 fDBiNQ+lD46SuImFU02rP3/CSSdMcu3qW+YSaWc8+g7AHgQtF4W3GVS4tdrgbQbQDrT/ NRa9WYYMPBcxe0tv5a0UldJeai9rOLiv6Tkuw3aoxWwqbwtkiYfXei6l1Fd+ungckhLS HU9bqr9UWqXobMMggmiQBwP6wohRB+78c2jcD2bUGxFuCXiQg4wL7+ZaA1mGpRp6uq+s IIjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5/20ck0XnHnMDNZglLbT8+iG6Jg9pQpAk9fG8nwb5/0=; b=T1TZmXjNpdzt6yux1Wz1CpnEgTSnb7/pXyebB0WX4hVF24iz/M2Dsn3eixhg6qsYSY 1ZsiNbaMEcA992nJCxQXND8/tkWPH1hjbwoVvJ0qmUrV9aQCnGe3z7a33s3XARS82n6e 6YeUdnQZKf1i/JOOQ9NC7ax8b/lhSAGqE3wtp4LXdpSMimVFxBG8XKZf1DHyv9VkILsS KicL79PfAgtfzT2IMchHxyNXJXhmcE3ir3iXa010i3866QhUr5iJ0VS2A0p84tNaRT2W iAhRLe3/pMt8/po4GxGEzNwgPG68+uSJ6/YJmQs9Hcy2jaCo/oJNsFC7DrbKqZlX1UR7 T78w== X-Gm-Message-State: AOAM532WNstDYuKXHU2av1VIGUUrI8+yji0VxrLx0p5gms//vfh5RI/V cLj5Z4xsMdmzdvExW+24CRLOwhVZD2naBlia X-Google-Smtp-Source: ABdhPJzbsh4KB8JIQ2bb5KBHUu2ulbvi6d22PFcHzZ1CKqSPxpxcowwM8m/XFMVhDV3N10PnSFoz8w== X-Received: by 2002:a17:90b:250f:: with SMTP id ns15mr1140198pjb.26.1628544250587; Mon, 09 Aug 2021 14:24:10 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.61]) by smtp.gmail.com with ESMTPSA id m16sm439885pjz.30.2021.08.09.14.24.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Aug 2021 14:24:10 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe Subject: [PATCH 3/4] io_uring: wire up bio allocation cache Date: Mon, 9 Aug 2021 15:24:00 -0600 Message-Id: <20210809212401.19807-4-axboe@kernel.dk> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210809212401.19807-1-axboe@kernel.dk> References: <20210809212401.19807-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Initialize a bio allocation cache, and mark it as being used for IOPOLL. We could use it for non-polled IO as well, but it'd need some locking and probably would negate much of the win in that case. We start with IOPOLL, as completions are locked by the ctx lock anyway. So no further locking is needed there. This brings an IOPOLL gen2 Optane QD=128 workload from ~3.0M IOPS to ~3.25M IOPS. Signed-off-by: Jens Axboe --- fs/io_uring.c | 52 ++++++++++++++++++++++++++++++++++++++++ include/linux/io_uring.h | 4 ++-- 2 files changed, 54 insertions(+), 2 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index 91a301bb1644..1d94a434b348 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -474,6 +474,10 @@ struct io_uring_task { atomic_t inflight_tracked; atomic_t in_idle; +#ifdef CONFIG_BLOCK + struct bio_alloc_cache bio_cache; +#endif + spinlock_t task_lock; struct io_wq_work_list task_list; unsigned long task_state; @@ -2268,6 +2272,8 @@ static void io_iopoll_complete(struct io_ring_ctx *ctx, unsigned int *nr_events, if (READ_ONCE(req->result) == -EAGAIN && resubmit && !(req->flags & REQ_F_DONT_REISSUE)) { req->iopoll_completed = 0; + /* Don't use cache for async retry, not locking safe */ + req->rw.kiocb.ki_flags &= ~IOCB_ALLOC_CACHE; req_ref_get(req); io_req_task_queue_reissue(req); continue; @@ -2675,6 +2681,29 @@ static bool io_file_supports_nowait(struct io_kiocb *req, int rw) return __io_file_supports_nowait(req->file, rw); } +static void io_mark_alloc_cache(struct kiocb *kiocb) +{ +#ifdef CONFIG_BLOCK + struct block_device *bdev = NULL; + + if (S_ISBLK(file_inode(kiocb->ki_filp)->i_mode)) + bdev = I_BDEV(kiocb->ki_filp->f_mapping->host); + else if (S_ISREG(file_inode(kiocb->ki_filp)->i_mode)) + bdev = kiocb->ki_filp->f_inode->i_sb->s_bdev; + + /* + * If the lower level device doesn't support polled IO, then + * we cannot safely use the alloc cache. This really should + * be a failure case for polled IO... + */ + if (!bdev || + !test_bit(QUEUE_FLAG_POLL, &bdev_get_queue(bdev)->queue_flags)) + return; + + kiocb->ki_flags |= IOCB_ALLOC_CACHE; +#endif /* CONFIG_BLOCK */ +} + static int io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe) { struct io_ring_ctx *ctx = req->ctx; @@ -2717,6 +2746,7 @@ static int io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe) return -EOPNOTSUPP; kiocb->ki_flags |= IOCB_HIPRI; + io_mark_alloc_cache(kiocb); kiocb->ki_complete = io_complete_rw_iopoll; req->iopoll_completed = 0; } else { @@ -2783,6 +2813,8 @@ static void kiocb_done(struct kiocb *kiocb, ssize_t ret, if (check_reissue && (req->flags & REQ_F_REISSUE)) { req->flags &= ~REQ_F_REISSUE; if (io_resubmit_prep(req)) { + /* Don't use cache for async retry, not locking safe */ + req->rw.kiocb.ki_flags &= ~IOCB_ALLOC_CACHE; req_ref_get(req); io_req_task_queue_reissue(req); } else { @@ -7966,10 +7998,17 @@ static int io_uring_alloc_task_context(struct task_struct *task, return ret; } +#ifdef CONFIG_BLOCK + bio_alloc_cache_init(&tctx->bio_cache); +#endif + tctx->io_wq = io_init_wq_offload(ctx, task); if (IS_ERR(tctx->io_wq)) { ret = PTR_ERR(tctx->io_wq); percpu_counter_destroy(&tctx->inflight); +#ifdef CONFIG_BLOCK + bio_alloc_cache_destroy(&tctx->bio_cache); +#endif kfree(tctx); return ret; } @@ -7993,6 +8032,10 @@ void __io_uring_free(struct task_struct *tsk) WARN_ON_ONCE(tctx->io_wq); WARN_ON_ONCE(tctx->cached_refs); +#ifdef CONFIG_BLOCK + bio_alloc_cache_destroy(&tctx->bio_cache); +#endif + percpu_counter_destroy(&tctx->inflight); kfree(tctx); tsk->io_uring = NULL; @@ -10247,6 +10290,15 @@ SYSCALL_DEFINE4(io_uring_register, unsigned int, fd, unsigned int, opcode, return ret; } +struct bio_alloc_cache *io_uring_bio_cache(void) +{ +#ifdef CONFIG_BLOCK + if (current->io_uring) + return ¤t->io_uring->bio_cache; +#endif + return NULL; +} + static int __init io_uring_init(void) { #define __BUILD_BUG_VERIFY_ELEMENT(stype, eoffset, etype, ename) do { \ diff --git a/include/linux/io_uring.h b/include/linux/io_uring.h index 2fb53047638e..a9bab9bd51d1 100644 --- a/include/linux/io_uring.h +++ b/include/linux/io_uring.h @@ -11,6 +11,7 @@ struct bio_alloc_cache; struct sock *io_uring_get_socket(struct file *file); void __io_uring_cancel(struct files_struct *files); void __io_uring_free(struct task_struct *tsk); +struct bio_alloc_cache *io_uring_bio_cache(void); static inline void io_uring_files_cancel(struct files_struct *files) { @@ -40,11 +41,10 @@ static inline void io_uring_files_cancel(struct files_struct *files) static inline void io_uring_free(struct task_struct *tsk) { } -#endif - static inline struct bio_alloc_cache *io_uring_bio_cache(void) { return NULL; } +#endif #endif From patchwork Mon Aug 9 21:24:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12427293 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 304F2C43214 for ; Mon, 9 Aug 2021 21:24:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1589160EB9 for ; Mon, 9 Aug 2021 21:24:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234687AbhHIVYf (ORCPT ); Mon, 9 Aug 2021 17:24:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55922 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236419AbhHIVYf (ORCPT ); Mon, 9 Aug 2021 17:24:35 -0400 Received: from mail-pl1-x62d.google.com (mail-pl1-x62d.google.com [IPv6:2607:f8b0:4864:20::62d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 58DFEC0613D3 for ; Mon, 9 Aug 2021 14:24:12 -0700 (PDT) Received: by mail-pl1-x62d.google.com with SMTP id a20so18107539plm.0 for ; Mon, 09 Aug 2021 14:24:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=dyOLwIAyn5igot9fdJEMYSDMYL7Dq8yEvViQYaBuv5Q=; b=IgN38Ju83GTulvCvZYlk1tCmOLuq05ZnrXLyiM9cJpOx72VXo8vLggCiEjPzdjwgyD mMK7Lct7hKsJ/9DeBUZkREk9FHfFpCUr77ggNxRruqvvZ4qbGG6Bt5NbSI/8AvjN+lsD 7DlNNRZMiSIZboSKJdB46Lk6VuIvJ8D/yY1x6ajWlTDv1lyKY0e52sAZDh9GjmZn5d9X S2zhHWdkcPKHSIdk8OT/QtCUFD4xvU3VyyZhgnonWJv2jzZEX6KXZYTft+GnfFxn4mCi 2hIadd1fFXCXYRljE3GILzNYvjMtdV6g6PhSoJG9kDpUkJFogd08mLnucB5oubB0dmNE nG+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=dyOLwIAyn5igot9fdJEMYSDMYL7Dq8yEvViQYaBuv5Q=; b=cuujV7fHHWYPf7Y5PZguUoBiIKdOromIpZhj3+fs3NNpTnVOtf4rgdfqyIyRMsLewR cbL75NkjhALeXweREfX9Ui9F2eQWL4571dJprofAT2GOvVOZGpKB5j03cwo6sgoNQ7fE lCRK8NC9nQuuA0KR0jdQXCxMO6zNARLd49CWJE15JOL2Pu7qF17zFSxKV7nx3OsbAVlJ E3vKqWct/cTInBD8yVJy0AIRosag32VihFV2QE/l+8w7gpwhz1Tw2c/6ILCuoRb7BDsL ssOIfh6t9ybUYu23vl5883MentkwkKY5NhqHzTWTudPSl5NAUw3KhsMl7S5NXDAGoXND uAbw== X-Gm-Message-State: AOAM531tXSKj7Zm+vjF0xnK2xAgPStnC5F1O111q3dyHAaFfEeGkFZj0 +TFUT1TcCRJQDFyoOj145jf+pzqDIuCsJ4dp X-Google-Smtp-Source: ABdhPJwsQKrvh7BBLMHgl6EZA+imD4LOvABq2HSlJV/q66inXi4kN9VHm6P7KHtJsfNePIQmLfyH3A== X-Received: by 2002:a17:90a:6541:: with SMTP id f1mr1114982pjs.184.1628544251710; Mon, 09 Aug 2021 14:24:11 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.61]) by smtp.gmail.com with ESMTPSA id m16sm439885pjz.30.2021.08.09.14.24.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Aug 2021 14:24:11 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe Subject: [PATCH 4/4] block: enable use of bio allocation cache Date: Mon, 9 Aug 2021 15:24:01 -0600 Message-Id: <20210809212401.19807-5-axboe@kernel.dk> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210809212401.19807-1-axboe@kernel.dk> References: <20210809212401.19807-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org If a kiocb is marked as being valid for bio caching, then use that to allocate a (and free) new bio if possible. Signed-off-by: Jens Axboe --- fs/block_dev.c | 30 ++++++++++++++++++++++++++---- 1 file changed, 26 insertions(+), 4 deletions(-) diff --git a/fs/block_dev.c b/fs/block_dev.c index 9ef4f1fc2cb0..36a3d53326c0 100644 --- a/fs/block_dev.c +++ b/fs/block_dev.c @@ -327,6 +327,14 @@ static int blkdev_iopoll(struct kiocb *kiocb, bool wait) return blk_poll(q, READ_ONCE(kiocb->ki_cookie), wait); } +static void dio_bio_put(struct blkdev_dio *dio) +{ + if (!dio->is_sync && (dio->iocb->ki_flags & IOCB_ALLOC_CACHE)) + bio_cache_put(&dio->bio); + else + bio_put(&dio->bio); +} + static void blkdev_bio_end_io(struct bio *bio) { struct blkdev_dio *dio = bio->bi_private; @@ -362,7 +370,7 @@ static void blkdev_bio_end_io(struct bio *bio) bio_check_pages_dirty(bio); } else { bio_release_pages(bio, false); - bio_put(bio); + dio_bio_put(dio); } } @@ -385,7 +393,14 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, (bdev_logical_block_size(bdev) - 1)) return -EINVAL; - bio = bio_alloc_bioset(GFP_KERNEL, nr_pages, &blkdev_dio_pool); + bio = NULL; + if (iocb->ki_flags & IOCB_ALLOC_CACHE) { + bio = bio_cache_get(GFP_KERNEL, nr_pages, &blkdev_dio_pool); + if (!bio) + iocb->ki_flags &= ~IOCB_ALLOC_CACHE; + } + if (!bio) + bio = bio_alloc_bioset(GFP_KERNEL, nr_pages, &blkdev_dio_pool); dio = container_of(bio, struct blkdev_dio, bio); dio->is_sync = is_sync = is_sync_kiocb(iocb); @@ -467,7 +482,14 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, } submit_bio(bio); - bio = bio_alloc(GFP_KERNEL, nr_pages); + bio = NULL; + if (iocb->ki_flags & IOCB_ALLOC_CACHE) { + bio = bio_cache_get(GFP_KERNEL, nr_pages, &fs_bio_set); + if (!bio) + iocb->ki_flags &= ~IOCB_ALLOC_CACHE; + } + if (!bio) + bio = bio_alloc(GFP_KERNEL, nr_pages); } if (!is_poll) @@ -492,7 +514,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, if (likely(!ret)) ret = dio->size; - bio_put(&dio->bio); + dio_bio_put(dio); return ret; }