From patchwork Sat Oct 9 12:25:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12547641 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A899C433EF for ; Sat, 9 Oct 2021 12:26:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3689960F6F for ; Sat, 9 Oct 2021 12:26:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233117AbhJIM23 (ORCPT ); Sat, 9 Oct 2021 08:28:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60314 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233152AbhJIM22 (ORCPT ); Sat, 9 Oct 2021 08:28:28 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B6207C061570; Sat, 9 Oct 2021 05:26:31 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id e3so4661283wrc.11; Sat, 09 Oct 2021 05:26:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=EP4Uspv4eyXqaJfsMkfjA4mFgZjpvK/F6u+Tfp4MRZM=; b=GfoSajW3KfmbqZ9hbaFuYwnH5kz1L5BBebRy2dqjnFJrbu0uxb5AHSzIIFlLYpVLAb 7LqxIkvJdbQ2wnNTuSS0W2tZFd2Q/aBPSWwfOy+elCK1fCNlZsKGE7cIazu4nLaTYwZV KUPCdAIeKp9jnmAPYJiwMEjllQq26511C0EoJjhYDwb+/UwrgSlMaDuUeyqJ7PLzRmJe C8YRlbpErBegg4rEbCqILoM0bkt7NnfL1v355NOasFGsyUE24aw5/zCxZl053w0GrtIM AvcVnh7ZFfrhqt5gyjv4Be2vVZXNqHpJzg7shDyKcBjpYBxyfJm7XU7mcQ2hINBBNqKG rbUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=EP4Uspv4eyXqaJfsMkfjA4mFgZjpvK/F6u+Tfp4MRZM=; b=368f43f7r29Zz2X5tVYn8G8Ew0UJxskkPyDalkFJZWHEcCivB1zJJrnvjBc/hTVOOh jYuMhMDVTmSTUFC9KA5S5JIoKhOGu0XdNMExCCWhcPck3U+BdKoe/FJFipo/TbYBoVH1 s6iquI2B1fl49NbHp2V4wc2/sqtFZqGgNn2+ykSt86NqHPYhmZ6we/V/knNIqdMSTr6f TnhnoQQaMCmGROhgJfYYVZYqWQpfqwCrcPz06O+eeUxYayY84UvxqVnG3N6mbG98dpxy aXr5NRR3z5RuhhO3VqAMd7fbf+U320UkkTRi5dsRIVR3HgXn7oH/Qi4A6H5Auy8fkoGO epzg== X-Gm-Message-State: AOAM532yBoD9EKVh4IM6mRY3hATLjxREtRwyyE/eWaKGzx6BbjkqcFSc Qkupevf0hByYmh4rc/LeN34= X-Google-Smtp-Source: ABdhPJxoa7HmQfNXeCmsXazUJW91f3ij8RPe/t1G7c+AsEU1OrWe6wGeWwPxJ21Y7JkmyALUkluzIw== X-Received: by 2002:adf:fb50:: with SMTP id c16mr11226446wrs.120.1633782390335; Sat, 09 Oct 2021 05:26:30 -0700 (PDT) Received: from localhost.localdomain ([85.255.236.155]) by smtp.gmail.com with ESMTPSA id o12sm2157468wrv.78.2021.10.09.05.26.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 09 Oct 2021 05:26:30 -0700 (PDT) From: Pavel Begunkov To: Jens Axboe , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH 1/6] block: cache bdev in struct file for raw bdev IO Date: Sat, 9 Oct 2021 13:25:38 +0100 Message-Id: X-Mailer: git-send-email 2.33.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org bdev = &BDEV_I(file->f_mapping->host)->bdev Getting struct block_device from a file requires 2 memory dereferences as illustrated above, that takes a toll on performance, so cache it in yet unused file->private_data. That gives a noticeable peak performance improvement. Signed-off-by: Pavel Begunkov --- block/fops.c | 36 +++++++++++++++++++----------------- 1 file changed, 19 insertions(+), 17 deletions(-) diff --git a/block/fops.c b/block/fops.c index 765086d51f8b..99e699427f31 100644 --- a/block/fops.c +++ b/block/fops.c @@ -17,11 +17,16 @@ #include #include "blk.h" -static struct inode *bdev_file_inode(struct file *file) +static inline struct inode *bdev_file_inode(struct file *file) { return file->f_mapping->host; } +static inline struct block_device *blkdev_get_bdev(struct file *file) +{ + return file->private_data; +} + static int blkdev_get_block(struct inode *inode, sector_t iblock, struct buffer_head *bh, int create) { @@ -54,8 +59,7 @@ static void blkdev_bio_end_io_simple(struct bio *bio) static ssize_t __blkdev_direct_IO_simple(struct kiocb *iocb, struct iov_iter *iter, unsigned int nr_pages) { - struct file *file = iocb->ki_filp; - struct block_device *bdev = I_BDEV(bdev_file_inode(file)); + struct block_device *bdev = blkdev_get_bdev(iocb->ki_filp); struct bio_vec inline_vecs[DIO_INLINE_BIO_VECS], *vecs; loff_t pos = iocb->ki_pos; bool should_dirty = false; @@ -143,7 +147,7 @@ static struct bio_set blkdev_dio_pool; static int blkdev_iopoll(struct kiocb *kiocb, struct io_batch *ib, bool wait) { - struct block_device *bdev = I_BDEV(kiocb->ki_filp->f_mapping->host); + struct block_device *bdev = blkdev_get_bdev(kiocb->ki_filp); struct request_queue *q = bdev_get_queue(bdev); return blk_poll(q, READ_ONCE(kiocb->ki_cookie), ib, wait); @@ -191,9 +195,7 @@ static void blkdev_bio_end_io(struct bio *bio) static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, unsigned int nr_pages) { - struct file *file = iocb->ki_filp; - struct inode *inode = bdev_file_inode(file); - struct block_device *bdev = I_BDEV(inode); + struct block_device *bdev = blkdev_get_bdev(iocb->ki_filp); struct blk_plug plug; struct blkdev_dio *dio; struct bio *bio; @@ -405,8 +407,7 @@ static loff_t blkdev_llseek(struct file *file, loff_t offset, int whence) static int blkdev_fsync(struct file *filp, loff_t start, loff_t end, int datasync) { - struct inode *bd_inode = bdev_file_inode(filp); - struct block_device *bdev = I_BDEV(bd_inode); + struct block_device *bdev = blkdev_get_bdev(filp); int error; error = file_write_and_wait_range(filp, start, end); @@ -448,6 +449,8 @@ static int blkdev_open(struct inode *inode, struct file *filp) bdev = blkdev_get_by_dev(inode->i_rdev, filp->f_mode, filp); if (IS_ERR(bdev)) return PTR_ERR(bdev); + + filp->private_data = bdev; filp->f_mapping = bdev->bd_inode->i_mapping; filp->f_wb_err = filemap_sample_wb_err(filp->f_mapping); return 0; @@ -455,7 +458,7 @@ static int blkdev_open(struct inode *inode, struct file *filp) static int blkdev_close(struct inode *inode, struct file *filp) { - struct block_device *bdev = I_BDEV(bdev_file_inode(filp)); + struct block_device *bdev = blkdev_get_bdev(filp); blkdev_put(bdev, filp->f_mode); return 0; @@ -463,7 +466,7 @@ static int blkdev_close(struct inode *inode, struct file *filp) static long block_ioctl(struct file *file, unsigned cmd, unsigned long arg) { - struct block_device *bdev = I_BDEV(bdev_file_inode(file)); + struct block_device *bdev = blkdev_get_bdev(file); fmode_t mode = file->f_mode; /* @@ -487,14 +490,14 @@ static long block_ioctl(struct file *file, unsigned cmd, unsigned long arg) */ static ssize_t blkdev_write_iter(struct kiocb *iocb, struct iov_iter *from) { - struct file *file = iocb->ki_filp; - struct inode *bd_inode = bdev_file_inode(file); + struct block_device *bdev = blkdev_get_bdev(iocb->ki_filp); + struct inode *bd_inode = bdev->bd_inode; loff_t size = i_size_read(bd_inode); struct blk_plug plug; size_t shorted = 0; ssize_t ret; - if (bdev_read_only(I_BDEV(bd_inode))) + if (bdev_read_only(bdev)) return -EPERM; if (IS_SWAPFILE(bd_inode) && !is_hibernate_resume_dev(bd_inode->i_rdev)) @@ -526,9 +529,8 @@ static ssize_t blkdev_write_iter(struct kiocb *iocb, struct iov_iter *from) static ssize_t blkdev_read_iter(struct kiocb *iocb, struct iov_iter *to) { - struct file *file = iocb->ki_filp; - struct inode *bd_inode = bdev_file_inode(file); - loff_t size = i_size_read(bd_inode); + struct block_device *bdev = blkdev_get_bdev(iocb->ki_filp); + loff_t size = (loff_t)bdev->bd_nr_sectors << SECTOR_SHIFT; loff_t pos = iocb->ki_pos; size_t shorted = 0; ssize_t ret; From patchwork Sat Oct 9 12:25:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12547645 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B9C3C433FE for ; Sat, 9 Oct 2021 12:26:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4EB2C60F59 for ; Sat, 9 Oct 2021 12:26:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233201AbhJIM2a (ORCPT ); Sat, 9 Oct 2021 08:28:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60320 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233186AbhJIM23 (ORCPT ); Sat, 9 Oct 2021 08:28:29 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AE3A5C061762; Sat, 9 Oct 2021 05:26:32 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id e12so38207168wra.4; Sat, 09 Oct 2021 05:26:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=lOkOoGCR8g+RaYSCtRD/VjPKL2otHRzFfysHjyl+ocY=; b=FDp7XT6/26f/AYeKRd6/diB0FlzRUVDak/RHzV867K5Im1qGg8AVVM1lACJY/7acY6 cld2zNvBL6Q5j9KAQAo5vJtAaM5rsEuIdw4iep+0xQ35Gi+kgdUjqoY5RuFRxBy/Tenf 7GQXnNIkLlWiwp4Uqv+7Y5bgRwzCUo89YtHjwIkQM8M01poTmoBp6XvMRsP5mOw9M2O1 9bBKBA+5nzrHc97Y8yJ990JU/AzVjoiTUyZa/WapXOuoVOLrewIFqyiD+3jelxFFzC4m 4GWLnAtftAQmhWYL8PiddpB0Hc3pBDEsPXDXa+XELrU38YnNy7J0a/TNPAt9E6zXNVoy tdpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lOkOoGCR8g+RaYSCtRD/VjPKL2otHRzFfysHjyl+ocY=; b=Fy7ZpkNFYC9I69R37NcRZnUir3lJyU4fTEaJM/onNRfs2ljq+hTdzBpxU3UEcZ+VbL h/NfxXzB/gjMxeW+JTc8qkKrMpLQ/F+D1GNRUPVvQfRO7IapaX9xAwGdjCndtj9y08VD LOY1z19DSvRhPYa+qqOIZFJShJBkus5mCN/xA1ncwhj4OQKN1jqrFwATH/yGKoyFichl iMU92oB+bWRUXBp+9oSqSFBB8khpnWpQN6FGp3lYYGhWIMSgxjqyQuvHv0zt147Diodx 38nCYXyBG/rosRl6yFZsZpLp4CnaEN4ZhQ+afWuVtlK7ps9F1dirLhuAAHOYU3DorzaP 4fLA== X-Gm-Message-State: AOAM530vQAgmXiPb82EJYG2TF5x5rOytQRolXXmhXZKCT+CKgu5LFj6D STUXuRbnym1ciy1Zpt8jI42L/3IROBo= X-Google-Smtp-Source: ABdhPJyO+HGYW7FAnINHgkVxQgTCzAdx/0I0dl+mhSzO3wVcTg2vIHhxSTwggKVzvJXElOxN/Wa6lQ== X-Received: by 2002:adf:b185:: with SMTP id q5mr10853402wra.213.1633782391235; Sat, 09 Oct 2021 05:26:31 -0700 (PDT) Received: from localhost.localdomain ([85.255.236.155]) by smtp.gmail.com with ESMTPSA id o12sm2157468wrv.78.2021.10.09.05.26.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 09 Oct 2021 05:26:30 -0700 (PDT) From: Pavel Begunkov To: Jens Axboe , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH 2/6] block: inline BDEV_I and friends Date: Sat, 9 Oct 2021 13:25:39 +0100 Message-Id: <7c112aa98c9268d3d488cbee8b2832dc707812f6.1633781740.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org I_BDEV and BDEV_I are very simple, they worth a single arith instruction or can even be almost completely compiled out. Inline them. Signed-off-by: Pavel Begunkov --- block/bdev.c | 16 ---------------- include/linux/blkdev.h | 16 +++++++++++++++- 2 files changed, 15 insertions(+), 17 deletions(-) diff --git a/block/bdev.c b/block/bdev.c index 567534c63f3d..a6cdfc49bc7e 100644 --- a/block/bdev.c +++ b/block/bdev.c @@ -30,22 +30,6 @@ #include "../fs/internal.h" #include "blk.h" -struct bdev_inode { - struct block_device bdev; - struct inode vfs_inode; -}; - -static inline struct bdev_inode *BDEV_I(struct inode *inode) -{ - return container_of(inode, struct bdev_inode, vfs_inode); -} - -struct block_device *I_BDEV(struct inode *inode) -{ - return &BDEV_I(inode)->bdev; -} -EXPORT_SYMBOL(I_BDEV); - static void bdev_write_inode(struct block_device *bdev) { struct inode *inode = bdev->bd_inode; diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 4522acee81fb..591f14522f78 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1271,10 +1271,24 @@ void blkdev_put_no_open(struct block_device *bdev); struct block_device *bdev_alloc(struct gendisk *disk, u8 partno); void bdev_add(struct block_device *bdev, dev_t dev); -struct block_device *I_BDEV(struct inode *inode); int truncate_bdev_range(struct block_device *bdev, fmode_t mode, loff_t lstart, loff_t lend); +struct bdev_inode { + struct block_device bdev; + struct inode vfs_inode; +}; + +static inline struct bdev_inode *BDEV_I(struct inode *inode) +{ + return container_of(inode, struct bdev_inode, vfs_inode); +} + +static inline struct block_device *I_BDEV(struct inode *inode) +{ + return &BDEV_I(inode)->bdev; +} + #ifdef CONFIG_BLOCK void invalidate_bdev(struct block_device *bdev); int sync_blockdev(struct block_device *bdev); From patchwork Sat Oct 9 12:25:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12547643 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BF4EC43217 for ; Sat, 9 Oct 2021 12:26:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 738D1610CC for ; Sat, 9 Oct 2021 12:26:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233152AbhJIM2c (ORCPT ); Sat, 9 Oct 2021 08:28:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60326 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233222AbhJIM2a (ORCPT ); Sat, 9 Oct 2021 08:28:30 -0400 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8A5D8C061570; Sat, 9 Oct 2021 05:26:33 -0700 (PDT) Received: by mail-wr1-x42d.google.com with SMTP id k7so38057755wrd.13; Sat, 09 Oct 2021 05:26:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=nvPx8lZAWyHDeejaspcE8O05AFHduIeFac/IHHVZ+NY=; b=Ob+CtPivfLbryyTc3OWjMUHPC8fmy15GQtK5cE+lTSzNQTDfkbJeVa1z8bd/I5lKsV eX8DNR37fuK8It3nxcDsUT8fYf6qxvwWaxNP/o2oMFndRbUg9tRadMC1vH74XBUbKGdl JSbL3tMs/Y3D7Y2Umh9i5EarZWk0Qe+Yw99E60Hsgqhhm3JgADNl4z7fDnPqVq7yCC9R y6y4BNvxnBFFUpa9Y43rV5VdowFostf+P2DdcLm+pCV+H9KUTmtxZrpINoKTHpuyU8gv rB7fiHrsddEBXZNt44GWBOf9MnRKJCpTg1wvQLeKXR3OuX6PVgaL3Lku1RlQ0tlGE5BK Sing== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nvPx8lZAWyHDeejaspcE8O05AFHduIeFac/IHHVZ+NY=; b=r97UXRQJxYg+cfMjSa8JvsPHz+Tixrm4HVc1D/2/Z7Q0eAe9V1ZkpZLMLcY3+EmeMG n37i0dO6UlR/VEE0GOyE/0T6p0MxCloHoon8wVpjABk32GzPUf/hKMP1aXNThs1UlTJm AEzgdDEuDGDldFebZ/MtOJAd0ZCpHlS0y71jfFFayRhOMgOLyRSc8s/sn9rrPhe6uaEP 8pM9s2WIQwEGXQSZWOBHYgoEU8AWg/4yGxq21b2d6ok5qLFXNiI4id/qxI9A1YyM78Jp ywZuWWJUs2/ePipZZqPDSW1dLTm81DAw/pPU+s0QWWY8+vJfbm4h2VzXcBcsohOPX8Yu it1Q== X-Gm-Message-State: AOAM533pvMrU6EXSrtgHQHbadigMbSdTmgdQRBAbCZrFomoZc9BIObgG 3UH6y/+HS1kIQX33QvfNP5xdHRwGqdU= X-Google-Smtp-Source: ABdhPJyP9iHPJrjMXhzMc/NvIXnQnJyJbiEI+pZenh3603+4fOQe2WsGm1YjdwC32u4F9ylg2L8NDg== X-Received: by 2002:a05:600c:4f42:: with SMTP id m2mr9221840wmq.151.1633782392162; Sat, 09 Oct 2021 05:26:32 -0700 (PDT) Received: from localhost.localdomain ([85.255.236.155]) by smtp.gmail.com with ESMTPSA id o12sm2157468wrv.78.2021.10.09.05.26.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 09 Oct 2021 05:26:31 -0700 (PDT) From: Pavel Begunkov To: Jens Axboe , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH 3/6] blk-mq: optimise *end_request non-stat path Date: Sat, 9 Oct 2021 13:25:40 +0100 Message-Id: <908aa33b09e1fd6afa14caa719684121f380be71.1633781740.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org We already have a blk_mq_need_time_stamp() check in __blk_mq_end_request() to get a timestamp, hide all the statistics accounting under it. It cuts some cycles for requests that don't need stats, and is free otherwise. Signed-off-by: Pavel Begunkov --- block/blk-mq.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 9430a0def2c9..c3da521efd35 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -584,12 +584,11 @@ static inline void __blk_mq_end_request_acct(struct request *rq, inline void __blk_mq_end_request(struct request *rq, blk_status_t error) { - u64 now = 0; - - if (blk_mq_need_time_stamp(rq)) - now = ktime_get_ns(); + if (blk_mq_need_time_stamp(rq)) { + u64 now = ktime_get_ns(); - __blk_mq_end_request_acct(rq, error, now); + __blk_mq_end_request_acct(rq, error, now); + } if (rq->end_io) { rq_qos_done(rq->q, rq); From patchwork Sat Oct 9 12:25:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12547647 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7F98C43219 for ; Sat, 9 Oct 2021 12:26:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8D261610CF for ; Sat, 9 Oct 2021 12:26:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244790AbhJIM2d (ORCPT ); Sat, 9 Oct 2021 08:28:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60332 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244718AbhJIM2b (ORCPT ); Sat, 9 Oct 2021 08:28:31 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 78638C061570; Sat, 9 Oct 2021 05:26:34 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id e12so38207336wra.4; Sat, 09 Oct 2021 05:26:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=4GnHAbXKqGZGYNjF8KNJmEmT59MtbTDG5MrsdXPqRsg=; b=ccrKi0VxIswycMApCYtdwGUjtn4Xq9SFQaprAK4SCCMwUfHeICQNUFPZcpSptLxRUC I3HbdixkxvgRWvyY3nlr//pkY/RQTeBGWkZHWeSFWTBkWetPg6Odk0oYDXUUA0GQ1ihq ma5CxtA96jLaT8XkzM4WhDYIdVvDYd6WGSVhdBkZzUv2tSgs4KqlZtj/27TWOSa8lUlI UVCwof3iNs5uH3tk/pclz8yawGS9auyASoMfkqDkGU/UJQ5xROtZTl4mVubI/Hl9S4HV Dyd4tYMtf5m4RhjeiekXyI50S+q56CQ3Y2NTm4Bts8Q25DvMWIAWhtuYxzM9RZtzPKUt sWMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4GnHAbXKqGZGYNjF8KNJmEmT59MtbTDG5MrsdXPqRsg=; b=c7jJqIbElHvibwoKLFz49/DF1POe5Qi/S/R9MeENQKy3waM6FYJpi/5crm8c2OaDDL YDCpJydBGABJmQjyQ/aoaOSPKF39UXoYJAiONHc5snGEnTc6Q8nT2hNiWfZ6h2HOnnVf zeKGxzeB7NTkymuZe9i7W8xkmSp1Q3nABm+e0n/DgGfXR4VRjSshSMf3SE+no1GPm2EK yTQPov1jrSKQSfYqA1Ac1ixyPSSi9/0Ay0rKCOAjKE9q/Ne9AB5fhoXYsQjZDkSOqV13 XZnubA8dZcDjV5Dm4sY1cAqZy4UBWJ5ZbIh6OI9PfGEduClBjOd9k5UVyo6IIdh5LfYt Pq4A== X-Gm-Message-State: AOAM531x39QDWkHdECpivYqmilp1bJCbPSHxKs9AHAINwCoAvwuVtHVS xbXa76vjA/Qsyg0p4Rnq9UI= X-Google-Smtp-Source: ABdhPJwt96jv6wrvfiB/AYbqhiqiyEPg9qkMQrzBcoxBhWiudVYrYCGD2xlDyvnJCdRyDQRPdR16WA== X-Received: by 2002:a5d:43c3:: with SMTP id v3mr1734403wrr.22.1633782393089; Sat, 09 Oct 2021 05:26:33 -0700 (PDT) Received: from localhost.localdomain ([85.255.236.155]) by smtp.gmail.com with ESMTPSA id o12sm2157468wrv.78.2021.10.09.05.26.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 09 Oct 2021 05:26:32 -0700 (PDT) From: Pavel Begunkov To: Jens Axboe , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH 4/6] block: inline hot paths of blk_account_io_*() Date: Sat, 9 Oct 2021 13:25:41 +0100 Message-Id: X-Mailer: git-send-email 2.33.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Extract hot paths of __blk_account_io_start() and __blk_account_io_done() into inline functions, so we don't always pay for function calls. Signed-off-by: Pavel Begunkov --- block/blk-core.c | 30 +++++++++--------------------- block/blk.h | 24 +++++++++++++++++++++--- 2 files changed, 30 insertions(+), 24 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 9b8c70670190..6a9607a22589 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1172,8 +1172,7 @@ blk_status_t blk_insert_cloned_request(struct request_queue *q, struct request * if (blk_crypto_insert_cloned_request(rq)) return BLK_STS_IOERR; - if (blk_queue_io_stat(q)) - blk_account_io_start(rq); + blk_account_io_start(rq); /* * Since we have a scheduler attached on the top device, @@ -1252,30 +1251,19 @@ static void blk_account_io_completion(struct request *req, unsigned int bytes) } } -void blk_account_io_done(struct request *req, u64 now) +void __blk_account_io_done(struct request *req, u64 now) { - /* - * Account IO completion. flush_rq isn't accounted as a - * normal IO on queueing nor completion. Accounting the - * containing request is enough. - */ - if (req->part && blk_do_io_stat(req) && - !(req->rq_flags & RQF_FLUSH_SEQ)) { - const int sgrp = op_stat_group(req_op(req)); + const int sgrp = op_stat_group(req_op(req)); - part_stat_lock(); - update_io_ticks(req->part, jiffies, true); - part_stat_inc(req->part, ios[sgrp]); - part_stat_add(req->part, nsecs[sgrp], now - req->start_time_ns); - part_stat_unlock(); - } + part_stat_lock(); + update_io_ticks(req->part, jiffies, true); + part_stat_inc(req->part, ios[sgrp]); + part_stat_add(req->part, nsecs[sgrp], now - req->start_time_ns); + part_stat_unlock(); } -void blk_account_io_start(struct request *rq) +void __blk_account_io_start(struct request *rq) { - if (!blk_do_io_stat(rq)) - return; - /* passthrough requests can hold bios that do not have ->bi_bdev set */ if (rq->bio && rq->bio->bi_bdev) rq->part = rq->bio->bi_bdev; diff --git a/block/blk.h b/block/blk.h index 38867b4c5c7e..5d74270314ea 100644 --- a/block/blk.h +++ b/block/blk.h @@ -219,8 +219,8 @@ bool blk_attempt_plug_merge(struct request_queue *q, struct bio *bio, bool blk_bio_list_merge(struct request_queue *q, struct list_head *list, struct bio *bio, unsigned int nr_segs); -void blk_account_io_start(struct request *req); -void blk_account_io_done(struct request *req, u64 now); +void __blk_account_io_start(struct request *req); +void __blk_account_io_done(struct request *req, u64 now); /* * Plug flush limits @@ -284,7 +284,25 @@ int blk_dev_init(void); */ static inline bool blk_do_io_stat(struct request *rq) { - return rq->rq_disk && (rq->rq_flags & RQF_IO_STAT); + return (rq->rq_flags & RQF_IO_STAT) && rq->rq_disk; +} + +static inline void blk_account_io_done(struct request *req, u64 now) +{ + /* + * Account IO completion. flush_rq isn't accounted as a + * normal IO on queueing nor completion. Accounting the + * containing request is enough. + */ + if (blk_do_io_stat(req) && req->part && + !(req->rq_flags & RQF_FLUSH_SEQ)) + __blk_account_io_done(req, now); +} + +static inline void blk_account_io_start(struct request *req) +{ + if (blk_do_io_stat(req)) + __blk_account_io_start(req); } static inline void req_set_nomerge(struct request_queue *q, struct request *req) From patchwork Sat Oct 9 12:25:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12547649 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C7B9C4321E for ; Sat, 9 Oct 2021 12:26:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 413C5610A4 for ; Sat, 9 Oct 2021 12:26:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233186AbhJIM2f (ORCPT ); Sat, 9 Oct 2021 08:28:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60338 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244766AbhJIM2c (ORCPT ); Sat, 9 Oct 2021 08:28:32 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 57DCFC061570; Sat, 9 Oct 2021 05:26:35 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id v25so38142603wra.2; Sat, 09 Oct 2021 05:26:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KXi3eQ73M3DvHkT3Ar7ro+5G3w3X7uWmCy95vClAZtU=; b=Uq8PQQp2XiQsEP9uP52glrC5owaiHl2ABYWB0ACGkuleldivxHIT2Q+ej6Ij/wSnGk dydYr2wBCu3JFT8MIHW0SWa1J/QZfUz5JK4MQZQQ1URM3pJ8p0k8qvI6CLWNU66UBkpe 1T1yMAiUNP+YfZ63iDN4KlYXY8udvFTiI6at3yPlBltd06l/9db0Zltw5hC/vL+mux+2 rHI9/l6GqURgNkdzfFYPUbiqJtIhKgqex9GfeD2URtQu1eKKJ89jC3YPFs9de/RQytAc lJNxHNp+qSM7JGlbGEOw3aFvXHP205el8zaKpLxh3Pk2bpbpcaQMz7dWz4QsfyqTYa3z 4lIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KXi3eQ73M3DvHkT3Ar7ro+5G3w3X7uWmCy95vClAZtU=; b=Iu2t+w8lm25ijvPt1QS0rvvSuFCPNJC6YqyvpdSDfNobmjFyZjdiYYy22g3/qWeT5X z1u4CsFW2iYUcm6jcDnsbEwhn0lxOdIZ2IZL/jNi7iRpXUsWygfW3NjrBT3UmrdLmjd5 7KkJBlo75PFxkdEzcAWQW7Iy3bhw8v9FyrcX5kcZF7AcS13GlazAN+uOwiz+ZhKOMCJm jxPJgWxbr12EgSyVl9DzfNimcRMevpzyjD3sbQ26slsxlo/kgUIP++RxU8OhDWWMI4mY 3so0M0DpSGC81C2MmVD2K8mp3C8KwD3UFIzFlm3DsK/b4VK9cZaMFF1D8tw0zEs3T22K 2S4A== X-Gm-Message-State: AOAM5309GByem/6iBEmlkr+WSQAQU2zeYu1gaezBhn7D0vogFEWbhrnX u53Bn+j9Ufze7CaTWkqkIDR8NBjQUk4= X-Google-Smtp-Source: ABdhPJzPb9VCN0YZ28ZNQDsxHWo5LefRacMk0FbD20qbYUOyAfqF/tAQloqgbgn4KEt1sv7EvADYVA== X-Received: by 2002:adf:aad7:: with SMTP id i23mr10579496wrc.209.1633782394002; Sat, 09 Oct 2021 05:26:34 -0700 (PDT) Received: from localhost.localdomain ([85.255.236.155]) by smtp.gmail.com with ESMTPSA id o12sm2157468wrv.78.2021.10.09.05.26.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 09 Oct 2021 05:26:33 -0700 (PDT) From: Pavel Begunkov To: Jens Axboe , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH 5/6] blk-mq: inline hot part of __blk_mq_sched_restart Date: Sat, 9 Oct 2021 13:25:42 +0100 Message-Id: <894abaa0998e5999f2fe18f271e5efdfc2c32bd2.1633781740.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Extract a fast check out of __block_mq_sched_restart() and inline it for performance reasons. Signed-off-by: Pavel Begunkov --- block/blk-mq-sched.c | 4 +--- block/blk-mq-sched.h | 8 +++++++- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c index 27312da7d638..efc1374b8831 100644 --- a/block/blk-mq-sched.c +++ b/block/blk-mq-sched.c @@ -57,10 +57,8 @@ void blk_mq_sched_mark_restart_hctx(struct blk_mq_hw_ctx *hctx) } EXPORT_SYMBOL_GPL(blk_mq_sched_mark_restart_hctx); -void blk_mq_sched_restart(struct blk_mq_hw_ctx *hctx) +void __blk_mq_sched_restart(struct blk_mq_hw_ctx *hctx) { - if (!test_bit(BLK_MQ_S_SCHED_RESTART, &hctx->state)) - return; clear_bit(BLK_MQ_S_SCHED_RESTART, &hctx->state); /* diff --git a/block/blk-mq-sched.h b/block/blk-mq-sched.h index e4d367e0b2a3..c97b816c3800 100644 --- a/block/blk-mq-sched.h +++ b/block/blk-mq-sched.h @@ -17,7 +17,7 @@ bool __blk_mq_sched_bio_merge(struct request_queue *q, struct bio *bio, bool blk_mq_sched_try_insert_merge(struct request_queue *q, struct request *rq, struct list_head *free); void blk_mq_sched_mark_restart_hctx(struct blk_mq_hw_ctx *hctx); -void blk_mq_sched_restart(struct blk_mq_hw_ctx *hctx); +void __blk_mq_sched_restart(struct blk_mq_hw_ctx *hctx); void blk_mq_sched_insert_request(struct request *rq, bool at_head, bool run_queue, bool async); @@ -31,6 +31,12 @@ int blk_mq_init_sched(struct request_queue *q, struct elevator_type *e); void blk_mq_exit_sched(struct request_queue *q, struct elevator_queue *e); void blk_mq_sched_free_rqs(struct request_queue *q); +static inline void blk_mq_sched_restart(struct blk_mq_hw_ctx *hctx) +{ + if (test_bit(BLK_MQ_S_SCHED_RESTART, &hctx->state)) + __blk_mq_sched_restart(hctx); +} + static inline bool blk_mq_sched_bio_merge(struct request_queue *q, struct bio *bio, unsigned int nr_segs) From patchwork Sat Oct 9 12:25:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12547651 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE9D2C4167D for ; Sat, 9 Oct 2021 12:26:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C7E7E60F59 for ; Sat, 9 Oct 2021 12:26:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244811AbhJIM2f (ORCPT ); Sat, 9 Oct 2021 08:28:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60346 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244802AbhJIM2d (ORCPT ); Sat, 9 Oct 2021 08:28:33 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4E21EC061764; Sat, 9 Oct 2021 05:26:36 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id u18so38278300wrg.5; Sat, 09 Oct 2021 05:26:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=f0JhrekTKY9vxYly7vj2LWwfA9ARSkjdGbzmdHw9WmY=; b=L1EMbvUqLZid8kfwv2zU/4SQiNs49xmit9AXaEtv1oLXKxy1Anf4+yUiXSxDcGlDxA zRaCfFr6biePLTycor4vApxY07CXxHK1L7CdVz+/cRfjcyVPQsCY6smJ1+nZkX1+IeF8 gpZNiVGEl432OjRZcSg45MNQDuz1z19zbQCyntRKdf0rEacU+vckuxVCiT3kcpjse4hA TNqpAyXxPGB4qbLe0TABk+0VYudh6IwdJOk+/92krQ2i2sAf3PmStG/UEuDj8fgzI9FB +HM+pj0jqX8UaT0vuJi1srBbwbiUVbUlkYYXVQUHrsIS34YJPzhk7zOr01JUL33kYHEn HMnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=f0JhrekTKY9vxYly7vj2LWwfA9ARSkjdGbzmdHw9WmY=; b=F0If0ezmFcdPkz//+BZsPyo1ot6i5l51drkQFtaI58a3M3S+7aF/vcEftTHfJ97vX4 rF9a3A42vceH6kDwvWTZ6FBxQzJZQezZYxrb0CblnXyBvTrspitP+ypaVFL1rhxdWe3X 2nyfLY4Z7LYk/jGtWpqQhtQ0Oy6Mu0wEVfMQbEGGlkP6Py8baalh9JDYa9cRqI96bGv2 uYK5v3bNoNOKumRM9zgawo7HOL/X3EE5EwTOl3oPKs2Yq5Y/MiWBsocJourEXZQ7TjJf zt+Pomi7BYFmPOJ4hZ1l29OIfzrwBhjmgWGQr1VS1DvPfKyDTUnBQsrmlnw1cyp8Tx+f Fjww== X-Gm-Message-State: AOAM530IxKjsNOVBXvabU/nlshSzed8Vj3H5HEuZNbf/vQ8kcQIYc2cp yN62BITy9TeHMDpWmoaXzr8= X-Google-Smtp-Source: ABdhPJz9Ha7rjVm1WKtlR2FyUUcbAmIXHm9H+DKclk9cd6VZk2xOznra66Hkuqt85ComN4d62H/1dw== X-Received: by 2002:a1c:9dcb:: with SMTP id g194mr9579136wme.174.1633782394916; Sat, 09 Oct 2021 05:26:34 -0700 (PDT) Received: from localhost.localdomain ([85.255.236.155]) by smtp.gmail.com with ESMTPSA id o12sm2157468wrv.78.2021.10.09.05.26.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 09 Oct 2021 05:26:34 -0700 (PDT) From: Pavel Begunkov To: Jens Axboe , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH 6/6] block: convert ->bd_inode to container_of() Date: Sat, 9 Oct 2021 13:25:43 +0100 Message-Id: <41af3da80d59b705eb2260f7f469955ad68a96d2.1633781740.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org We don't need bdev->bd_inode as we know the layout, they are stored in the same structure and so offset_of is enough. Convert extra dereferencing to offseting starting from the block layer. Signed-off-by: Pavel Begunkov --- block/fops.c | 10 ++++++---- include/linux/blkdev.h | 15 ++++++++++----- 2 files changed, 16 insertions(+), 9 deletions(-) diff --git a/block/fops.c b/block/fops.c index 99e699427f31..5438ed9cfcf7 100644 --- a/block/fops.c +++ b/block/fops.c @@ -17,14 +17,16 @@ #include #include "blk.h" -static inline struct inode *bdev_file_inode(struct file *file) +static inline struct block_device *blkdev_get_bdev(struct file *file) { - return file->f_mapping->host; + return file->private_data; } -static inline struct block_device *blkdev_get_bdev(struct file *file) +static inline struct inode *bdev_file_inode(struct file *file) { - return file->private_data; + struct block_device *bdev = blkdev_get_bdev(file); + + return bdev_get_inode(bdev); } static int blkdev_get_block(struct inode *inode, sector_t iblock, diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 591f14522f78..a56f3a852206 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1139,11 +1139,6 @@ static inline unsigned int blksize_bits(unsigned int size) return bits; } -static inline unsigned int block_size(struct block_device *bdev) -{ - return 1 << bdev->bd_inode->i_blkbits; -} - int kblockd_schedule_work(struct work_struct *work); int kblockd_mod_delayed_work_on(int cpu, struct delayed_work *dwork, unsigned long delay); @@ -1289,6 +1284,16 @@ static inline struct block_device *I_BDEV(struct inode *inode) return &BDEV_I(inode)->bdev; } +static inline struct inode *bdev_get_inode(struct block_device *bdev) +{ + return &container_of(bdev, struct bdev_inode, bdev)->vfs_inode; +} + +static inline unsigned int block_size(struct block_device *bdev) +{ + return 1 << bdev_get_inode(bdev)->i_blkbits; +} + #ifdef CONFIG_BLOCK void invalidate_bdev(struct block_device *bdev); int sync_blockdev(struct block_device *bdev);