From patchwork Mon Feb 14 17:43:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Roesch X-Patchwork-Id: 12745938 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5793C433EF for ; Mon, 14 Feb 2022 17:44:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1357223AbiBNRow (ORCPT ); Mon, 14 Feb 2022 12:44:52 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:34924 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1357214AbiBNRol (ORCPT ); Mon, 14 Feb 2022 12:44:41 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B450765483 for ; Mon, 14 Feb 2022 09:44:33 -0800 (PST) Received: from pps.filterd (m0148461.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 21ECHRdS024089 for ; Mon, 14 Feb 2022 09:44:33 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=BwSx3L7ogY20VkY+MOK+kIbEIgoxq4mLJVLVeNC1mH0=; b=gWzKTHA9Q8bnDJVSOb/BQJBEf1myXs00kLUwYKEFzyYI5z9Omj8w8qNYDeAvjQnZFgeL jQ4z0T9orIjQZjVv/IhD6TY/7q5eyEAsNOihoyKFMtphZO1k8hSZE2UTe2I5/0MNzGM2 jPKHBUpYZ2ky9a9gUQOXud+hlGIurjVPQT4= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3e7py4j44e-9 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 14 Feb 2022 09:44:33 -0800 Received: from twshared12416.02.ash9.facebook.com (2620:10d:c085:208::11) by mail.thefacebook.com (2620:10d:c085:21d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.21; Mon, 14 Feb 2022 09:44:21 -0800 Received: by devvm225.atn0.facebook.com (Postfix, from userid 425415) id 8F333ABBD0F5; Mon, 14 Feb 2022 09:44:09 -0800 (PST) From: Stefan Roesch To: , , , CC: Subject: [PATCH v1 01/14] fs: Add flags parameter to __block_write_begin_int Date: Mon, 14 Feb 2022 09:43:50 -0800 Message-ID: <20220214174403.4147994-2-shr@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220214174403.4147994-1-shr@fb.com> References: <20220214174403.4147994-1-shr@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: KtgDqtVTkGJBv8soK3nbysKnPfKZUnk8 X-Proofpoint-GUID: KtgDqtVTkGJBv8soK3nbysKnPfKZUnk8 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.816,Hydra:6.0.425,FMLib:17.11.62.513 definitions=2022-02-14_07,2022-02-14_03,2021-12-02_01 X-Proofpoint-Spam-Details: rule=fb_outbound_notspam policy=fb_outbound score=0 phishscore=0 clxscore=1015 spamscore=0 bulkscore=0 adultscore=0 priorityscore=1501 suspectscore=0 mlxlogscore=820 impostorscore=0 lowpriorityscore=0 mlxscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2201110000 definitions=main-2202140105 X-FB-Internal: deliver Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org This adds a flags parameter to the __begin_write_begin_int() function. This allows to pass flags down the stack. Signed-off-by: Stefan Roesch --- fs/buffer.c | 7 ++++--- fs/internal.h | 3 ++- fs/iomap/buffered-io.c | 4 ++-- 3 files changed, 8 insertions(+), 6 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index 8e112b6bd371..6e6a69a12eed 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -1970,7 +1970,8 @@ iomap_to_bh(struct inode *inode, sector_t block, struct buffer_head *bh, } int __block_write_begin_int(struct folio *folio, loff_t pos, unsigned len, - get_block_t *get_block, const struct iomap *iomap) + get_block_t *get_block, const struct iomap *iomap, + unsigned int flags) { unsigned from = pos & (PAGE_SIZE - 1); unsigned to = from + len; @@ -2058,7 +2059,7 @@ int __block_write_begin(struct page *page, loff_t pos, unsigned len, get_block_t *get_block) { return __block_write_begin_int(page_folio(page), pos, len, get_block, - NULL); + NULL, 0); } EXPORT_SYMBOL(__block_write_begin); @@ -2118,7 +2119,7 @@ int block_write_begin(struct address_space *mapping, loff_t pos, unsigned len, if (!page) return -ENOMEM; - status = __block_write_begin(page, pos, len, get_block); + status = __block_write_begin_int(page_folio(page), pos, len, get_block, NULL, flags); if (unlikely(status)) { unlock_page(page); put_page(page); diff --git a/fs/internal.h b/fs/internal.h index 8590c973c2f4..7432df23f3ce 100644 --- a/fs/internal.h +++ b/fs/internal.h @@ -38,7 +38,8 @@ static inline int emergency_thaw_bdev(struct super_block *sb) * buffer.c */ int __block_write_begin_int(struct folio *folio, loff_t pos, unsigned len, - get_block_t *get_block, const struct iomap *iomap); + get_block_t *get_block, const struct iomap *iomap, + unsigned int flags); /* * char_dev.c diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c index 6c51a75d0be6..47c519952725 100644 --- a/fs/iomap/buffered-io.c +++ b/fs/iomap/buffered-io.c @@ -646,7 +646,7 @@ static int iomap_write_begin(const struct iomap_iter *iter, loff_t pos, if (srcmap->type == IOMAP_INLINE) status = iomap_write_begin_inline(iter, folio); else if (srcmap->flags & IOMAP_F_BUFFER_HEAD) - status = __block_write_begin_int(folio, pos, len, NULL, srcmap); + status = __block_write_begin_int(folio, pos, len, NULL, srcmap, 0); else status = __iomap_write_begin(iter, pos, len, folio); @@ -979,7 +979,7 @@ static loff_t iomap_folio_mkwrite_iter(struct iomap_iter *iter, if (iter->iomap.flags & IOMAP_F_BUFFER_HEAD) { ret = __block_write_begin_int(folio, iter->pos, length, NULL, - &iter->iomap); + &iter->iomap, 0); if (ret) return ret; block_commit_write(&folio->page, 0, length); From patchwork Mon Feb 14 17:43:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Roesch X-Patchwork-Id: 12745929 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10090C433FE for ; Mon, 14 Feb 2022 17:44:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1357199AbiBNRo0 (ORCPT ); Mon, 14 Feb 2022 12:44:26 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:34650 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1357176AbiBNRoZ (ORCPT ); Mon, 14 Feb 2022 12:44:25 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B4E5165438 for ; Mon, 14 Feb 2022 09:44:16 -0800 (PST) Received: from pps.filterd (m0148461.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 21ECHTrT024115 for ; Mon, 14 Feb 2022 09:44:16 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=HeRa2xrPrzZWD3nBwmQhy5/qvO2TqyLuQ8Y3Q09V+fQ=; b=YGm+a3Ur7LIiy+gx4ZBHMIrifsmQgdAq+U5TZR801eM2CaRQD2Dbl5PGZavpyjmIaUh0 2HJ3Tik4cbS04Ig1GFIc/daIKtiM9duwIfj5jKpa0VCLv1wSudwNKBG2ouqiyS0R9NBj caAmL71NKoIq/CF5kLWb8R6jGskEFenEWHI= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3e7py4j416-8 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 14 Feb 2022 09:44:16 -0800 Received: from twshared0983.05.ash8.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::f) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.21; Mon, 14 Feb 2022 09:44:15 -0800 Received: by devvm225.atn0.facebook.com (Postfix, from userid 425415) id 94151ABBD0F7; Mon, 14 Feb 2022 09:44:09 -0800 (PST) From: Stefan Roesch To: , , , CC: Subject: [PATCH v1 02/14] mm: Introduce do_generic_perform_write Date: Mon, 14 Feb 2022 09:43:51 -0800 Message-ID: <20220214174403.4147994-3-shr@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220214174403.4147994-1-shr@fb.com> References: <20220214174403.4147994-1-shr@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: MLKNRVNjg1k_kRc8-2kGuSMHOgk8MxpX X-Proofpoint-GUID: MLKNRVNjg1k_kRc8-2kGuSMHOgk8MxpX X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.816,Hydra:6.0.425,FMLib:17.11.62.513 definitions=2022-02-14_07,2022-02-14_03,2021-12-02_01 X-Proofpoint-Spam-Details: rule=fb_outbound_notspam policy=fb_outbound score=0 phishscore=0 clxscore=1015 spamscore=0 bulkscore=0 adultscore=0 priorityscore=1501 suspectscore=0 mlxlogscore=750 impostorscore=0 lowpriorityscore=0 mlxscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2201110000 definitions=main-2202140105 X-FB-Internal: deliver Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org This splits off the do generic_perform_write() function, so an additional flags parameter can be specified. It uses the new flag parameter to support async buffered writes. Signed-off-by: Stefan Roesch --- include/linux/fs.h | 1 + mm/filemap.c | 20 +++++++++++++++----- 2 files changed, 16 insertions(+), 5 deletions(-) diff --git a/include/linux/fs.h b/include/linux/fs.h index e2d892b201b0..e62dba6ed453 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -278,6 +278,7 @@ enum positive_aop_returns { #define AOP_FLAG_NOFS 0x0002 /* used by filesystem to direct * helper code (eg buffer layer) * to clear GFP_FS from alloc */ +#define AOP_FLAGS_NOWAIT 0x0004 /* async nowait buffered writes */ /* * oh the beauties of C type declarations. diff --git a/mm/filemap.c b/mm/filemap.c index ad8c39d90bf9..d2fb817c0845 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3725,14 +3725,13 @@ generic_file_direct_write(struct kiocb *iocb, struct iov_iter *from) } EXPORT_SYMBOL(generic_file_direct_write); -ssize_t generic_perform_write(struct file *file, - struct iov_iter *i, loff_t pos) +static ssize_t do_generic_perform_write(struct file *file, struct iov_iter *i, + loff_t pos, int flags) { struct address_space *mapping = file->f_mapping; const struct address_space_operations *a_ops = mapping->a_ops; long status = 0; ssize_t written = 0; - unsigned int flags = 0; do { struct page *page; @@ -3801,6 +3800,12 @@ ssize_t generic_perform_write(struct file *file, return written ? written : status; } + +ssize_t generic_perform_write(struct file *file, + struct iov_iter *i, loff_t pos) +{ + return do_generic_perform_write(file, i, pos, 0); +} EXPORT_SYMBOL(generic_perform_write); /** @@ -3832,6 +3837,10 @@ ssize_t __generic_file_write_iter(struct kiocb *iocb, struct iov_iter *from) ssize_t written = 0; ssize_t err; ssize_t status; + int flags = 0; + + if (iocb->ki_flags & IOCB_NOWAIT) + flags |= AOP_FLAGS_NOWAIT; /* We can write back this queue in page reclaim */ current->backing_dev_info = inode_to_bdi(inode); @@ -3857,7 +3866,8 @@ ssize_t __generic_file_write_iter(struct kiocb *iocb, struct iov_iter *from) if (written < 0 || !iov_iter_count(from) || IS_DAX(inode)) goto out; - status = generic_perform_write(file, from, pos = iocb->ki_pos); + status = do_generic_perform_write(file, from, pos = iocb->ki_pos, flags); + /* * If generic_perform_write() returned a synchronous error * then we want to return the number of bytes which were @@ -3889,7 +3899,7 @@ ssize_t __generic_file_write_iter(struct kiocb *iocb, struct iov_iter *from) */ } } else { - written = generic_perform_write(file, from, iocb->ki_pos); + written = do_generic_perform_write(file, from, iocb->ki_pos, flags); if (likely(written > 0)) iocb->ki_pos += written; } From patchwork Mon Feb 14 17:43:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Roesch X-Patchwork-Id: 12745935 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 47311C433FE for ; Mon, 14 Feb 2022 17:44:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1357221AbiBNRop (ORCPT ); Mon, 14 Feb 2022 12:44:45 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:34926 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1357218AbiBNRoj (ORCPT ); Mon, 14 Feb 2022 12:44:39 -0500 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6EB8065490 for ; Mon, 14 Feb 2022 09:44:28 -0800 (PST) Received: from pps.filterd (m0148460.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 21ECHEOC005239 for ; Mon, 14 Feb 2022 09:44:27 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=LSdyRHAc+fsIsWQJgyJviG9+Wzpmsxt7o7RJSYAE/qo=; b=a2NSIZRDQjUIc/rxJ19LXaDPQ1pgcWrQ8PPmaQ8O6UV7g2p3zJCPOt7WEQUXKDEpA3/X 7KfKna1RI81sgGaSsWMWHh6EeyXZwkS9V2FZ8jhdz93kE8EYaDOfxTfkIxDqQRZPOrm5 IKbCrCeR+PquW4gFE1y70dIoI7zvRMIjNNY= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3e7pxr25wr-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 14 Feb 2022 09:44:27 -0800 Received: from twshared14630.35.frc1.facebook.com (2620:10d:c085:108::4) by mail.thefacebook.com (2620:10d:c085:11d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.21; Mon, 14 Feb 2022 09:44:26 -0800 Received: by devvm225.atn0.facebook.com (Postfix, from userid 425415) id 9B758ABBD0F9; Mon, 14 Feb 2022 09:44:09 -0800 (PST) From: Stefan Roesch To: , , , CC: Subject: [PATCH v1 03/14] mm: add noio support in filemap_get_pages Date: Mon, 14 Feb 2022 09:43:52 -0800 Message-ID: <20220214174403.4147994-4-shr@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220214174403.4147994-1-shr@fb.com> References: <20220214174403.4147994-1-shr@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: nADUjkUGKTpyFLA4rp5tApJFnPRWmU1o X-Proofpoint-GUID: nADUjkUGKTpyFLA4rp5tApJFnPRWmU1o X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.816,Hydra:6.0.425,FMLib:17.11.62.513 definitions=2022-02-14_07,2022-02-14_03,2021-12-02_01 X-Proofpoint-Spam-Details: rule=fb_outbound_notspam policy=fb_outbound score=0 adultscore=0 bulkscore=0 mlxlogscore=959 priorityscore=1501 lowpriorityscore=0 phishscore=0 impostorscore=0 clxscore=1015 mlxscore=0 suspectscore=0 malwarescore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2201110000 definitions=main-2202140105 X-FB-Internal: deliver Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org This adds noio support for async buffered writes in filemap_get_pages. The idea is to handle the failure gracefully and return -EAGAIN if we can't get the memory quickly. Signed-off-by: Stefan Roesch --- mm/filemap.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/mm/filemap.c b/mm/filemap.c index d2fb817c0845..0ff4278c3961 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2591,10 +2591,15 @@ static int filemap_get_pages(struct kiocb *iocb, struct iov_iter *iter, filemap_get_read_batch(mapping, index, last_index, fbatch); } if (!folio_batch_count(fbatch)) { + unsigned int pflags; + if (iocb->ki_flags & (IOCB_NOWAIT | IOCB_WAITQ)) - return -EAGAIN; + pflags = memalloc_noio_save(); err = filemap_create_folio(filp, mapping, iocb->ki_pos >> PAGE_SHIFT, fbatch); + if (iocb->ki_flags & (IOCB_NOWAIT | IOCB_WAITQ)) + memalloc_noio_restore(pflags); + if (err == AOP_TRUNCATED_PAGE) goto retry; return err; From patchwork Mon Feb 14 17:43:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Roesch X-Patchwork-Id: 12745930 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88543C433EF for ; Mon, 14 Feb 2022 17:44:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1357206AbiBNRoc (ORCPT ); Mon, 14 Feb 2022 12:44:32 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:34758 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1357192AbiBNRo3 (ORCPT ); Mon, 14 Feb 2022 12:44:29 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D229F65483 for ; Mon, 14 Feb 2022 09:44:21 -0800 (PST) Received: from pps.filterd (m0109333.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 21EHiKfY026958 for ; Mon, 14 Feb 2022 09:44:21 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=TpLM1ylp8BkAtDLY4Cd3m5Nmn6K/IDr/a4obTwyIV9A=; b=qzsEjVOq93gYQwti+IpWN0r+BsO/fvrj0gStOAMfxm0DwRSUkqRaHEivKjuhGLa+cK0L udPl1LE1ExpULb/UkeCxmpl3gZlg7zL/s7Fs2Rtjk4ciySKQQpwL6VNQTe3Rbv+sewKe CcUEyvagMjFaan0VF53J25RkzY91YpbaWbE= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3e7hpcbju6-5 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 14 Feb 2022 09:44:21 -0800 Received: from twshared7634.08.ash8.facebook.com (2620:10d:c085:108::4) by mail.thefacebook.com (2620:10d:c085:21d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.21; Mon, 14 Feb 2022 09:44:20 -0800 Received: by devvm225.atn0.facebook.com (Postfix, from userid 425415) id A1E7CABBD0FB; Mon, 14 Feb 2022 09:44:09 -0800 (PST) From: Stefan Roesch To: , , , CC: Subject: [PATCH v1 04/14] mm: Add support for async buffered writes Date: Mon, 14 Feb 2022 09:43:53 -0800 Message-ID: <20220214174403.4147994-5-shr@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220214174403.4147994-1-shr@fb.com> References: <20220214174403.4147994-1-shr@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: Ex3qWCZdvSWgpL-Cgedy35cl-Qg1ZGF- X-Proofpoint-ORIG-GUID: Ex3qWCZdvSWgpL-Cgedy35cl-Qg1ZGF- X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.816,Hydra:6.0.425,FMLib:17.11.62.513 definitions=2022-02-14_07,2022-02-14_03,2021-12-02_01 X-Proofpoint-Spam-Details: rule=fb_outbound_notspam policy=fb_outbound score=0 mlxscore=0 phishscore=0 clxscore=1015 suspectscore=0 adultscore=0 mlxlogscore=812 lowpriorityscore=0 malwarescore=0 impostorscore=0 spamscore=0 priorityscore=1501 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2201110000 definitions=main-2202140105 X-FB-Internal: deliver Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org This adds support for async buffered writes in the mm layer. When the AOP_FLAGS_BUF_WASYNC flag is set, if the page is not already loaded, the page gets created without blocking on the allocation. Signed-off-by: Stefan Roesch --- mm/filemap.c | 5 +++++ mm/folio-compat.c | 4 ++++ 2 files changed, 9 insertions(+) diff --git a/mm/filemap.c b/mm/filemap.c index 0ff4278c3961..19065ad95a4c 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -42,6 +42,7 @@ #include #include #include +#include #include #include #include "internal.h" @@ -1986,6 +1987,10 @@ struct folio *__filemap_get_folio(struct address_space *mapping, pgoff_t index, gfp |= __GFP_WRITE; if (fgp_flags & FGP_NOFS) gfp &= ~__GFP_FS; + if (fgp_flags & FGP_NOWAIT) { + gfp |= GFP_ATOMIC; + gfp &= ~__GFP_DIRECT_RECLAIM; + } folio = filemap_alloc_folio(gfp, 0); if (!folio) diff --git a/mm/folio-compat.c b/mm/folio-compat.c index 749555a232a8..a1d05509b29f 100644 --- a/mm/folio-compat.c +++ b/mm/folio-compat.c @@ -136,6 +136,10 @@ struct page *grab_cache_page_write_begin(struct address_space *mapping, if (flags & AOP_FLAG_NOFS) fgp_flags |= FGP_NOFS; + + if (flags & AOP_FLAGS_NOWAIT) + fgp_flags |= FGP_NOWAIT; + return pagecache_get_page(mapping, index, fgp_flags, mapping_gfp_mask(mapping)); } From patchwork Mon Feb 14 17:43:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Roesch X-Patchwork-Id: 12745932 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EE438C4332F for ; Mon, 14 Feb 2022 17:44:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1357176AbiBNRok (ORCPT ); Mon, 14 Feb 2022 12:44:40 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:34864 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1357211AbiBNRof (ORCPT ); Mon, 14 Feb 2022 12:44:35 -0500 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0FFB765483 for ; Mon, 14 Feb 2022 09:44:26 -0800 (PST) Received: from pps.filterd (m0148460.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 21ECHW46005573 for ; Mon, 14 Feb 2022 09:44:25 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=sk+j4IRk3PfnWSQ2NA03xb7dzT85yytAB7SgmygMQh8=; b=C9br4Imcr4Dz9XV03g5RrEW09KpRTM6o2Zr61B94ak/fsklep4gEDdvz3Qsrr3t0dy92 1ipwghG2/NZ9Y8ZmJu1geUHMy7bvZGWIMM3R5Q9D7DH5bGeOqBM7j6ctBJeNdPYaMe1q 4e8w7J6r9gy3r52RqJYP+EiUqn6oOEXkKLg= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3e7pxr25us-6 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 14 Feb 2022 09:44:25 -0800 Received: from twshared7634.08.ash8.facebook.com (2620:10d:c085:208::f) by mail.thefacebook.com (2620:10d:c085:21d::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.21; Mon, 14 Feb 2022 09:44:24 -0800 Received: by devvm225.atn0.facebook.com (Postfix, from userid 425415) id A818AABBD0FD; Mon, 14 Feb 2022 09:44:09 -0800 (PST) From: Stefan Roesch To: , , , CC: Subject: [PATCH v1 05/14] fs: split off __alloc_page_buffers function Date: Mon, 14 Feb 2022 09:43:54 -0800 Message-ID: <20220214174403.4147994-6-shr@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220214174403.4147994-1-shr@fb.com> References: <20220214174403.4147994-1-shr@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: TyluuNSRzxtdtarBOdTdkLpshj3ot3Im X-Proofpoint-GUID: TyluuNSRzxtdtarBOdTdkLpshj3ot3Im X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.816,Hydra:6.0.425,FMLib:17.11.62.513 definitions=2022-02-14_07,2022-02-14_03,2021-12-02_01 X-Proofpoint-Spam-Details: rule=fb_outbound_notspam policy=fb_outbound score=0 adultscore=0 bulkscore=0 mlxlogscore=999 priorityscore=1501 lowpriorityscore=0 phishscore=0 impostorscore=0 clxscore=1015 mlxscore=0 suspectscore=0 malwarescore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2201110000 definitions=main-2202140105 X-FB-Internal: deliver Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org This splits off the __alloc_page_buffers() function from the alloc_page_buffers_function(). In addition it adds a gfp_t parameter, so the caller can specify the allocation flags. Signed-off-by: Stefan Roesch Reported-by: kernel test robot Reported-by: kernel test robot Reported-by: kernel test robot --- fs/buffer.c | 37 ++++++++++++++++++++++--------------- 1 file changed, 22 insertions(+), 15 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index 6e6a69a12eed..a1986f95a39a 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -802,26 +802,13 @@ int remove_inode_buffers(struct inode *inode) return ret; } -/* - * Create the appropriate buffers when given a page for data area and - * the size of each buffer.. Use the bh->b_this_page linked list to - * follow the buffers created. Return NULL if unable to create more - * buffers. - * - * The retry flag is used to differentiate async IO (paging, swapping) - * which may not fail from ordinary buffer allocations. - */ -struct buffer_head *alloc_page_buffers(struct page *page, unsigned long size, - bool retry) +struct buffer_head *__alloc_page_buffers(struct page *page, unsigned long size, + gfp_t gfp) { struct buffer_head *bh, *head; - gfp_t gfp = GFP_NOFS | __GFP_ACCOUNT; long offset; struct mem_cgroup *memcg, *old_memcg; - if (retry) - gfp |= __GFP_NOFAIL; - /* The page lock pins the memcg */ memcg = page_memcg(page); old_memcg = set_active_memcg(memcg); @@ -859,6 +846,26 @@ struct buffer_head *alloc_page_buffers(struct page *page, unsigned long size, goto out; } + +/* + * Create the appropriate buffers when given a page for data area and + * the size of each buffer.. Use the bh->b_this_page linked list to + * follow the buffers created. Return NULL if unable to create more + * buffers. + * + * The retry flag is used to differentiate async IO (paging, swapping) + * which may not fail from ordinary buffer allocations. + */ +struct buffer_head *alloc_page_buffers(struct page *page, unsigned long size, + bool retry) +{ + gfp_t gfp = GFP_NOFS | __GFP_ACCOUNT; + + if (retry) + gfp |= __GFP_NOFAIL; + + return __alloc_page_buffers(page, size, gfp); +} EXPORT_SYMBOL_GPL(alloc_page_buffers); static inline void From patchwork Mon Feb 14 17:43:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Roesch X-Patchwork-Id: 12745939 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99010C433FE for ; Mon, 14 Feb 2022 17:44:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1354709AbiBNRpB (ORCPT ); Mon, 14 Feb 2022 12:45:01 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:34816 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1357233AbiBNRom (ORCPT ); Mon, 14 Feb 2022 12:44:42 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B96C365487 for ; Mon, 14 Feb 2022 09:44:34 -0800 (PST) Received: from pps.filterd (m0148461.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 21ECHRdV024089 for ; Mon, 14 Feb 2022 09:44:34 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=zdWhSChvqhF2fmfEaurlE4EGbKOQKMRzCLYDx/9AfHo=; b=qiIC9nO4C8g/b/yIJ0VqOVOTvbSaWfhF415m2rk4uv8/sZsGbM2vtYkwVfMv+ueVWRwV KNAdN27ymErA7rrpOutG5SnmnaSrqvbvg1i2F6pFiHTiKZBVZx+GEfxMTTi06OrjNI+o Cj8DRAP98GVYmi14RYoWw63XIpnALeDsepE= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3e7py4j44e-13 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 14 Feb 2022 09:44:34 -0800 Received: from twshared7634.08.ash8.facebook.com (2620:10d:c085:208::f) by mail.thefacebook.com (2620:10d:c085:21d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.21; Mon, 14 Feb 2022 09:44:25 -0800 Received: by devvm225.atn0.facebook.com (Postfix, from userid 425415) id ADF1DABBD0FF; Mon, 14 Feb 2022 09:44:09 -0800 (PST) From: Stefan Roesch To: , , , CC: Subject: [PATCH v1 06/14] fs: split off __create_empty_buffers function Date: Mon, 14 Feb 2022 09:43:55 -0800 Message-ID: <20220214174403.4147994-7-shr@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220214174403.4147994-1-shr@fb.com> References: <20220214174403.4147994-1-shr@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: JWspIi2Dc20FbFWuXHXAdcjWmmTCOKZu X-Proofpoint-GUID: JWspIi2Dc20FbFWuXHXAdcjWmmTCOKZu X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.816,Hydra:6.0.425,FMLib:17.11.62.513 definitions=2022-02-14_07,2022-02-14_03,2021-12-02_01 X-Proofpoint-Spam-Details: rule=fb_outbound_notspam policy=fb_outbound score=0 phishscore=0 clxscore=1015 spamscore=0 bulkscore=0 adultscore=0 priorityscore=1501 suspectscore=0 mlxlogscore=999 impostorscore=0 lowpriorityscore=0 mlxscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2201110000 definitions=main-2202140105 X-FB-Internal: deliver Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org This splits off the function __create_empty_buffers() from the function create_empty_buffers. The __create_empty_buffers has an additional gfp parameter. This allows the caller to specify the allocation properties. Signed-off-by: Stefan Roesch --- fs/buffer.c | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index a1986f95a39a..948505480b43 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -1554,17 +1554,12 @@ void block_invalidatepage(struct page *page, unsigned int offset, EXPORT_SYMBOL(block_invalidatepage); -/* - * We attach and possibly dirty the buffers atomically wrt - * __set_page_dirty_buffers() via private_lock. try_to_free_buffers - * is already excluded via the page lock. - */ -void create_empty_buffers(struct page *page, - unsigned long blocksize, unsigned long b_state) +static void __create_empty_buffers(struct page *page, unsigned long blocksize, + unsigned long b_state, gfp_t gfp) { struct buffer_head *bh, *head, *tail; - head = alloc_page_buffers(page, blocksize, true); + head = __alloc_page_buffers(page, blocksize, gfp); bh = head; do { bh->b_state |= b_state; @@ -1587,6 +1582,17 @@ void create_empty_buffers(struct page *page, attach_page_private(page, head); spin_unlock(&page->mapping->private_lock); } +/* + * We attach and possibly dirty the buffers atomically wrt + * __set_page_dirty_buffers() via private_lock. try_to_free_buffers + * is already excluded via the page lock. + */ +void create_empty_buffers(struct page *page, + unsigned long blocksize, unsigned long b_state) +{ + return __create_empty_buffers(page, blocksize, b_state, + GFP_NOFS | __GFP_ACCOUNT | __GFP_NOFAIL); +} EXPORT_SYMBOL(create_empty_buffers); /** From patchwork Mon Feb 14 17:43:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Roesch X-Patchwork-Id: 12745933 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EF4BAC43219 for ; Mon, 14 Feb 2022 17:44:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1357229AbiBNRol (ORCPT ); Mon, 14 Feb 2022 12:44:41 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:34908 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1357213AbiBNRoi (ORCPT ); Mon, 14 Feb 2022 12:44:38 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B7EA265489 for ; Mon, 14 Feb 2022 09:44:27 -0800 (PST) Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 21E9Zknk004107 for ; Mon, 14 Feb 2022 09:44:27 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=DK6Utv099ANrtOcaZ7fZtH5NvGZGKQSxBl8g8H8/nic=; b=lO4JWKs1kVRnRt6nmQ00sdvMxMDt6vsU1frFegMQiLpP2gjP/39Kfx8yrvg+iDUwYW+c V3rwV7LfRJSPV2rbtB2LeXcR4PsTKpLT4gTycdM+QX8fA7ejdaiiK/c3B+WU1q+Y4Kmj 93DZkuvtTlEUsw6AgvGHWloXlCsMkrzNaI8= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3e7mk82u9b-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 14 Feb 2022 09:44:27 -0800 Received: from twshared14630.35.frc1.facebook.com (2620:10d:c085:208::11) by mail.thefacebook.com (2620:10d:c085:11d::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.21; Mon, 14 Feb 2022 09:44:26 -0800 Received: by devvm225.atn0.facebook.com (Postfix, from userid 425415) id B44B6ABBD101; Mon, 14 Feb 2022 09:44:09 -0800 (PST) From: Stefan Roesch To: , , , CC: Subject: [PATCH v1 07/14] fs: Add aop_flags parameter to create_page_buffers() Date: Mon, 14 Feb 2022 09:43:56 -0800 Message-ID: <20220214174403.4147994-8-shr@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220214174403.4147994-1-shr@fb.com> References: <20220214174403.4147994-1-shr@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: RAE3TXYceUgMo-kwAnghmy0OFMh2FC6v X-Proofpoint-ORIG-GUID: RAE3TXYceUgMo-kwAnghmy0OFMh2FC6v X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.816,Hydra:6.0.425,FMLib:17.11.62.513 definitions=2022-02-14_07,2022-02-14_03,2021-12-02_01 X-Proofpoint-Spam-Details: rule=fb_outbound_notspam policy=fb_outbound score=0 mlxscore=0 priorityscore=1501 bulkscore=0 phishscore=0 impostorscore=0 adultscore=0 spamscore=0 malwarescore=0 suspectscore=0 clxscore=1015 mlxlogscore=938 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2201110000 definitions=main-2202140105 X-FB-Internal: deliver Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org This adds the aop_flags parameter to the create_page_buffers function. When AOP_FLAGS_NOWAIT parameter is set, the atomic allocation flag is set. The AOP_FLAGS_NOWAIT flag is set, when async buffered writes are enabled. Signed-off-by: Stefan Roesch --- fs/buffer.c | 28 +++++++++++++++++++++------- 1 file changed, 21 insertions(+), 7 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index 948505480b43..5e3067173580 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -1682,13 +1682,27 @@ static inline int block_size_bits(unsigned int blocksize) return ilog2(blocksize); } -static struct buffer_head *create_page_buffers(struct page *page, struct inode *inode, unsigned int b_state) +static struct buffer_head *create_page_buffers(struct page *page, + struct inode *inode, + unsigned int b_state, + unsigned int aop_flags) { BUG_ON(!PageLocked(page)); - if (!page_has_buffers(page)) - create_empty_buffers(page, 1 << READ_ONCE(inode->i_blkbits), - b_state); + if (!page_has_buffers(page)) { + gfp_t gfp = GFP_NOFS | __GFP_ACCOUNT; + + if (aop_flags & AOP_FLAGS_NOWAIT) { + gfp |= GFP_ATOMIC | __GFP_NOWARN; + gfp &= ~__GFP_DIRECT_RECLAIM; + } else { + gfp |= __GFP_NOFAIL; + } + + __create_empty_buffers(page, 1 << READ_ONCE(inode->i_blkbits), + b_state, gfp); + } + return page_buffers(page); } @@ -1734,7 +1748,7 @@ int __block_write_full_page(struct inode *inode, struct page *page, int write_flags = wbc_to_write_flags(wbc); head = create_page_buffers(page, inode, - (1 << BH_Dirty)|(1 << BH_Uptodate)); + (1 << BH_Dirty)|(1 << BH_Uptodate), 0); /* * Be very careful. We have no exclusion from __set_page_dirty_buffers @@ -2000,7 +2014,7 @@ int __block_write_begin_int(struct folio *folio, loff_t pos, unsigned len, BUG_ON(to > PAGE_SIZE); BUG_ON(from > to); - head = create_page_buffers(&folio->page, inode, 0); + head = create_page_buffers(&folio->page, inode, 0, flags); blocksize = head->b_size; bbits = block_size_bits(blocksize); @@ -2280,7 +2294,7 @@ int block_read_full_page(struct page *page, get_block_t *get_block) int nr, i; int fully_mapped = 1; - head = create_page_buffers(page, inode, 0); + head = create_page_buffers(page, inode, 0, 0); blocksize = head->b_size; bbits = block_size_bits(blocksize); From patchwork Mon Feb 14 17:43:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Roesch X-Patchwork-Id: 12745934 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C635BC4332F for ; Mon, 14 Feb 2022 17:44:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1357231AbiBNRom (ORCPT ); Mon, 14 Feb 2022 12:44:42 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:34906 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1357203AbiBNRoi (ORCPT ); Mon, 14 Feb 2022 12:44:38 -0500 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0B9A96548B for ; Mon, 14 Feb 2022 09:44:27 -0800 (PST) Received: from pps.filterd (m0148460.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 21ECHEOB005239 for ; Mon, 14 Feb 2022 09:44:27 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=NyHGbvRDPUnqdpOE6VaZfsgtyNPgunO2oZLXVjCcuJk=; b=LXKE5cZOW1sSeGRTkXphI9MF6ttdrSHbLt6EkJHPda1//FOL2GHqe4pss9zowQij0p8P aNhNxXwujFs0OMMiHs54VnMHV1qTxX8+aN9qcvKiggPgc3s0a0ziGccTh/4FD+vS6tNe phNG8EcKyXQkyaQkXjc0DtCafMZI8SJsR6U= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3e7pxr25wr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 14 Feb 2022 09:44:27 -0800 Received: from twshared7634.08.ash8.facebook.com (2620:10d:c085:208::11) by mail.thefacebook.com (2620:10d:c085:11d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.21; Mon, 14 Feb 2022 09:44:25 -0800 Received: by devvm225.atn0.facebook.com (Postfix, from userid 425415) id BACDEABBD103; Mon, 14 Feb 2022 09:44:09 -0800 (PST) From: Stefan Roesch To: , , , CC: Subject: [PATCH v1 08/14] fs: add support for async buffered writes Date: Mon, 14 Feb 2022 09:43:57 -0800 Message-ID: <20220214174403.4147994-9-shr@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220214174403.4147994-1-shr@fb.com> References: <20220214174403.4147994-1-shr@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: wXnkPJjWPxUBmyVGsscfP1dzwwaNgB_j X-Proofpoint-GUID: wXnkPJjWPxUBmyVGsscfP1dzwwaNgB_j X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.816,Hydra:6.0.425,FMLib:17.11.62.513 definitions=2022-02-14_07,2022-02-14_03,2021-12-02_01 X-Proofpoint-Spam-Details: rule=fb_outbound_notspam policy=fb_outbound score=0 adultscore=0 bulkscore=0 mlxlogscore=837 priorityscore=1501 lowpriorityscore=0 phishscore=0 impostorscore=0 clxscore=1015 mlxscore=0 suspectscore=0 malwarescore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2201110000 definitions=main-2202140105 X-FB-Internal: deliver Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org This adds support for the AOP_FLAGS_BUF_WASYNC flag to the fs layer. If a page that is required for writing is not in the page cache, it returns EAGAIN instead of ENOMEM. Signed-off-by: Stefan Roesch --- fs/buffer.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/fs/buffer.c b/fs/buffer.c index 5e3067173580..140f57c1cbdd 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -2069,6 +2069,10 @@ int __block_write_begin_int(struct folio *folio, loff_t pos, unsigned len, *wait_bh++=bh; } } + + /* No wait specified, don't wait for reads to complete. */ + if (!err && wait_bh > wait && (flags & AOP_FLAGS_NOWAIT)) + return -EAGAIN; /* * If we issued read requests - let them complete. */ @@ -2143,8 +2147,11 @@ int block_write_begin(struct address_space *mapping, loff_t pos, unsigned len, int status; page = grab_cache_page_write_begin(mapping, index, flags); - if (!page) + if (!page) { + if (flags & AOP_FLAGS_NOWAIT) + return -EAGAIN; return -ENOMEM; + } status = __block_write_begin_int(page_folio(page), pos, len, get_block, NULL, flags); if (unlikely(status)) { From patchwork Mon Feb 14 17:43:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Roesch X-Patchwork-Id: 12745943 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3DA26C433EF for ; Mon, 14 Feb 2022 17:45:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243993AbiBNRpQ (ORCPT ); Mon, 14 Feb 2022 12:45:16 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:35156 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1357251AbiBNRor (ORCPT ); Mon, 14 Feb 2022 12:44:47 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 60AF5654A3 for ; Mon, 14 Feb 2022 09:44:38 -0800 (PST) Received: from pps.filterd (m0148461.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 21ECHpcc024573 for ; Mon, 14 Feb 2022 09:44:38 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=WqJGTjlBszlWWKZYtp71MBllUosGp30baKNJ+u9prKQ=; b=eTFgI6MxxCJDd+hMmqvY/GQq8D+Mwsn1gF53kcc87W9Y2R6XfndmXJ0XK1CnPoCyB5KU DNx7xEAtXDVQwkx7NpdCf7biIWxbnWLu1GfyNi++WazJT5TBhANMjdZiFq/q8UG/t7vr FPMIH309p+sCJm+u3hDB3/XKWqdbHGhjYhw= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3e7py4j45q-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 14 Feb 2022 09:44:38 -0800 Received: from twshared22811.39.frc1.facebook.com (2620:10d:c085:208::f) by mail.thefacebook.com (2620:10d:c085:21d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.21; Mon, 14 Feb 2022 09:44:32 -0800 Received: by devvm225.atn0.facebook.com (Postfix, from userid 425415) id C0AFEABBD105; Mon, 14 Feb 2022 09:44:09 -0800 (PST) From: Stefan Roesch To: , , , CC: Subject: [PATCH v1 09/14] io_uring: add support for async buffered writes Date: Mon, 14 Feb 2022 09:43:58 -0800 Message-ID: <20220214174403.4147994-10-shr@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220214174403.4147994-1-shr@fb.com> References: <20220214174403.4147994-1-shr@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: _e8f0v-3sTFXFXkeZ5cwKYosBNKF-PyJ X-Proofpoint-GUID: _e8f0v-3sTFXFXkeZ5cwKYosBNKF-PyJ X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.816,Hydra:6.0.425,FMLib:17.11.62.513 definitions=2022-02-14_07,2022-02-14_03,2021-12-02_01 X-Proofpoint-Spam-Details: rule=fb_outbound_notspam policy=fb_outbound score=0 phishscore=0 clxscore=1015 spamscore=0 bulkscore=0 adultscore=0 priorityscore=1501 suspectscore=0 mlxlogscore=953 impostorscore=0 lowpriorityscore=0 mlxscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2201110000 definitions=main-2202140105 X-FB-Internal: deliver Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org This enables the async buffered writes for block devices in io_uring. Buffered writes are enabled for blocks that are already in the page cache or can be acquired with noio. It is possible that a write request cannot be completely fullfilled (short write). In that case the request is punted and sent to the io workers to be completed. Before submitting the request to the io workers, the request is updated with how much has already been written. Signed-off-by: Stefan Roesch --- fs/io_uring.c | 29 ++++++++++++++++++++++++----- 1 file changed, 24 insertions(+), 5 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index 2e04f718319d..76b1ff602470 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -3615,7 +3615,7 @@ static inline int io_iter_do_read(struct io_kiocb *req, struct iov_iter *iter) return -EINVAL; } -static bool need_read_all(struct io_kiocb *req) +static bool need_complete_io(struct io_kiocb *req) { return req->flags & REQ_F_ISREG || S_ISBLK(file_inode(req->file)->i_mode); @@ -3679,7 +3679,7 @@ static int io_read(struct io_kiocb *req, unsigned int issue_flags) } else if (ret == -EIOCBQUEUED) { goto out_free; } else if (ret == req->result || ret <= 0 || !force_nonblock || - (req->flags & REQ_F_NOWAIT) || !need_read_all(req)) { + (req->flags & REQ_F_NOWAIT) || !need_complete_io(req)) { /* read all, failed, already did sync or don't want to retry */ goto done; } @@ -3777,9 +3777,10 @@ static int io_write(struct io_kiocb *req, unsigned int issue_flags) if (unlikely(!io_file_supports_nowait(req))) goto copy_iov; - /* file path doesn't support NOWAIT for non-direct_IO */ - if (force_nonblock && !(kiocb->ki_flags & IOCB_DIRECT) && - (req->flags & REQ_F_ISREG)) + /* File path supports NOWAIT for non-direct_IO only for block devices. */ + if (!(kiocb->ki_flags & IOCB_DIRECT) && + !(kiocb->ki_filp->f_mode & FMODE_BUF_WASYNC) && + (req->flags & REQ_F_ISREG)) goto copy_iov; kiocb->ki_flags |= IOCB_NOWAIT; @@ -3831,6 +3832,24 @@ static int io_write(struct io_kiocb *req, unsigned int issue_flags) /* IOPOLL retry should happen for io-wq threads */ if (ret2 == -EAGAIN && (req->ctx->flags & IORING_SETUP_IOPOLL)) goto copy_iov; + + if (ret2 != req->result && ret2 >= 0 && need_complete_io(req)) { + struct io_async_rw *rw; + + /* This is a partial write. The file pos has already been + * updated, setup the async struct to complete the request + * in the worker. Also update bytes_done to account for + * the bytes already written. + */ + iov_iter_save_state(&s->iter, &s->iter_state); + ret = io_setup_async_rw(req, iovec, s, true); + + rw = req->async_data; + if (rw) + rw->bytes_done += ret2; + + return ret ? ret : -EAGAIN; + } done: kiocb_done(req, ret2, issue_flags); } else { From patchwork Mon Feb 14 17:43:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Roesch X-Patchwork-Id: 12745937 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A33C2C4321E for ; Mon, 14 Feb 2022 17:44:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1357189AbiBNRou (ORCPT ); Mon, 14 Feb 2022 12:44:50 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:34930 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1357224AbiBNRoj (ORCPT ); Mon, 14 Feb 2022 12:44:39 -0500 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7DAE565484 for ; Mon, 14 Feb 2022 09:44:31 -0800 (PST) Received: from pps.filterd (m0148460.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 21ECH7rj005103 for ; Mon, 14 Feb 2022 09:44:30 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=3ywzASqRXEEJ9XSq0ACPCijbM5X1A9Yxp6sQB3vKKIQ=; b=VuKwdu4B8gJ3TfFlk4Rf4oyf/WjffGYDORsZTeHCxuoNtqGrm9+W/mQcXY9t/VviW22i MLw/DbwdUSlIOJP4uVAjcXPbWWm5wqXOjUES3SSUxv/uAA6uJwo0q7eKtS41rPzemx07 Ivo1b66x7V6b56StF0T7g/JIjsaOxISg9Iw= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3e7pxr25xj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 14 Feb 2022 09:44:30 -0800 Received: from twshared9880.08.ash8.facebook.com (2620:10d:c085:208::f) by mail.thefacebook.com (2620:10d:c085:21d::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.21; Mon, 14 Feb 2022 09:44:29 -0800 Received: by devvm225.atn0.facebook.com (Postfix, from userid 425415) id C730AABBD107; Mon, 14 Feb 2022 09:44:09 -0800 (PST) From: Stefan Roesch To: , , , CC: Subject: [PATCH v1 10/14] io_uring: Add tracepoint for short writes Date: Mon, 14 Feb 2022 09:43:59 -0800 Message-ID: <20220214174403.4147994-11-shr@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220214174403.4147994-1-shr@fb.com> References: <20220214174403.4147994-1-shr@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: ot0CbUKUkvcaMWsH4ZkWIoczdBpdu9ql X-Proofpoint-GUID: ot0CbUKUkvcaMWsH4ZkWIoczdBpdu9ql X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.816,Hydra:6.0.425,FMLib:17.11.62.513 definitions=2022-02-14_07,2022-02-14_03,2021-12-02_01 X-Proofpoint-Spam-Details: rule=fb_outbound_notspam policy=fb_outbound score=0 adultscore=0 bulkscore=0 mlxlogscore=999 priorityscore=1501 lowpriorityscore=0 phishscore=0 impostorscore=0 clxscore=1015 mlxscore=0 suspectscore=0 malwarescore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2201110000 definitions=main-2202140105 X-FB-Internal: deliver Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org This adds the io_uring_short_write tracepoint to io_uring. A short write is issued if not all pages that are required for a write are in the page cache and the async buffered writes have to return EAGAIN. Signed-off-by: Stefan Roesch --- fs/io_uring.c | 3 +++ include/trace/events/io_uring.h | 25 +++++++++++++++++++++++++ 2 files changed, 28 insertions(+) diff --git a/fs/io_uring.c b/fs/io_uring.c index 76b1ff602470..507f28b5b2bb 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -3836,6 +3836,9 @@ static int io_write(struct io_kiocb *req, unsigned int issue_flags) if (ret2 != req->result && ret2 >= 0 && need_complete_io(req)) { struct io_async_rw *rw; + trace_io_uring_short_write(req->ctx, kiocb->ki_pos - ret2, + req->result, ret2); + /* This is a partial write. The file pos has already been * updated, setup the async struct to complete the request * in the worker. Also update bytes_done to account for diff --git a/include/trace/events/io_uring.h b/include/trace/events/io_uring.h index 7346f0164cf4..ce1cfdf4b015 100644 --- a/include/trace/events/io_uring.h +++ b/include/trace/events/io_uring.h @@ -558,6 +558,31 @@ TRACE_EVENT(io_uring_req_failed, (unsigned long long) __entry->pad2, __entry->error) ); +TRACE_EVENT(io_uring_short_write, + + TP_PROTO(void *ctx, u64 fpos, u64 wanted, u64 got), + + TP_ARGS(ctx, fpos, wanted, got), + + TP_STRUCT__entry( + __field(void *, ctx) + __field(u64, fpos) + __field(u64, wanted) + __field(u64, got) + ), + + TP_fast_assign( + __entry->ctx = ctx; + __entry->fpos = fpos; + __entry->wanted = wanted; + __entry->got = got; + ), + + TP_printk("ring %p, fpos %lld, wanted %lld, got %lld", + __entry->ctx, __entry->fpos, + __entry->wanted, __entry->got) +); + #endif /* _TRACE_IO_URING_H */ /* This part must be outside protection */ From patchwork Mon Feb 14 17:44:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Roesch X-Patchwork-Id: 12745940 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8364BC43217 for ; Mon, 14 Feb 2022 17:44:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1357330AbiBNRpC (ORCPT ); Mon, 14 Feb 2022 12:45:02 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:35022 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1357239AbiBNRon (ORCPT ); Mon, 14 Feb 2022 12:44:43 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EE5676548E for ; Mon, 14 Feb 2022 09:44:35 -0800 (PST) Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 21E9ZrwI004219 for ; Mon, 14 Feb 2022 09:44:35 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=PrMhbqwOga++xb/Ni5mVt/2MYejdr64cMoOG2i2n2Bc=; b=f3065dX9Wvo7rDZcaMXO+dR/bs3k5q3Fb+DEHplZ1K2UGXhXY6/soCbrFFnvgxqjhsx8 FKmu0flFTG38bXys011JtFNREu2y6fTTr7NGJqBb/pY9b6NtX2dvm3v/FP5FKV6XstRC DD37MSHLvoJb6u73nMWXOtTQ/m+D6mbDG4k= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3e7mk82uaf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 14 Feb 2022 09:44:35 -0800 Received: from twshared19733.18.frc3.facebook.com (2620:10d:c085:108::4) by mail.thefacebook.com (2620:10d:c085:21d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.21; Mon, 14 Feb 2022 09:44:34 -0800 Received: by devvm225.atn0.facebook.com (Postfix, from userid 425415) id CCA3CABBD109; Mon, 14 Feb 2022 09:44:09 -0800 (PST) From: Stefan Roesch To: , , , CC: Subject: [PATCH v1 11/14] sched: add new fields to task_struct Date: Mon, 14 Feb 2022 09:44:00 -0800 Message-ID: <20220214174403.4147994-12-shr@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220214174403.4147994-1-shr@fb.com> References: <20220214174403.4147994-1-shr@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: N-yVgQXC258NTfT5vR9ILNFf3JLh9AZx X-Proofpoint-ORIG-GUID: N-yVgQXC258NTfT5vR9ILNFf3JLh9AZx X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.816,Hydra:6.0.425,FMLib:17.11.62.513 definitions=2022-02-14_07,2022-02-14_03,2021-12-02_01 X-Proofpoint-Spam-Details: rule=fb_outbound_notspam policy=fb_outbound score=0 mlxscore=0 priorityscore=1501 bulkscore=0 phishscore=0 impostorscore=0 adultscore=0 spamscore=0 malwarescore=0 suspectscore=0 clxscore=1015 mlxlogscore=999 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2201110000 definitions=main-2202140105 X-FB-Internal: deliver Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Add two new fields to the task_struct to support async write throttling. - One field to store how long writes are throttled: bdp_pause - The other field to store the number of dirtied pages: bdp_nr_dirtied_pause Signed-off-by: Stefan Roesch --- include/linux/sched.h | 3 +++ kernel/fork.c | 1 + 2 files changed, 4 insertions(+) diff --git a/include/linux/sched.h b/include/linux/sched.h index 75ba8aa60248..97146b7539c5 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1324,6 +1324,9 @@ struct task_struct { /* Start of a write-and-pause period: */ unsigned long dirty_paused_when; + unsigned long bdp_pause; + int bdp_nr_dirtied_pause; + #ifdef CONFIG_LATENCYTOP int latency_record_count; struct latency_record latency_record[LT_SAVECOUNT]; diff --git a/kernel/fork.c b/kernel/fork.c index d75a528f7b21..d34c9c00baea 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -2246,6 +2246,7 @@ static __latent_entropy struct task_struct *copy_process( p->nr_dirtied = 0; p->nr_dirtied_pause = 128 >> (PAGE_SHIFT - 10); p->dirty_paused_when = 0; + p->bdp_nr_dirtied_pause = -1; p->pdeath_signal = 0; INIT_LIST_HEAD(&p->thread_group); From patchwork Mon Feb 14 17:44:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Roesch X-Patchwork-Id: 12745942 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B872DC433EF for ; Mon, 14 Feb 2022 17:45:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1357249AbiBNRpO (ORCPT ); Mon, 14 Feb 2022 12:45:14 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:34924 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1357248AbiBNRor (ORCPT ); Mon, 14 Feb 2022 12:44:47 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 949DC6549B for ; Mon, 14 Feb 2022 09:44:37 -0800 (PST) Received: from pps.filterd (m0148461.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 21ECHRdb024089 for ; Mon, 14 Feb 2022 09:44:37 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=LHVvoEZfE6JaHZroD1k0ORc9e9YEoF8oOTWuN+537Qc=; b=m8IUD7cIb0bY1zrxnNi/Wp7ND5BOIKoqg2fLChxD1KbWdy4R8eKjp03nsRoywYY5ccC9 aovmYfV1fYvtRyyTvncJ+7FM4WYupFi7gW/JB7NqLu4D5bI3btoKl1EwLDptwkLkwcnP XLym/5qbBGRfzYEgTHu0kZK1RUSUSmArQ+0= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3e7py4j44e-19 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 14 Feb 2022 09:44:37 -0800 Received: from twshared7634.08.ash8.facebook.com (2620:10d:c085:208::f) by mail.thefacebook.com (2620:10d:c085:21d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.21; Mon, 14 Feb 2022 09:44:29 -0800 Received: by devvm225.atn0.facebook.com (Postfix, from userid 425415) id D2FC8ABBD10B; Mon, 14 Feb 2022 09:44:09 -0800 (PST) From: Stefan Roesch To: , , , CC: Subject: [PATCH v1 12/14] mm: support write throttling for async buffered writes Date: Mon, 14 Feb 2022 09:44:01 -0800 Message-ID: <20220214174403.4147994-13-shr@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220214174403.4147994-1-shr@fb.com> References: <20220214174403.4147994-1-shr@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: -82GtROvFUo3LpZQpQinlX4dTOAYlLSP X-Proofpoint-GUID: -82GtROvFUo3LpZQpQinlX4dTOAYlLSP X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.816,Hydra:6.0.425,FMLib:17.11.62.513 definitions=2022-02-14_07,2022-02-14_03,2021-12-02_01 X-Proofpoint-Spam-Details: rule=fb_outbound_notspam policy=fb_outbound score=0 phishscore=0 clxscore=1015 spamscore=0 bulkscore=0 adultscore=0 priorityscore=1501 suspectscore=0 mlxlogscore=999 impostorscore=0 lowpriorityscore=0 mlxscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2201110000 definitions=main-2202140105 X-FB-Internal: deliver Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org This change adds support for async write throttling in the function balance_dirty_pages(). So far if throttling was required, the code was waiting synchronously as long as the writes were throttled. This change introduces asynchronous throttling. Instead of waiting in the function balance_dirty_pages(), the timeout is set in the task_struct field bdp_pause. Once the timeout has expired, the writes are no longer throttled. - Add a new parameter to the balance_dirty_pages() function - This allows the caller to pass in the nowait flag - When the nowait flag is specified, the code does not wait in balance_dirty_pages(), but instead stores the wait expiration in the new task_struct field bdp_pause. - The function balance_dirty_pages_ratelimited() resets the new values in the task_struct, once the timeout has expired This change is required to support write throttling for the async buffered writes. While the writes are throttled, io_uring still can make progress with processing other requests. Signed-off-by: Stefan Roesch --- include/linux/writeback.h | 1 + mm/filemap.c | 2 +- mm/page-writeback.c | 54 ++++++++++++++++++++++++++++----------- 3 files changed, 41 insertions(+), 16 deletions(-) diff --git a/include/linux/writeback.h b/include/linux/writeback.h index fec248ab1fec..48176a8047db 100644 --- a/include/linux/writeback.h +++ b/include/linux/writeback.h @@ -373,6 +373,7 @@ unsigned long wb_calc_thresh(struct bdi_writeback *wb, unsigned long thresh); void wb_update_bandwidth(struct bdi_writeback *wb); void balance_dirty_pages_ratelimited(struct address_space *mapping); +void balance_dirty_pages_ratelimited_flags(struct address_space *mapping, bool is_async); bool wb_over_bg_thresh(struct bdi_writeback *wb); typedef int (*writepage_t)(struct page *page, struct writeback_control *wbc, diff --git a/mm/filemap.c b/mm/filemap.c index 19065ad95a4c..aa51ff1a0e8f 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3805,7 +3805,7 @@ static ssize_t do_generic_perform_write(struct file *file, struct iov_iter *i, pos += status; written += status; - balance_dirty_pages_ratelimited(mapping); + balance_dirty_pages_ratelimited_flags(mapping, flags & AOP_FLAGS_NOWAIT); } while (iov_iter_count(i)); return written ? written : status; diff --git a/mm/page-writeback.c b/mm/page-writeback.c index 91d163f8d36b..767d0b997da5 100644 --- a/mm/page-writeback.c +++ b/mm/page-writeback.c @@ -1558,7 +1558,7 @@ static inline void wb_dirty_limits(struct dirty_throttle_control *dtc) * perform some writeout. */ static void balance_dirty_pages(struct bdi_writeback *wb, - unsigned long pages_dirtied) + unsigned long pages_dirtied, bool is_async) { struct dirty_throttle_control gdtc_stor = { GDTC_INIT(wb) }; struct dirty_throttle_control mdtc_stor = { MDTC_INIT(wb, &gdtc_stor) }; @@ -1792,6 +1792,14 @@ static void balance_dirty_pages(struct bdi_writeback *wb, period, pause, start_time); + if (is_async) { + if (current->bdp_nr_dirtied_pause == -1) { + current->bdp_pause = now + pause; + current->bdp_nr_dirtied_pause = nr_dirtied_pause; + } + break; + } + __set_current_state(TASK_KILLABLE); wb->dirty_sleep = now; io_schedule_timeout(pause); @@ -1799,6 +1807,8 @@ static void balance_dirty_pages(struct bdi_writeback *wb, current->dirty_paused_when = now + pause; current->nr_dirtied = 0; current->nr_dirtied_pause = nr_dirtied_pause; + current->bdp_nr_dirtied_pause = -1; + current->bdp_pause = 0; /* * This is typically equal to (dirty < thresh) and can also @@ -1863,19 +1873,7 @@ static DEFINE_PER_CPU(int, bdp_ratelimits); */ DEFINE_PER_CPU(int, dirty_throttle_leaks) = 0; -/** - * balance_dirty_pages_ratelimited - balance dirty memory state - * @mapping: address_space which was dirtied - * - * Processes which are dirtying memory should call in here once for each page - * which was newly dirtied. The function will periodically check the system's - * dirty state and will initiate writeback if needed. - * - * Once we're over the dirty memory limit we decrease the ratelimiting - * by a lot, to prevent individual processes from overshooting the limit - * by (ratelimit_pages) each. - */ -void balance_dirty_pages_ratelimited(struct address_space *mapping) +void balance_dirty_pages_ratelimited_flags(struct address_space *mapping, bool is_async) { struct inode *inode = mapping->host; struct backing_dev_info *bdi = inode_to_bdi(inode); @@ -1886,6 +1884,15 @@ void balance_dirty_pages_ratelimited(struct address_space *mapping) if (!(bdi->capabilities & BDI_CAP_WRITEBACK)) return; + if (current->bdp_nr_dirtied_pause != -1 && time_after(jiffies, current->bdp_pause)) { + current->dirty_paused_when = current->bdp_pause; + current->nr_dirtied = 0; + current->nr_dirtied_pause = current->bdp_nr_dirtied_pause; + + current->bdp_nr_dirtied_pause = -1; + current->bdp_pause = 0; + } + if (inode_cgwb_enabled(inode)) wb = wb_get_create_current(bdi, GFP_KERNEL); if (!wb) @@ -1924,10 +1931,27 @@ void balance_dirty_pages_ratelimited(struct address_space *mapping) preempt_enable(); if (unlikely(current->nr_dirtied >= ratelimit)) - balance_dirty_pages(wb, current->nr_dirtied); + balance_dirty_pages(wb, current->nr_dirtied, is_async); wb_put(wb); } + +/** + * balance_dirty_pages_ratelimited - balance dirty memory state + * @mapping: address_space which was dirtied + * + * Processes which are dirtying memory should call in here once for each page + * which was newly dirtied. The function will periodically check the system's + * dirty state and will initiate writeback if needed. + * + * Once we're over the dirty memory limit we decrease the ratelimiting + * by a lot, to prevent individual processes from overshooting the limit + * by (ratelimit_pages) each. + */ +void balance_dirty_pages_ratelimited(struct address_space *mapping) +{ + balance_dirty_pages_ratelimited_flags(mapping, false); +} EXPORT_SYMBOL(balance_dirty_pages_ratelimited); /** From patchwork Mon Feb 14 17:44:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Roesch X-Patchwork-Id: 12745936 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 545C3C43217 for ; Mon, 14 Feb 2022 17:44:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1357256AbiBNRot (ORCPT ); Mon, 14 Feb 2022 12:44:49 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:34928 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1357223AbiBNRoj (ORCPT ); Mon, 14 Feb 2022 12:44:39 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8D09A65494 for ; Mon, 14 Feb 2022 09:44:30 -0800 (PST) Received: from pps.filterd (m0109333.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 21E6Hehb026724 for ; Mon, 14 Feb 2022 09:44:30 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=KoRVs42BPo4ThyQIGEACMhtxksoiHz76C/WJo1qdZ/g=; b=k2RfwbAsSfCHjpIygCGQbQ2XY5LPB1TchOIKqvBaqTHLcLMMll1X9BGrxodR15UfVbGL LdWHBETia6Lj2o5b3aiZwi76dxbeIpRlbNW9dd4UahoXDwqj2tJTWqWFA31S420gq70k Cs4/E2RmHZBP1E8GoeGnSBsMCN1lRPyG5vo= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3e7hpcbjvt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 14 Feb 2022 09:44:30 -0800 Received: from twshared9880.08.ash8.facebook.com (2620:10d:c085:108::4) by mail.thefacebook.com (2620:10d:c085:11d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.21; Mon, 14 Feb 2022 09:44:29 -0800 Received: by devvm225.atn0.facebook.com (Postfix, from userid 425415) id D9C9FABBD10D; Mon, 14 Feb 2022 09:44:09 -0800 (PST) From: Stefan Roesch To: , , , CC: Subject: [PATCH v1 13/14] io_uring: support write throttling for async buffered writes Date: Mon, 14 Feb 2022 09:44:02 -0800 Message-ID: <20220214174403.4147994-14-shr@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220214174403.4147994-1-shr@fb.com> References: <20220214174403.4147994-1-shr@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: tyKaEPfg_kTUeomgTO_0KtN9rPwrI_pg X-Proofpoint-ORIG-GUID: tyKaEPfg_kTUeomgTO_0KtN9rPwrI_pg X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.816,Hydra:6.0.425,FMLib:17.11.62.513 definitions=2022-02-14_07,2022-02-14_03,2021-12-02_01 X-Proofpoint-Spam-Details: rule=fb_outbound_notspam policy=fb_outbound score=0 mlxscore=0 phishscore=0 clxscore=1015 suspectscore=0 adultscore=0 mlxlogscore=999 lowpriorityscore=0 malwarescore=0 impostorscore=0 spamscore=0 priorityscore=1501 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2201110000 definitions=main-2202140105 X-FB-Internal: deliver Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org This adds the process-level throttling for the block layer for async buffered writes to io-uring.In io_write the code now checks if the write needs to be throttled. If this is required, it adds the request to the list of pending io requests and starts a timer. After the timer expires, it submits the list of pending writes. - Add new list called pending_ios for delayed writes (throttled writes) to struct io_uring_task. The list is protected by the task_lock spin lock. - Add new timer to struct io_uring_task. Signed-off-by: Stefan Roesch --- fs/io_uring.c | 98 +++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 91 insertions(+), 7 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index 507f28b5b2bb..7bb77700ffac 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -461,6 +461,11 @@ struct io_ring_ctx { }; }; +struct pending_list { + struct list_head list; + struct io_kiocb *req; +}; + struct io_uring_task { /* submission side */ int cached_refs; @@ -477,6 +482,9 @@ struct io_uring_task { struct io_wq_work_list prior_task_list; struct callback_head task_work; bool task_running; + + struct pending_list pending_ios; + struct timer_list timer; }; /* @@ -1134,13 +1142,14 @@ static void io_rsrc_put_work(struct work_struct *work); static void io_req_task_queue(struct io_kiocb *req); static void __io_submit_flush_completions(struct io_ring_ctx *ctx); -static int io_req_prep_async(struct io_kiocb *req); +static int io_req_prep_async(struct io_kiocb *req, bool force); static int io_install_fixed_file(struct io_kiocb *req, struct file *file, unsigned int issue_flags, u32 slot_index); static int io_close_fixed(struct io_kiocb *req, unsigned int issue_flags); static enum hrtimer_restart io_link_timeout_fn(struct hrtimer *timer); +static void delayed_write_fn(struct timer_list *tmr); static struct kmem_cache *req_cachep; @@ -2462,6 +2471,31 @@ static void io_req_task_queue_reissue(struct io_kiocb *req) io_req_task_work_add(req, false); } +static int io_req_task_queue_reissue_delayed(struct io_kiocb *req) +{ + struct io_uring_task *tctx = req->task->io_uring; + struct pending_list *pending = kmalloc(sizeof(struct pending_list), GFP_KERNEL); + bool empty; + + if (!pending) + return -ENOMEM; + pending->req = req; + + spin_lock_irq(&tctx->task_lock); + empty = list_empty(&tctx->pending_ios.list); + list_add_tail(&pending->list, &tctx->pending_ios.list); + + if (empty) { + timer_setup(&tctx->timer, delayed_write_fn, 0); + + tctx->timer.expires = current->bdp_pause; + add_timer(&tctx->timer); + } + spin_unlock_irq(&tctx->task_lock); + + return 0; +} + static inline void io_queue_next(struct io_kiocb *req) { struct io_kiocb *nxt = io_req_find_next(req); @@ -2770,7 +2804,7 @@ static bool io_resubmit_prep(struct io_kiocb *req) struct io_async_rw *rw = req->async_data; if (!req_has_async_data(req)) - return !io_req_prep_async(req); + return !io_req_prep_async(req, false); iov_iter_restore(&rw->s.iter, &rw->s.iter_state); return true; } @@ -3751,6 +3785,38 @@ static int io_write_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) return io_prep_rw(req, sqe); } +static inline unsigned long write_delay(void) +{ + if (likely(current->bdp_nr_dirtied_pause == -1 || + !time_before(jiffies, current->bdp_pause))) + return 0; + + return current->bdp_pause; +} + +static void delayed_write_fn(struct timer_list *tmr) +{ + struct io_uring_task *tctx = from_timer(tctx, tmr, timer); + struct list_head *curr; + struct list_head *next; + LIST_HEAD(pending_ios); + + /* Move list to temporary list. */ + spin_lock_irq(&tctx->task_lock); + list_splice_init(&tctx->pending_ios.list, &pending_ios); + spin_unlock_irq(&tctx->task_lock); + + list_for_each_safe(curr, next, &pending_ios) { + struct pending_list *io; + + io = list_entry(curr, struct pending_list, list); + io_req_task_queue_reissue(io->req); + + list_del(curr); + kfree(io); + } +} + static int io_write(struct io_kiocb *req, unsigned int issue_flags) { struct io_rw_state __s, *s = &__s; @@ -3759,6 +3825,18 @@ static int io_write(struct io_kiocb *req, unsigned int issue_flags) bool force_nonblock = issue_flags & IO_URING_F_NONBLOCK; ssize_t ret, ret2; + /* Write throttling active? */ + if (unlikely(write_delay()) && !(kiocb->ki_flags & IOCB_DIRECT)) { + int ret = io_req_prep_async(req, true); + + if (unlikely(ret)) + io_req_complete_failed(req, ret); + else + ret = io_req_task_queue_reissue_delayed(req); + + return ret; + } + if (!req_has_async_data(req)) { ret = io_import_iovec(WRITE, req, &iovec, s, issue_flags); if (unlikely(ret < 0)) @@ -6597,9 +6675,9 @@ static int io_req_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) return -EINVAL; } -static int io_req_prep_async(struct io_kiocb *req) +static int io_req_prep_async(struct io_kiocb *req, bool force) { - if (!io_op_defs[req->opcode].needs_async_setup) + if (!force && !io_op_defs[req->opcode].needs_async_setup) return 0; if (WARN_ON_ONCE(req_has_async_data(req))) return -EFAULT; @@ -6609,6 +6687,10 @@ static int io_req_prep_async(struct io_kiocb *req) switch (req->opcode) { case IORING_OP_READV: return io_rw_prep_async(req, READ); + case IORING_OP_WRITE: + if (!force) + break; + fallthrough; case IORING_OP_WRITEV: return io_rw_prep_async(req, WRITE); case IORING_OP_SENDMSG: @@ -6618,6 +6700,7 @@ static int io_req_prep_async(struct io_kiocb *req) case IORING_OP_CONNECT: return io_connect_prep_async(req); } + printk_once(KERN_WARNING "io_uring: prep_async() bad opcode %d\n", req->opcode); return -EFAULT; @@ -6651,7 +6734,7 @@ static __cold void io_drain_req(struct io_kiocb *req) } spin_unlock(&ctx->completion_lock); - ret = io_req_prep_async(req); + ret = io_req_prep_async(req, false); if (ret) { fail: io_req_complete_failed(req, ret); @@ -7146,7 +7229,7 @@ static void io_queue_sqe_fallback(struct io_kiocb *req) } else if (unlikely(req->ctx->drain_active)) { io_drain_req(req); } else { - int ret = io_req_prep_async(req); + int ret = io_req_prep_async(req, false); if (unlikely(ret)) io_req_complete_failed(req, ret); @@ -7345,7 +7428,7 @@ static int io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req, struct io_kiocb *head = link->head; if (!(req->flags & REQ_F_FAIL)) { - ret = io_req_prep_async(req); + ret = io_req_prep_async(req, false); if (unlikely(ret)) { req_fail_link_node(req, ret); if (!(head->flags & REQ_F_FAIL)) @@ -8785,6 +8868,7 @@ static __cold int io_uring_alloc_task_context(struct task_struct *task, INIT_WQ_LIST(&tctx->task_list); INIT_WQ_LIST(&tctx->prior_task_list); init_task_work(&tctx->task_work, tctx_task_work); + INIT_LIST_HEAD(&tctx->pending_ios.list); return 0; } From patchwork Mon Feb 14 17:44:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Roesch X-Patchwork-Id: 12745941 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C6C1C433EF for ; Mon, 14 Feb 2022 17:44:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242977AbiBNRpE (ORCPT ); Mon, 14 Feb 2022 12:45:04 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:34816 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1355937AbiBNRpA (ORCPT ); Mon, 14 Feb 2022 12:45:00 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 22422654AB for ; Mon, 14 Feb 2022 09:44:39 -0800 (PST) Received: from pps.filterd (m0148461.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 21ECHpcf024573 for ; Mon, 14 Feb 2022 09:44:38 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=64+ajZhnUuMl/qgmBc/VwtWMjnpE/622h5Tkk0U7sHs=; b=PDvXTN981j8YyoyPJZ51ZMI/ToVCqF8GJ+X7gPtjfeKEUsf6iafpL8cq4oO8CEv0PveY WUbv2Rg1b3uuSg8188Fg6gAYgx/eEbSfSsXenchMz1W+5tq59TVWYhPimg41nTKdw0Z2 23yysLSmEJ+c3Qy0ec3Is3bof+gCZ/9nnco= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3e7py4j45q-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 14 Feb 2022 09:44:38 -0800 Received: from twshared7634.08.ash8.facebook.com (2620:10d:c085:208::f) by mail.thefacebook.com (2620:10d:c085:21d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.21; Mon, 14 Feb 2022 09:44:34 -0800 Received: by devvm225.atn0.facebook.com (Postfix, from userid 425415) id DFE22ABBD10F; Mon, 14 Feb 2022 09:44:09 -0800 (PST) From: Stefan Roesch To: , , , CC: Subject: [PATCH v1 14/14] block: enable async buffered writes for block devices. Date: Mon, 14 Feb 2022 09:44:03 -0800 Message-ID: <20220214174403.4147994-15-shr@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220214174403.4147994-1-shr@fb.com> References: <20220214174403.4147994-1-shr@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: iFYzVwhdjceccEAA2GndbqOBiFJmh_ty X-Proofpoint-GUID: iFYzVwhdjceccEAA2GndbqOBiFJmh_ty X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.816,Hydra:6.0.425,FMLib:17.11.62.513 definitions=2022-02-14_07,2022-02-14_03,2021-12-02_01 X-Proofpoint-Spam-Details: rule=fb_outbound_notspam policy=fb_outbound score=0 phishscore=0 clxscore=1015 spamscore=0 bulkscore=0 adultscore=0 priorityscore=1501 suspectscore=0 mlxlogscore=815 impostorscore=0 lowpriorityscore=0 mlxscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2201110000 definitions=main-2202140105 X-FB-Internal: deliver Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org This introduces the flag FMODE_BUF_WASYNC. If devices support async buffered writes, this flag can be set. It also enables async buffered writes for block devices. Signed-off-by: Stefan Roesch --- block/fops.c | 5 +---- fs/read_write.c | 3 ++- include/linux/fs.h | 3 +++ 3 files changed, 6 insertions(+), 5 deletions(-) diff --git a/block/fops.c b/block/fops.c index 4f59e0f5bf30..75b36f8b5e71 100644 --- a/block/fops.c +++ b/block/fops.c @@ -489,7 +489,7 @@ static int blkdev_open(struct inode *inode, struct file *filp) * during an unstable branch. */ filp->f_flags |= O_LARGEFILE; - filp->f_mode |= FMODE_NOWAIT | FMODE_BUF_RASYNC; + filp->f_mode |= FMODE_NOWAIT | FMODE_BUF_RASYNC | FMODE_BUF_WASYNC; if (filp->f_flags & O_NDELAY) filp->f_mode |= FMODE_NDELAY; @@ -544,9 +544,6 @@ static ssize_t blkdev_write_iter(struct kiocb *iocb, struct iov_iter *from) if (iocb->ki_pos >= size) return -ENOSPC; - if ((iocb->ki_flags & (IOCB_NOWAIT | IOCB_DIRECT)) == IOCB_NOWAIT) - return -EOPNOTSUPP; - size -= iocb->ki_pos; if (iov_iter_count(from) > size) { shorted = iov_iter_count(from) - size; diff --git a/fs/read_write.c b/fs/read_write.c index 0074afa7ecb3..58233844a9d8 100644 --- a/fs/read_write.c +++ b/fs/read_write.c @@ -1641,7 +1641,8 @@ ssize_t generic_write_checks(struct kiocb *iocb, struct iov_iter *from) if (iocb->ki_flags & IOCB_APPEND) iocb->ki_pos = i_size_read(inode); - if ((iocb->ki_flags & IOCB_NOWAIT) && !(iocb->ki_flags & IOCB_DIRECT)) + if ((iocb->ki_flags & IOCB_NOWAIT) && + (!(iocb->ki_flags & IOCB_DIRECT) && !(file->f_mode & FMODE_BUF_WASYNC))) return -EINVAL; count = iov_iter_count(from); diff --git a/include/linux/fs.h b/include/linux/fs.h index e62dba6ed453..a19c7903e031 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -176,6 +176,9 @@ typedef int (dio_iodone_t)(struct kiocb *iocb, loff_t offset, /* File supports async buffered reads */ #define FMODE_BUF_RASYNC ((__force fmode_t)0x40000000) +/* File supports async nowait buffered writes */ +#define FMODE_BUF_WASYNC ((__force fmode_t)0x80000000) + /* * Attribute flags. These should be or-ed together to figure out what * has been changed!