From patchwork Mon Jul 21 06:40:44 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 4593501 Return-Path: X-Original-To: patchwork-ceph-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 85965C0514 for ; Mon, 21 Jul 2014 06:41:12 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 82D9520103 for ; Mon, 21 Jul 2014 06:41:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id B7287200F3 for ; Mon, 21 Jul 2014 06:41:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752522AbaGUGlA (ORCPT ); Mon, 21 Jul 2014 02:41:00 -0400 Received: from cantor2.suse.de ([195.135.220.15]:42664 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750883AbaGUGk7 (ORCPT ); Mon, 21 Jul 2014 02:40:59 -0400 Received: from relay2.suse.de (charybdis-ext.suse.de [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id 6282EAB1E; Mon, 21 Jul 2014 06:40:57 +0000 (UTC) Date: Mon, 21 Jul 2014 16:40:44 +1000 From: NeilBrown To: Milosz Tanski Cc: ceph-devel , "linux-fsdevel@vger.kernel.org" , "linux-cachefs@redhat.com" , David Howells Subject: Re: fscache recursive hang -- similar to loopback NFS issues Message-ID: <20140721164044.2845f3fd@notabene.brown> In-Reply-To: References: X-Mailer: Claws Mail 3.10.1-123-gae895c (GTK+ 2.24.22; x86_64-suse-linux-gnu) MIME-Version: 1.0 Sender: ceph-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, T_TVD_MIME_EPI, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On Sat, 19 Jul 2014 16:20:01 -0400 Milosz Tanski wrote: > Neil, > > I saw your recent patcheset for improving the wait_on_bit interface > (particular: SCHED: allow wait_on_bit_action functions to support a > timeout.) I'm looking on some guidance on leveraging that work to > solve other recursive lock hang in fscache. > > I've ran into similar issues you're trying to solve with loopback NFS > but in the fscache code. This happens under heavy vma preasure when > the kernel is aggressively trying to trim the page cache. > > The hang is caused by this serious of events > 1. cachefiles_write_page - cachefiles (the fscache backend, sitting on > ext4) tries to write page to disk > 2. ext4 tries to allocate a page in writeback (without GPF_NOFS and > with wait flag) > 3. due to vma preasure the kernel tries to free-up pages > 4. this causes release pages in ceph to be called > 5. the selected page is cached page in process of write out (from step #1) > 6. fscache_wait_on_page_write hangs forever > > Is there a solution that you have to NFS as another patch that > implements the timeout that I can use a template? I'm not familiar > with that piece of the code base. It looks like the comment in __fscache_maybe_release_page /* We will wait here if we're allowed to, but that could deadlock the * allocator as the work threads writing to the cache may all end up * sleeping on memory allocation, so we may need to impose a timeout * too. */ is correct when it says "we may need to impose a timeout". The following __fscache_wait_on_page_write() needs to timeout. However that doesn't use wait_on_bit(), it just has a simple wait_event. So something like this should fix it (or should at least move the problem along a bit). NeilBrown diff --git a/fs/fscache/page.c b/fs/fscache/page.c index ed70714503fa..58035024c5cf 100644 --- a/fs/fscache/page.c +++ b/fs/fscache/page.c @@ -43,6 +43,13 @@ void __fscache_wait_on_page_write(struct fscache_cookie *cookie, struct page *pa } EXPORT_SYMBOL(__fscache_wait_on_page_write); +void __fscache_wait_on_page_write_timeout(struct fscache_cookie *cookie, struct page *page, unsigned long timeout) +{ + wait_queue_head_t *wq = bit_waitqueue(&cookie->flags, 0); + + wait_event_timeout(*wq, !__fscache_check_page_write(cookie, page), timeout); +} + /* * decide whether a page can be released, possibly by cancelling a store to it * - we're allowed to sleep if __GFP_WAIT is flagged @@ -115,7 +122,7 @@ page_busy: } fscache_stat(&fscache_n_store_vmscan_wait); - __fscache_wait_on_page_write(cookie, page); + __fscache_wait_on_page_write_timeout(cookie, page, HZ); gfp &= ~__GFP_WAIT; goto try_again; }