From patchwork Fri Mar 13 13:53:19 2009 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alasdair G Kergon X-Patchwork-Id: 11743 X-Patchwork-Delegate: agk@redhat.com Received: from hormel.redhat.com (hormel1.redhat.com [209.132.177.33]) by demeter.kernel.org (8.14.2/8.14.2) with ESMTP id n2DDrO3j021044 for ; Fri, 13 Mar 2009 13:53:24 GMT Received: from listman.util.phx.redhat.com (listman.util.phx.redhat.com [10.8.4.110]) by hormel.redhat.com (Postfix) with ESMTP id 2FB9061AD8F; Fri, 13 Mar 2009 09:53:23 -0400 (EDT) Received: from int-mx2.corp.redhat.com (nat-pool.util.phx.redhat.com [10.8.5.200]) by listman.util.phx.redhat.com (8.13.1/8.13.1) with ESMTP id n2DDrL7e006277 for ; Fri, 13 Mar 2009 09:53:21 -0400 Received: from ns3.rdu.redhat.com (ns3.rdu.redhat.com [10.11.255.199]) by int-mx2.corp.redhat.com (8.13.1/8.13.1) with ESMTP id n2DDrL9W004830; Fri, 13 Mar 2009 09:53:21 -0400 Received: from agk.fab.redhat.com (agk.fab.redhat.com [10.33.0.19]) by ns3.rdu.redhat.com (8.13.8/8.13.8) with ESMTP id n2DDrKgr010583; Fri, 13 Mar 2009 09:53:20 -0400 Received: from agk by agk.fab.redhat.com with local (Exim 4.34) id 1Li7p9-0000Yu-8D; Fri, 13 Mar 2009 13:53:19 +0000 Date: Fri, 13 Mar 2009 13:53:19 +0000 From: Alasdair G Kergon To: device-mapper development Subject: Re: [dm-devel] [PATCH] dm crypt: wait for possible unfinished endio() call in destructor Message-ID: <20090313135319.GJ7445@agk.fab.redhat.com> Mail-Followup-To: device-mapper development , Milan Broz References: <49AFA565.6030902@ahsoftware.de> <49AFBE4A.1010605@redhat.com> <49B0BFBD.7060208@ahsoftware.de> <49B0DDD6.2080608@redhat.com> <49B8F726.8040200@redhat.com> Mime-Version: 1.0 Content-Disposition: inline In-Reply-To: <49B8F726.8040200@redhat.com> User-Agent: Mutt/1.4.1i Organization: Red Hat UK Ltd. Registered in England and Wales, number 03798903. Registered Office: Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 1TE. X-Scanned-By: MIMEDefang 2.58 on 172.16.27.26 X-loop: dm-devel@redhat.com Cc: Milan Broz X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.5 Precedence: junk Reply-To: device-mapper development List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com Latest version of this patch. Please would some people test and/or review it asap so we can get it fixed in 2.6.29? Alasdair From: Milan Broz The following oops has been reported when dm-crypt runs over a loop device. ... [ 70.381058] Process loop0 (pid: 4268, ti=cf3b2000 task=cf1cc1f0 task.ti=cf3b2000) ... [ 70.381058] Call Trace: [ 70.381058] [] ? crypt_dec_pending+0x5e/0x62 [dm_crypt] [ 70.381058] [] ? crypt_endio+0xa2/0xaa [dm_crypt] [ 70.381058] [] ? crypt_endio+0x0/0xaa [dm_crypt] [ 70.381058] [] ? bio_endio+0x2b/0x2e [ 70.381058] [] ? dec_pending+0x224/0x23b [dm_mod] [ 70.381058] [] ? clone_endio+0x79/0xa4 [dm_mod] [ 70.381058] [] ? clone_endio+0x0/0xa4 [dm_mod] [ 70.381058] [] ? bio_endio+0x2b/0x2e [ 70.381058] [] ? loop_thread+0x380/0x3b7 [ 70.381058] [] ? do_lo_send_aops+0x0/0x165 [ 70.381058] [] ? autoremove_wake_function+0x0/0x33 [ 70.381058] [] ? loop_thread+0x0/0x3b7 When a table is being replaced, it waits for I/O to complete before destroying the mempool, but the endio function doesn't call mempool_free() until after completing the bio. Fix it by swapping the order of those two operations. The same problem occurs in dm.c with referenced after dec_pending. Again, we swap the order. Signed-off-by: Milan Broz Signed-off-by: Alasdair G Kergon --- drivers/md/dm-crypt.c | 17 ++++++++++------- drivers/md/dm.c | 32 +++++++++++++++++++------------- 2 files changed, 29 insertions(+), 20 deletions(-) -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel Index: linux-2.6.29-rc7/drivers/md/dm-crypt.c =================================================================== --- linux-2.6.29-rc7.orig/drivers/md/dm-crypt.c 2009-03-13 11:06:18.000000000 +0000 +++ linux-2.6.29-rc7/drivers/md/dm-crypt.c 2009-03-13 11:07:46.000000000 +0000 @@ -568,19 +568,22 @@ static void crypt_inc_pending(struct dm_ static void crypt_dec_pending(struct dm_crypt_io *io) { struct crypt_config *cc = io->target->private; + struct bio *base_bio = io->base_bio; + struct dm_crypt_io *base_io = io->base_io; + int error = io->error; if (!atomic_dec_and_test(&io->pending)) return; - if (likely(!io->base_io)) - bio_endio(io->base_bio, io->error); + mempool_free(io, cc->io_pool); + + if (likely(!base_io)) + bio_endio(base_bio, error); else { - if (io->error && !io->base_io->error) - io->base_io->error = io->error; - crypt_dec_pending(io->base_io); + if (error && !base_io->error) + base_io->error = error; + crypt_dec_pending(base_io); } - - mempool_free(io, cc->io_pool); } /* Index: linux-2.6.29-rc7/drivers/md/dm.c =================================================================== --- linux-2.6.29-rc7.orig/drivers/md/dm.c 2009-03-09 22:12:09.000000000 +0000 +++ linux-2.6.29-rc7/drivers/md/dm.c 2009-03-13 12:34:19.000000000 +0000 @@ -525,9 +525,12 @@ static int __noflush_suspending(struct m static void dec_pending(struct dm_io *io, int error) { unsigned long flags; + int io_error; + struct bio *bio; + struct mapped_device *md = io->md; /* Push-back supersedes any I/O errors */ - if (error && !(io->error > 0 && __noflush_suspending(io->md))) + if (error && !(io->error > 0 && __noflush_suspending(md))) io->error = error; if (atomic_dec_and_test(&io->io_count)) { @@ -537,24 +540,27 @@ static void dec_pending(struct dm_io *io * This must be handled before the sleeper on * suspend queue merges the pushback list. */ - spin_lock_irqsave(&io->md->pushback_lock, flags); - if (__noflush_suspending(io->md)) - bio_list_add(&io->md->pushback, io->bio); + spin_lock_irqsave(&md->pushback_lock, flags); + if (__noflush_suspending(md)) + bio_list_add(&md->pushback, io->bio); else /* noflush suspend was interrupted. */ io->error = -EIO; - spin_unlock_irqrestore(&io->md->pushback_lock, flags); + spin_unlock_irqrestore(&md->pushback_lock, flags); } end_io_acct(io); - if (io->error != DM_ENDIO_REQUEUE) { - trace_block_bio_complete(io->md->queue, io->bio); + io_error = io->error; + bio = io->bio; - bio_endio(io->bio, io->error); - } + free_io(md, io); - free_io(io->md, io); + if (io_error != DM_ENDIO_REQUEUE) { + trace_block_bio_complete(md->queue, bio); + + bio_endio(bio, io_error); + } } } @@ -562,6 +568,7 @@ static void clone_endio(struct bio *bio, { int r = 0; struct dm_target_io *tio = bio->bi_private; + struct dm_io *io = tio->io; struct mapped_device *md = tio->io->md; dm_endio_fn endio = tio->ti->type->end_io; @@ -585,15 +592,14 @@ static void clone_endio(struct bio *bio, } } - dec_pending(tio->io, error); - /* * Store md for cleanup instead of tio which is about to get freed. */ bio->bi_private = md->bs; - bio_put(bio); free_tio(md, tio); + bio_put(bio); + dec_pending(io, error); } static sector_t max_io_len(struct mapped_device *md,