From patchwork Sun Jul 31 17:04:03 2011 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sage Weil X-Patchwork-Id: 1023902 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by demeter2.kernel.org (8.14.4/8.14.4) with ESMTP id p6VH06Vv024096 for ; Sun, 31 Jul 2011 17:00:06 GMT Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752566Ab1GaRAE (ORCPT ); Sun, 31 Jul 2011 13:00:04 -0400 Received: from cobra.newdream.net ([66.33.216.30]:43631 "EHLO cobra.newdream.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752692Ab1GaRAA (ORCPT ); Sun, 31 Jul 2011 13:00:00 -0400 Received: from cobra.newdream.net (localhost [127.0.0.1]) by cobra.newdream.net (Postfix) with ESMTP id 2A8CEBC71A; Sun, 31 Jul 2011 10:04:04 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=newdream.net; h=date:from:to:cc :subject:in-reply-to:message-id:references:mime-version: content-type; q=dns; s=newdream.net; b=CvAfbxGzDqBVwxepfDUNWtSHz J67Pd7z4RQqzw+FzHnI8pc+pN71gXML6J4DwdY9DJkiakRnvIKn4XarwtpQ10xyl JP2wCCre9SNPXbM1NnWFqM9Nd/wi7GjkxUo+25p2QsZd8nAthjoHjMyfTdWQoLYE 9ia/V73z7WD0lji3gM= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=newdream.net; h=date:from :to:cc:subject:in-reply-to:message-id:references:mime-version: content-type; s=newdream.net; bh=FO/D/t81yuwDWH6W4yGwmH05Y0A=; b=D3Of+A1yekHuaHA+9EW56wW3TcGolwB00JKDhjc+Fi9vOfg1Jsn8UxZjcg/A4 s0D7Q6S6Ta/wkNWW2X44F0pneaVZEaVQtAf1kzbaEBJOTyybIpYOZfemWbmOc9ub yQrhNeBB4h4PnSWSSqQH10Sw2HijpxXurGE3oZ5+RXjfOQ= Received: by cobra.newdream.net (Postfix, from userid 1031) id 10808BC764; Sun, 31 Jul 2011 10:04:04 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by cobra.newdream.net (Postfix) with ESMTP id 030BBBC71A; Sun, 31 Jul 2011 10:04:04 -0700 (PDT) Date: Sun, 31 Jul 2011 10:04:03 -0700 (PDT) From: Sage Weil To: Fyodor Ustinov cc: "Ted Ts'o" , ceph-devel@vger.kernel.org Subject: Re: Kernel 3.0.0 + ext4 + ceph == ... In-Reply-To: <4E353D9E.5080802@ufm.su> Message-ID: References: <4E33D101.1050504@ufm.su> <9BF9E529-C532-4A94-8362-93C2D1B778DB@mit.edu> <4E3432FC.9030204@ufm.su> <20110730165001.GI7361@thunk.org> <20110730221900.GK7361@thunk.org> <4E353D9E.5080802@ufm.su> MIME-Version: 1.0 Sender: ceph-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org X-Greylist: IP, sender and recipient auto-whitelisted, not delayed by milter-greylist-4.2.6 (demeter2.kernel.org [140.211.167.43]); Sun, 31 Jul 2011 17:00:06 +0000 (UTC) On Sun, 31 Jul 2011, Fyodor Ustinov wrote: > On 07/31/2011 07:54 AM, Sage Weil wrote: > > On Sat, 30 Jul 2011, Ted Ts'o wrote: > > > On Sat, Jul 30, 2011 at 10:21:13AM -0700, Sage Weil wrote: > > > > We do use xattrs extensively, though; that was the last extN bug we > > > > uncovered. That's where my money is. > > > Hmm, yes. That could very well be. How big are the xattrs, and are > > > there cases where you: > > > > > > a) start with a small xattr (where the total size is less than 128 > > > bytes, so it can be stored in the inode table), and then increase it > > > something where it needs to be stored in an external block? > > > > > > b) start with enough xattrs so it's large, and then delete all or most > > > of them? > > > > > > I could easily believe we might have some bugs as we transition from > > > in-inode to external block storage, or vice versa. I'll take a look > > > at the code and try to create some reproduction cases, but if you > > > could give me a handle on workload patterns of ceph around xattrs, > > > that would be interesting. > > I would guess a, but it could also be a+b. > > > > Fyodor, can you take some of the corrupt inos that fsck complained about > > and see what files/directories they are? find /osd.0 -inum NNN. (I'm > > guessing the largest xattrs are on the collection directories, like > > /osd.0/current/something_head/.) Then grep that filename out of the log > > to see exactly which operations took place. The setattr log normally > > includes xattr size. > > > /etc/init.d/ceph stop > umount /mnt/osd.0 > mke2fs -t ext4 -I 128 /dev/sdc1 > tune2fs -o journal_data_writeback /dev/sdc1 > mount -a > mon getmap -o /tmp/monmap > cosd --mkfs -i 0 --monmap /tmp/monmap > /etc/init.d/ceph start > sleep 300 > /etc/init.d/ceph stop > umount /osd.0 > fsck.ext4 -f /dev/sdc1 > > Inode 99356878, i_blocks is 8208, should be 8200. > > mount -a > root@osd0:~# find /osd.0 -inum 99356878 > > /osd.0/current/0.2a4_head/10000000468.0000007e_head > > root@osd0:~# grep "10000000468\.0000007e" /var/log/ceph/osd.0.log > 2011-07-31 09:57:20.859834 7f624c82a700 filestore(/osd.0) remove > temp/10000000468.0000007e/head = -1 > 2011-07-31 09:57:20.861166 7f624c82a700 filestore(/osd.0) write > temp/10000000468.0000007e/head 0~1048576 = 1048576 > 2011-07-31 09:57:20.990464 7f624c029700 filestore(/osd.0) write > temp/10000000468.0000007e/head 1048576~1048576 = 1048576 > 2011-07-31 09:57:21.121648 7f624c029700 filestore(/osd.0) write > temp/10000000468.0000007e/head 2097152~1048576 = 1048576 > 2011-07-31 09:57:21.265879 7f624c029700 filestore(/osd.0) write > temp/10000000468.0000007e/head 3145728~1048576 = 1048576 > 2011-07-31 09:57:21.265952 7f624c029700 filestore(/osd.0) remove > 0.2a4_head/10000000468.0000007e/head = -1 > 2011-07-31 09:57:21.265995 7f624c029700 filestore(/osd.0) collection_add > 0.2a4_head/10000000468.0000007e/head temp/10000000468.0000007e/head = 0 > 2011-07-31 09:57:21.266025 7f624c029700 filestore(/osd.0) collection_remove > temp/10000000468.0000007e/head = 0 > 2011-07-31 09:57:21.266134 7f624c029700 filestore(/osd.0) setattrs > 0.2a4_head/10000000468.0000007e/head = 26 Hrm, I was hoping it wouldn't be a setattrs call. The below will tell us what xattrs it is setting, but sadly you'll need to reproduce the whole thing again. The above is only telling us the size of the last xattr of the set. Ted, I'm assuming the i_blocks mismatch is likely on the same files that make the transition from/to in-inode xattrs? sage --- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/src/os/FileStore.cc b/src/os/FileStore.cc index 8bdee6b..fb00cff 100644 --- a/src/os/FileStore.cc +++ b/src/os/FileStore.cc @@ -3570,6 +3570,7 @@ int FileStore::_setattrs(coll_t cid, const sobject_t& oid, map val = ""; // ??? Why do we skip setting all the other attrs if one fails? r = lfn_setxattr(cid, oid, n, val, p->second.length()); + dout(10) << "setattrs " << cid << "/" << oid << " '" << p->first << "' len " << p->second.length() << " = " << r << dendl; if (r < 0) { derr << "FileStore::_setattrs: do_setxattr returned " << r << dendl; break;