From patchwork Wed Nov 6 12:35:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13864639 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F864D44D54 for ; Wed, 6 Nov 2024 12:38:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AB29C6B00C1; Wed, 6 Nov 2024 07:38:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A634B6B00C2; Wed, 6 Nov 2024 07:38:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 903786B00C3; Wed, 6 Nov 2024 07:38:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 717576B00C1 for ; Wed, 6 Nov 2024 07:38:10 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 3175D121B57 for ; Wed, 6 Nov 2024 12:38:10 +0000 (UTC) X-FDA: 82755620850.09.A15E3F8 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf12.hostedemail.com (Postfix) with ESMTP id D0D4E4001B for ; Wed, 6 Nov 2024 12:37:52 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UpP8LlFS; spf=pass (imf12.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730896464; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RVoEs9N2N3+Yy5ytz8xwWGrwrjCUqjBQVK6VxNHCBSE=; b=W1KT/xik00zoRWNhdXb5tNBauRA/aKW+P+j/6BWrKfasQWHwpiB9hlPNZ8SRiKxryyqzER UwxXQLoQhYgDfJ1BlWT4Z/GqHf+RQpHej/wbDwH/ZjVFfu3wE8mVOSbRoXe9ZpSWBuru9B V2R4ToTye5t6arHCES87wCvGYrRdl+8= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UpP8LlFS; spf=pass (imf12.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730896464; a=rsa-sha256; cv=none; b=WnFvASDI5djpslFanQTn6ETIqF9xTqBlB+kRbf/edjK6Nb7CCN8AEiX7mD2wu2ESG1GSQo CciljMOx39+J8dUHcWtGzX0S61WJIgYlrDTn73WxhHUOSYeI7k9XQDTjmhvI0ANs4BC/Xg 7yyvlk2/RceIxG+7LgK3FsfCajRVbBc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1730896687; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RVoEs9N2N3+Yy5ytz8xwWGrwrjCUqjBQVK6VxNHCBSE=; b=UpP8LlFS0wJ+2/ZPP2FP5a7erlFMbqHHU55b24GyXDSMmw8xU3ZpvYoTuVu0GD0rB7VOT1 SQGN/ZR0nLJELFway2kdWSL80JAqYvko7s6DFhTR8IRAVeBylNuUEX6UC1peyWv7Ssm9Bv 5OytG+6oHPQNg4iBYZ6wLW7KhHgjOEU= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-626-Lke04AvaNPCkEvwplLVMIg-1; Wed, 06 Nov 2024 07:37:54 -0500 X-MC-Unique: Lke04AvaNPCkEvwplLVMIg-1 X-Mimecast-MFC-AGG-ID: Lke04AvaNPCkEvwplLVMIg Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id D05A51955D99; Wed, 6 Nov 2024 12:37:50 +0000 (UTC) Received: from warthog.procyon.org.uk.com (unknown [10.42.28.231]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 31C6F30000DF; Wed, 6 Nov 2024 12:37:44 +0000 (UTC) From: David Howells To: Christian Brauner , Steve French , Matthew Wilcox Cc: David Howells , Jeff Layton , Gao Xiang , Dominique Martinet , Marc Dionne , Paulo Alcantara , Shyam Prasad N , Tom Talpey , Eric Van Hensbergen , Ilya Dryomov , netfs@lists.linux.dev, linux-afs@lists.infradead.org, linux-cifs@vger.kernel.org, linux-nfs@vger.kernel.org, ceph-devel@vger.kernel.org, v9fs@lists.linux.dev, linux-erofs@lists.ozlabs.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 13/33] afs: Don't use mutex for I/O operation lock Date: Wed, 6 Nov 2024 12:35:37 +0000 Message-ID: <20241106123559.724888-14-dhowells@redhat.com> In-Reply-To: <20241106123559.724888-1-dhowells@redhat.com> References: <20241106123559.724888-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: D0D4E4001B X-Stat-Signature: egsc6gu6c4htoh5xi5xkk7bar56isizo X-Rspam-User: X-HE-Tag: 1730896672-858343 X-HE-Meta: U2FsdGVkX1/sHNWNtHW/Tfcetka1GkdwqnJ2A72CqOrP5Trcbd5MTLfXeZMg/yJtpqFxqXTJ6q9FjTlFw3sodKryuXTbhrxpkAHOXvzY+zLSm9FzHss72vC6uFpnUOb0ailo6AiKrt+OODIouhTLkO/fD7eRR0QUHEE9uJuBRIwWPUPk1U9FqybqKiaPCktcemyFS1VbVDp4I6F6Y/oxuumXEOYdYD8z9mDZxobTjEKnw2vBaztuUUTrTqcLEKpJuMeXfxVBdIvdQsG7HYljC64Vyd25P5Gd9Nh680WpHicXT03RIhtHBbTs3krlDs2jKJl/2e4Xm2mD/mFVagUaMSbnhTMh7mjzb//09/S8cB1VmzzGHs1S5FbfREvdEKxK6XkIbEp6FsJWbVIfCNITJokhabNMSLL0gqVX7MdyRlMHOWOTN4ino1lXDOhFzgVG6+F5LVDzi2WFzGzg9k3sr7tk3KDppH5MtCiUjW7zT3Hi1UUb/yfdjIkDGWcPrINC409nnt3IvV8ZGtXxsJYg2cyTQ913kIJ+H+xUylR+zvzIq2S98YLruXUrMNVNKefxE5bW/loJHpZp7HayGecqLLuJTeqoo+IyiFJzHX3F+H4kyAiJaj609RNBtq2TBCydMCk/oRwBo8ok5250PKoPf/b8WxVIE4z+wb7EBEjjwQv2iu1PH0G07LnSk2pJXS4kmH0kD6X0sMtMM3+BSPV0Oc7NEPO5Hm+aA+zobTSN/o/Qey2DZbIxuvYadCz6JViE7XjecCnAyeUKVGSHajPobnQEWhPD0mOYjxBRYZadBBNRp1M3qcYQYcBG7Q/bk06EkytawCwKGV5DtENym4wxFodA0l5IPHaOhJaMkmyw4hYlrKz8cLGStzYxRnEukTAM9pUUsSlMm4wsj1agUFrZgy/f6PeemDdiNKjgUbeNkgxjN8F5wdQJNuD+d0FdrYurJSr1I79LiMIkDmihelD 5YVfRidP DHOZHOeaz81H8cHNS+JQH3n4oOcql7GgsGeVmc9UQoFCRCXrr9dUgVzD7DHjHSZnppCsVH1AO0crB1JbHlTBZLDGrYeErOxDtS/qlXAhOveqZO5Zir0/a5KoRefEOKbB93k1yqbY3EYmhdloacEPbrRBuyWNgcZQRbgAxPZyqdJP2wsXOFZd6KSwM9eiRNqK7gr1F8Jno96Mz/PGGbz+nTwb8ie0VT/sfLgWKdHy1/zjsV2jG/EO4B3LjwGqHfh3DD+oXUirVD51n1RpSIbrWnJhgIkZEmTAFe02UUF2fKd6uvZvUybJPBGCTj2jE1iyZnVzIa/QdGa382JpRx8NZz9gvEUPKH3zd93Guw284zhD6LVgc+msDXBOoZYi+eVHQ3vpfPUf/CvvqXjTiEvjtj/BtPVDGS63q5TA6i001Tkqkqjlk4bj4WPsMIaLhwF0sNs8pf+UuQ5Ed0fdEWe1VWuUiELiM7GlwXOeNr9TGmEZEtVw5z1JyAmpraS+hOOS8aER6fA7ELrcNE9c= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Don't use the standard mutex for the I/O operation lock, but rather implement our own as the standard mutex must be released in the same thread as locked it. This is a problem when it comes to doing async FetchData where the lock will be dropped from the workqueue that processed the incoming data and not from the issuing thread. Signed-off-by: David Howells cc: Marc Dionne cc: linux-afs@lists.infradead.org --- fs/afs/fs_operation.c | 111 +++++++++++++++++++++++++++++++++++++++--- fs/afs/internal.h | 3 +- fs/afs/super.c | 2 +- 3 files changed, 108 insertions(+), 8 deletions(-) diff --git a/fs/afs/fs_operation.c b/fs/afs/fs_operation.c index 428721bbe4f6..8488ff8183fa 100644 --- a/fs/afs/fs_operation.c +++ b/fs/afs/fs_operation.c @@ -49,6 +49,105 @@ struct afs_operation *afs_alloc_operation(struct key *key, struct afs_volume *vo return op; } +struct afs_io_locker { + struct list_head link; + struct task_struct *task; + unsigned long have_lock; +}; + +/* + * Unlock the I/O lock on a vnode. + */ +static void afs_unlock_for_io(struct afs_vnode *vnode) +{ + struct afs_io_locker *locker; + + spin_lock(&vnode->lock); + locker = list_first_entry_or_null(&vnode->io_lock_waiters, + struct afs_io_locker, link); + if (locker) { + list_del(&locker->link); + smp_store_release(&locker->have_lock, 1); + smp_mb__after_atomic(); /* Store have_lock before task state */ + wake_up_process(locker->task); + } else { + clear_bit(AFS_VNODE_IO_LOCK, &vnode->flags); + } + spin_unlock(&vnode->lock); +} + +/* + * Lock the I/O lock on a vnode uninterruptibly. We can't use an ordinary + * mutex as lockdep will complain if we unlock it in the wrong thread. + */ +static void afs_lock_for_io(struct afs_vnode *vnode) +{ + struct afs_io_locker myself = { .task = current, }; + + spin_lock(&vnode->lock); + + if (!test_and_set_bit(AFS_VNODE_IO_LOCK, &vnode->flags)) { + spin_unlock(&vnode->lock); + return; + } + + list_add_tail(&myself.link, &vnode->io_lock_waiters); + spin_unlock(&vnode->lock); + + for (;;) { + set_current_state(TASK_UNINTERRUPTIBLE); + if (smp_load_acquire(&myself.have_lock)) + break; + schedule(); + } + __set_current_state(TASK_RUNNING); +} + +/* + * Lock the I/O lock on a vnode interruptibly. We can't use an ordinary mutex + * as lockdep will complain if we unlock it in the wrong thread. + */ +static int afs_lock_for_io_interruptible(struct afs_vnode *vnode) +{ + struct afs_io_locker myself = { .task = current, }; + int ret = 0; + + spin_lock(&vnode->lock); + + if (!test_and_set_bit(AFS_VNODE_IO_LOCK, &vnode->flags)) { + spin_unlock(&vnode->lock); + return 0; + } + + list_add_tail(&myself.link, &vnode->io_lock_waiters); + spin_unlock(&vnode->lock); + + for (;;) { + set_current_state(TASK_INTERRUPTIBLE); + if (smp_load_acquire(&myself.have_lock) || + signal_pending(current)) + break; + schedule(); + } + __set_current_state(TASK_RUNNING); + + /* If we got a signal, try to transfer the lock onto the next + * waiter. + */ + if (unlikely(signal_pending(current))) { + spin_lock(&vnode->lock); + if (myself.have_lock) { + spin_unlock(&vnode->lock); + afs_unlock_for_io(vnode); + } else { + list_del(&myself.link); + spin_unlock(&vnode->lock); + } + ret = -ERESTARTSYS; + } + return ret; +} + /* * Lock the vnode(s) being operated upon. */ @@ -60,7 +159,7 @@ static bool afs_get_io_locks(struct afs_operation *op) _enter(""); if (op->flags & AFS_OPERATION_UNINTR) { - mutex_lock(&vnode->io_lock); + afs_lock_for_io(vnode); op->flags |= AFS_OPERATION_LOCK_0; _leave(" = t [1]"); return true; @@ -72,7 +171,7 @@ static bool afs_get_io_locks(struct afs_operation *op) if (vnode2 > vnode) swap(vnode, vnode2); - if (mutex_lock_interruptible(&vnode->io_lock) < 0) { + if (afs_lock_for_io_interruptible(vnode) < 0) { afs_op_set_error(op, -ERESTARTSYS); op->flags |= AFS_OPERATION_STOP; _leave(" = f [I 0]"); @@ -81,10 +180,10 @@ static bool afs_get_io_locks(struct afs_operation *op) op->flags |= AFS_OPERATION_LOCK_0; if (vnode2) { - if (mutex_lock_interruptible_nested(&vnode2->io_lock, 1) < 0) { + if (afs_lock_for_io_interruptible(vnode2) < 0) { afs_op_set_error(op, -ERESTARTSYS); op->flags |= AFS_OPERATION_STOP; - mutex_unlock(&vnode->io_lock); + afs_unlock_for_io(vnode); op->flags &= ~AFS_OPERATION_LOCK_0; _leave(" = f [I 1]"); return false; @@ -104,9 +203,9 @@ static void afs_drop_io_locks(struct afs_operation *op) _enter(""); if (op->flags & AFS_OPERATION_LOCK_1) - mutex_unlock(&vnode2->io_lock); + afs_unlock_for_io(vnode2); if (op->flags & AFS_OPERATION_LOCK_0) - mutex_unlock(&vnode->io_lock); + afs_unlock_for_io(vnode); } static void afs_prepare_vnode(struct afs_operation *op, struct afs_vnode_param *vp, diff --git a/fs/afs/internal.h b/fs/afs/internal.h index c9d620175e80..07b8f7083e73 100644 --- a/fs/afs/internal.h +++ b/fs/afs/internal.h @@ -702,13 +702,14 @@ struct afs_vnode { struct afs_file_status status; /* AFS status info for this file */ afs_dataversion_t invalid_before; /* Child dentries are invalid before this */ struct afs_permits __rcu *permit_cache; /* cache of permits so far obtained */ - struct mutex io_lock; /* Lock for serialising I/O on this mutex */ + struct list_head io_lock_waiters; /* Threads waiting for the I/O lock */ struct rw_semaphore validate_lock; /* lock for validating this vnode */ struct rw_semaphore rmdir_lock; /* Lock for rmdir vs sillyrename */ struct key *silly_key; /* Silly rename key */ spinlock_t wb_lock; /* lock for wb_keys */ spinlock_t lock; /* waitqueue/flags lock */ unsigned long flags; +#define AFS_VNODE_IO_LOCK 0 /* Set if the I/O serialisation lock is held */ #define AFS_VNODE_UNSET 1 /* set if vnode attributes not yet set */ #define AFS_VNODE_DIR_VALID 2 /* Set if dir contents are valid */ #define AFS_VNODE_ZAP_DATA 3 /* set if vnode's data should be invalidated */ diff --git a/fs/afs/super.c b/fs/afs/super.c index f3ba1c3e72f5..7631302c1984 100644 --- a/fs/afs/super.c +++ b/fs/afs/super.c @@ -663,7 +663,7 @@ static void afs_i_init_once(void *_vnode) memset(vnode, 0, sizeof(*vnode)); inode_init_once(&vnode->netfs.inode); - mutex_init(&vnode->io_lock); + INIT_LIST_HEAD(&vnode->io_lock_waiters); init_rwsem(&vnode->validate_lock); spin_lock_init(&vnode->wb_lock); spin_lock_init(&vnode->lock);