From patchwork Mon Dec 16 20:41:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13910336 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35F16E77180 for ; Mon, 16 Dec 2024 20:43:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C23306B007B; Mon, 16 Dec 2024 15:43:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BD2566B0088; Mon, 16 Dec 2024 15:43:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A4C426B00CB; Mon, 16 Dec 2024 15:43:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 867536B007B for ; Mon, 16 Dec 2024 15:43:06 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 38AA016103A for ; Mon, 16 Dec 2024 20:43:06 +0000 (UTC) X-FDA: 82901996436.28.2D7BFA7 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf21.hostedemail.com (Postfix) with ESMTP id BC6041C0002 for ; Mon, 16 Dec 2024 20:42:07 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=APwkUQjI; spf=pass (imf21.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734381764; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MklTcYfvPOLy1NwZc1kfQfg10kbAaSIvx2D4rHXDyWY=; b=02inlzCf6+yZxE4BVtd8Cwsel6g0SHxVXXiVhjyGgIWlKdc2mF2zKlxZMeyt2mrWH+3MOA W5JUozCba1dqEkI5PatCN1jTpGMgFYbZuaykxMSDbCacwE+nIi24YPE5tlcI3rn1xB9e8f +5OLsRP6kdRutEqIXU/r033NIOC+Etc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734381764; a=rsa-sha256; cv=none; b=TdW+hLMrT1BkkGFMVoPhP81zNNG5WxwGMUQkBrYWAdXMKWUiLvftMX6AC1m49MtoC+ZRcB ZBDYaIPFZnBgTYp+UK9b1X7N3zjeA3yF2tmkHzvZ9Iql/mMF9J/df1niO4C4x+NU3sA8fY aeS6FNjDCfBEIVcDoNvkvGO/GRbhh5U= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=APwkUQjI; spf=pass (imf21.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1734381783; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MklTcYfvPOLy1NwZc1kfQfg10kbAaSIvx2D4rHXDyWY=; b=APwkUQjI+ehxAzyCsdPg6QGfT76ab2/AKndZM8Bpf7+ZTNmzXN+acEjgMs9rfYfVOc22rw zzgKeTMMKP+ztrYJdXK88vXzcpW1LqsU7YbmmBHV/Jz+uvv5fTV3IniYpAT8Co9lWpzc+A G5GzILyA0XS0A8OpvI39fRo6X/OxxRk= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-423-1s12ay6wOs2Tgrvj1-IX_g-1; Mon, 16 Dec 2024 15:43:00 -0500 X-MC-Unique: 1s12ay6wOs2Tgrvj1-IX_g-1 X-Mimecast-MFC-AGG-ID: 1s12ay6wOs2Tgrvj1-IX_g Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id D71951956054; Mon, 16 Dec 2024 20:42:57 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.48]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id DA7DE19560B0; Mon, 16 Dec 2024 20:42:51 +0000 (UTC) From: David Howells To: Christian Brauner , Steve French , Matthew Wilcox Cc: David Howells , Jeff Layton , Gao Xiang , Dominique Martinet , Marc Dionne , Paulo Alcantara , Shyam Prasad N , Tom Talpey , Eric Van Hensbergen , Ilya Dryomov , netfs@lists.linux.dev, linux-afs@lists.infradead.org, linux-cifs@vger.kernel.org, linux-nfs@vger.kernel.org, ceph-devel@vger.kernel.org, v9fs@lists.linux.dev, linux-erofs@lists.ozlabs.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 11/32] afs: Don't use mutex for I/O operation lock Date: Mon, 16 Dec 2024 20:41:01 +0000 Message-ID: <20241216204124.3752367-12-dhowells@redhat.com> In-Reply-To: <20241216204124.3752367-1-dhowells@redhat.com> References: <20241216204124.3752367-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: BC6041C0002 X-Stat-Signature: fdo5cme8cqabqgazmt1gmihtki9xkhs6 X-Rspam-User: X-HE-Tag: 1734381727-494161 X-HE-Meta: U2FsdGVkX199NU/FsrR0ZUA9kv/7k11x7sAVJ+ejIW8Kk905q0mMqtBx87J7C0uz2IREhgJet6fAlpD6qlSTBu8+yAiZ6v9JB7PlrMoDz8F5QFN+07wHUNJ5BGcVbwJij1fg/nfGog5gNQby/x5l4+V6iKHS3O0e/5jm4tEoZXzQ+MBXwqgVaiLdaixl9omm2QhScBTcn/0YRlvHChouHYHQ9SC0BUgvhQLrCc0HMRBDvmlT3kQDXMtwvUJnJ6f51Yi/4i7mCN0rDqXQpVygsnzrOp4sDfLD8ViqAa9Q5UKB6vMU1Lqr4yftBQslbRXn+uyWbcjUx9slyq6nsZIzot95NWMk1hPHOPGQ7OCgRVQnkfiYl0BvNZErOjvuoZoOGlr/XYU6ahCwRZWKQNkDVtKeMqGVFjxB2m3QT+Xwfc8KuVvsKSx/0PIkrcY3ovGK11LV01ow+DHVVJSJYO+OgZ9BrCwQ1+CTUWrnZs0DEWbyFhElTzuM3gGRtmpt1jj+G1DWjLb9dNTqUx/cmkB2OIUaZ3ORaTkYa7sb+5CH8/b4erkGgk2PXZcYP1ezTSCwwo16ZEhOHjLHAGRBlqT0sI1lQfHIOqnNlVxWg+pBrjatLKI51w9KVmbbynX1bZtvrB37W1og6qNEz4RWgwPuqjGL9oWLVHD+ae4zc53BSiKNvV7KVXpzpFBTtqS87tQU7elAnlkAISa+7Yv88GaTYrwcerJnGD+EC0GI83b5t9OgYXpo4IPOaKWxAUkvumnAmep6VFduprntdajkWiMatIbhjGHuBivVUt24Jdu1tBJe5lWVy7LsSYoH9UG7hORLn0esybh3fQYBOCE82yXB9C4Z5y0QZkcf3Z5M5C/D5k8RKT4qMla7hb7kzp8leSw21AKMkl2HO2TnZFVe6SN49KF6bGfkiSRDTauVrOK2gm8WLVQqHd8yLGZO/20sSd7W4DMLPB9LqrgZeuzv5RL xo8oCHz3 MkRB9zOoqkamDUbxjq/v6vIczzrXzrwXMGq2ixi4keBTVBHd71vJPaaK2SyF1QpPptVf+eAhB+eAaznreGlrSwdbrPX8ZIpoMhrKVgwmexg03P5KgMQoY1jF6Z5+cdvP/5RYTWqJ2DOEnkxJAyXCImGUbC9ytxFMaOhq/1bB1S0egUMFLlTOXPly5CGR5w4sFM99HfZBHuWetj3ulirN66js59ae8diFctqn7YQsgKTzUn0RrdoVg6uJW7ev3S65W4UeQz6DO5tofpdBWF3GIdC47V3DPA9jyrY41koMK/Fn76qUO0g342fDW+GeM4bxrzZWxrvFY12vmPVOv2JIG8JY4mr1sDpMPi2ITTlw6g8NOINN67FkdvcQsUE1UWcmlLvUr+NjUy1wLd75+7vOw6fdQlRz8tRZi8EwvrIw54V3OUhPkZQhmdU09y3AqkigKBnPY X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Don't use the standard mutex for the I/O operation lock, but rather implement our own as the standard mutex must be released in the same thread as locked it. This is a problem when it comes to doing async FetchData where the lock will be dropped from the workqueue that processed the incoming data and not from the issuing thread. Signed-off-by: David Howells cc: Marc Dionne cc: linux-afs@lists.infradead.org --- fs/afs/fs_operation.c | 111 +++++++++++++++++++++++++++++++++++++++--- fs/afs/internal.h | 3 +- fs/afs/super.c | 2 +- 3 files changed, 108 insertions(+), 8 deletions(-) diff --git a/fs/afs/fs_operation.c b/fs/afs/fs_operation.c index 428721bbe4f6..0175d7a31332 100644 --- a/fs/afs/fs_operation.c +++ b/fs/afs/fs_operation.c @@ -49,6 +49,105 @@ struct afs_operation *afs_alloc_operation(struct key *key, struct afs_volume *vo return op; } +struct afs_io_locker { + struct list_head link; + struct task_struct *task; + unsigned long have_lock; +}; + +/* + * Unlock the I/O lock on a vnode. + */ +static void afs_unlock_for_io(struct afs_vnode *vnode) +{ + struct afs_io_locker *locker; + + spin_lock(&vnode->lock); + locker = list_first_entry_or_null(&vnode->io_lock_waiters, + struct afs_io_locker, link); + if (locker) { + list_del(&locker->link); + smp_store_release(&locker->have_lock, 1); /* The unlock barrier. */ + smp_mb__after_atomic(); /* Store have_lock before task state */ + wake_up_process(locker->task); + } else { + clear_bit(AFS_VNODE_IO_LOCK, &vnode->flags); + } + spin_unlock(&vnode->lock); +} + +/* + * Lock the I/O lock on a vnode uninterruptibly. We can't use an ordinary + * mutex as lockdep will complain if we unlock it in the wrong thread. + */ +static void afs_lock_for_io(struct afs_vnode *vnode) +{ + struct afs_io_locker myself = { .task = current, }; + + spin_lock(&vnode->lock); + + if (!test_and_set_bit(AFS_VNODE_IO_LOCK, &vnode->flags)) { + spin_unlock(&vnode->lock); + return; + } + + list_add_tail(&myself.link, &vnode->io_lock_waiters); + spin_unlock(&vnode->lock); + + for (;;) { + set_current_state(TASK_UNINTERRUPTIBLE); + if (smp_load_acquire(&myself.have_lock)) /* The lock barrier */ + break; + schedule(); + } + __set_current_state(TASK_RUNNING); +} + +/* + * Lock the I/O lock on a vnode interruptibly. We can't use an ordinary mutex + * as lockdep will complain if we unlock it in the wrong thread. + */ +static int afs_lock_for_io_interruptible(struct afs_vnode *vnode) +{ + struct afs_io_locker myself = { .task = current, }; + int ret = 0; + + spin_lock(&vnode->lock); + + if (!test_and_set_bit(AFS_VNODE_IO_LOCK, &vnode->flags)) { + spin_unlock(&vnode->lock); + return 0; + } + + list_add_tail(&myself.link, &vnode->io_lock_waiters); + spin_unlock(&vnode->lock); + + for (;;) { + set_current_state(TASK_INTERRUPTIBLE); + if (smp_load_acquire(&myself.have_lock) || /* The lock barrier */ + signal_pending(current)) + break; + schedule(); + } + __set_current_state(TASK_RUNNING); + + /* If we got a signal, try to transfer the lock onto the next + * waiter. + */ + if (unlikely(signal_pending(current))) { + spin_lock(&vnode->lock); + if (myself.have_lock) { + spin_unlock(&vnode->lock); + afs_unlock_for_io(vnode); + } else { + list_del(&myself.link); + spin_unlock(&vnode->lock); + } + ret = -ERESTARTSYS; + } + return ret; +} + /* * Lock the vnode(s) being operated upon. */ @@ -60,7 +159,7 @@ static bool afs_get_io_locks(struct afs_operation *op) _enter(""); if (op->flags & AFS_OPERATION_UNINTR) { - mutex_lock(&vnode->io_lock); + afs_lock_for_io(vnode); op->flags |= AFS_OPERATION_LOCK_0; _leave(" = t [1]"); return true; @@ -72,7 +171,7 @@ static bool afs_get_io_locks(struct afs_operation *op) if (vnode2 > vnode) swap(vnode, vnode2); - if (mutex_lock_interruptible(&vnode->io_lock) < 0) { + if (afs_lock_for_io_interruptible(vnode) < 0) { afs_op_set_error(op, -ERESTARTSYS); op->flags |= AFS_OPERATION_STOP; _leave(" = f [I 0]"); @@ -81,10 +180,10 @@ static bool afs_get_io_locks(struct afs_operation *op) op->flags |= AFS_OPERATION_LOCK_0; if (vnode2) { - if (mutex_lock_interruptible_nested(&vnode2->io_lock, 1) < 0) { + if (afs_lock_for_io_interruptible(vnode2) < 0) { afs_op_set_error(op, -ERESTARTSYS); op->flags |= AFS_OPERATION_STOP; - mutex_unlock(&vnode->io_lock); + afs_unlock_for_io(vnode); op->flags &= ~AFS_OPERATION_LOCK_0; _leave(" = f [I 1]"); return false; @@ -104,9 +203,9 @@ static void afs_drop_io_locks(struct afs_operation *op) _enter(""); if (op->flags & AFS_OPERATION_LOCK_1) - mutex_unlock(&vnode2->io_lock); + afs_unlock_for_io(vnode2); if (op->flags & AFS_OPERATION_LOCK_0) - mutex_unlock(&vnode->io_lock); + afs_unlock_for_io(vnode); } static void afs_prepare_vnode(struct afs_operation *op, struct afs_vnode_param *vp, diff --git a/fs/afs/internal.h b/fs/afs/internal.h index c9d620175e80..07b8f7083e73 100644 --- a/fs/afs/internal.h +++ b/fs/afs/internal.h @@ -702,13 +702,14 @@ struct afs_vnode { struct afs_file_status status; /* AFS status info for this file */ afs_dataversion_t invalid_before; /* Child dentries are invalid before this */ struct afs_permits __rcu *permit_cache; /* cache of permits so far obtained */ - struct mutex io_lock; /* Lock for serialising I/O on this mutex */ + struct list_head io_lock_waiters; /* Threads waiting for the I/O lock */ struct rw_semaphore validate_lock; /* lock for validating this vnode */ struct rw_semaphore rmdir_lock; /* Lock for rmdir vs sillyrename */ struct key *silly_key; /* Silly rename key */ spinlock_t wb_lock; /* lock for wb_keys */ spinlock_t lock; /* waitqueue/flags lock */ unsigned long flags; +#define AFS_VNODE_IO_LOCK 0 /* Set if the I/O serialisation lock is held */ #define AFS_VNODE_UNSET 1 /* set if vnode attributes not yet set */ #define AFS_VNODE_DIR_VALID 2 /* Set if dir contents are valid */ #define AFS_VNODE_ZAP_DATA 3 /* set if vnode's data should be invalidated */ diff --git a/fs/afs/super.c b/fs/afs/super.c index f3ba1c3e72f5..7631302c1984 100644 --- a/fs/afs/super.c +++ b/fs/afs/super.c @@ -663,7 +663,7 @@ static void afs_i_init_once(void *_vnode) memset(vnode, 0, sizeof(*vnode)); inode_init_once(&vnode->netfs.inode); - mutex_init(&vnode->io_lock); + INIT_LIST_HEAD(&vnode->io_lock_waiters); init_rwsem(&vnode->validate_lock); spin_lock_init(&vnode->wb_lock); spin_lock_init(&vnode->lock);