From patchwork Fri Oct 25 20:39:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13851565 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E95DAD149F0 for ; Fri, 25 Oct 2024 20:41:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 79A2A6B00A8; Fri, 25 Oct 2024 16:41:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 723D06B00A9; Fri, 25 Oct 2024 16:41:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5506B6B00AA; Fri, 25 Oct 2024 16:41:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 2C87E6B00A8 for ; Fri, 25 Oct 2024 16:41:50 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id ED6D714113F for ; Fri, 25 Oct 2024 20:41:28 +0000 (UTC) X-FDA: 82713295224.06.6ABAF2E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf27.hostedemail.com (Postfix) with ESMTP id D5D764000E for ; Fri, 25 Oct 2024 20:41:27 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dDMb2q3A; spf=pass (imf27.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729888752; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RVoEs9N2N3+Yy5ytz8xwWGrwrjCUqjBQVK6VxNHCBSE=; b=YuxW6o0VDotERP/PDvwRUc0wMyljhinieYwpaKeX26c7gkNQzgSajUNOnVXz1Hf1j5cpUZ N0qoloTiMqsYYSeDDckcUfD5EBVkP6m+atqIBLDDhDJ7Yshrm/NybpfAV+AVu+bltMJI2h rEdLtIvIhJcGHZgcQroHHuV6SZmIyck= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729888752; a=rsa-sha256; cv=none; b=c5i7DoUMzIUiHM0C4KqBuD688VYMbsIH7tUtY1YpnCb5ozyI11WI+4XgnUiAuCNFS3fur3 xeGLM+b1gQQRD60y1bLpzyOBtrXsLQDV59CsRRR7xMcFgqsiP7a/aIiXP865Hsck09YRmk /zhvF3/eHUOrxlYaZzAynWztPUKXgOg= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dDMb2q3A; spf=pass (imf27.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1729888907; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RVoEs9N2N3+Yy5ytz8xwWGrwrjCUqjBQVK6VxNHCBSE=; b=dDMb2q3ARD/meDiQwh/4N6KMw0M1QgES25kVHqI1a/kou4N5+7bgikCselY4md9zSd+DBp GFK6kq9fI5FN6aJ23JNKO6Fgf5y3e+CipQ8WHP31nbqAjYScBrpHtONTvwSgQ5M8LM3LCA 8UmOpLQREGyd53NHnkFa9yQwGl6F7TU= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-57-55EgSfbFMoajSZAVz0ac0g-1; Fri, 25 Oct 2024 16:41:44 -0400 X-MC-Unique: 55EgSfbFMoajSZAVz0ac0g-1 Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 15F4D19560BD; Fri, 25 Oct 2024 20:41:42 +0000 (UTC) Received: from warthog.procyon.org.uk.com (unknown [10.42.28.231]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 3BF7A1956056; Fri, 25 Oct 2024 20:41:37 +0000 (UTC) From: David Howells To: Christian Brauner , Steve French , Matthew Wilcox Cc: David Howells , Jeff Layton , Gao Xiang , Dominique Martinet , Marc Dionne , Paulo Alcantara , Shyam Prasad N , Tom Talpey , Eric Van Hensbergen , Ilya Dryomov , netfs@lists.linux.dev, linux-afs@lists.infradead.org, linux-cifs@vger.kernel.org, linux-nfs@vger.kernel.org, ceph-devel@vger.kernel.org, v9fs@lists.linux.dev, linux-erofs@lists.ozlabs.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 12/31] afs: Don't use mutex for I/O operation lock Date: Fri, 25 Oct 2024 21:39:39 +0100 Message-ID: <20241025204008.4076565-13-dhowells@redhat.com> In-Reply-To: <20241025204008.4076565-1-dhowells@redhat.com> References: <20241025204008.4076565-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 X-Stat-Signature: gehidqyqrgdpfheypos679af599baiht X-Rspamd-Queue-Id: D5D764000E X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1729888887-20496 X-HE-Meta: U2FsdGVkX1/z8ImTZR/8eC6cV6qXCXDgnooPFaTqNUSHg8Y1cLwphjddwNi0CuMSotFTM7FobvGLUe3YmI7PjGnA+TC0j7fRY4gi098dNz+XjCBHlTM1MauIKSjUW8yL19qPCwbtLl0VL++kQub+C7Dr+U+Bc/X8py5w5nwr3B43kn9r9ou8GnfJUY3uuBomh9dMYNf9TmbGx/ePRaHWiwXuTCwAJE6K8jk9k1n1FRvJABNxMb+bZke7KYKBQmnZ8DQM7vStipwZ54SQRmU/76iLGOTU9RyZUAyD4sJ6pfc4V5ZskDj/VTLBx0b5qwmacrOgaDAR5tyiak5VQPvUAuVxN+iFw2Zp16h3OPsa7utD3KDUe99VQskyaXiOEX311zsFzF5wzooKIn5ZtI446z5FtCk4C3zK6YHoVbm/JA6h/IEMXbSBduLCNPgjI1zFs5gu4wAUXgRoYyHJ4sbrbFzOL5DmCqdBAgVPtSb95yd8kqYshMPLu7+39mxXzZO5DlfJG6IPxlEp01EMVhqTqZ6simqK9UIrcxXdbs0a7lKWNHj0rWOM/K6XFervq9vklULnUhivoAruMcNS/3HkZ1U4DbENcFloV9pKnKrAdNgmFhex5PYsS7HwryLhA9sL6I2oHFvTvVX3Fq9Zj+v/ehML0d2KPDOcr9K/1oaSkiBIs/rc+BInTh7OY5MLg63yBh/tMg8A7tbWD+8ZVDwJUumiYClx9o8VQIePDL4kttmV26MQAjYRgwQIpOjBv4FscC5uA37K/1mHhigxYskcto7DSXwVXgs+oqpDRSH7skHl9HpGMFjRlk1jY2EXSPb1gL/RhB05IAgMZL4WoMTvHllIOmSQ7kkTV3nUH3pWWjpacaSl11PtZf4Nr2YH3fi2/quUDLxOLba+6vMSDJHsfWhX3KpzChBuVbE5uXxEFXSsXw+A9ROVfOeIG9giuPA5BJbJ2M+31Eo5U50++UT JYu40nSu +Ex6S4a1QB4WP2TAlZofriAW2/I7FK9HLmQoFkY4PUNIxnhKpUUi3FT1gwxO0VWZJPa2mbfNi38NQ4abPuuuHe+WzwQJuztNNhOyFmDAGT5PGx0AAOrTA7WcgS9t2jhCz5/ZeiauH8TTkgkClNRew9PrZ4afNf62xTCb/+OrPKNUjASwrgOVdGI63QmOjwyAIWi73hoN6KE+x2O5Dyh/wIQDczeHcQzfWptz6J+gnCdOfPR2PrVmXXYd5hwIgiitpp3Y9GfQJo4jx0TP4KIZanq2DQDSIF+LLniYjNi78NWDZbRLWJmhICwTtAcllpOIpDgE1MI5cP5n6NEKeg/D2L4yG2JI0RkPazHUouTXaciep2lZQalHqgMPhejqs82rF8rwWPoNf64gXn63BwV09LJLCqJuHJmM1NBEotxXSAOKa5HeXIrZuz0zJojnP2HrbZ8pQ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Don't use the standard mutex for the I/O operation lock, but rather implement our own as the standard mutex must be released in the same thread as locked it. This is a problem when it comes to doing async FetchData where the lock will be dropped from the workqueue that processed the incoming data and not from the issuing thread. Signed-off-by: David Howells cc: Marc Dionne cc: linux-afs@lists.infradead.org --- fs/afs/fs_operation.c | 111 +++++++++++++++++++++++++++++++++++++++--- fs/afs/internal.h | 3 +- fs/afs/super.c | 2 +- 3 files changed, 108 insertions(+), 8 deletions(-) diff --git a/fs/afs/fs_operation.c b/fs/afs/fs_operation.c index 428721bbe4f6..8488ff8183fa 100644 --- a/fs/afs/fs_operation.c +++ b/fs/afs/fs_operation.c @@ -49,6 +49,105 @@ struct afs_operation *afs_alloc_operation(struct key *key, struct afs_volume *vo return op; } +struct afs_io_locker { + struct list_head link; + struct task_struct *task; + unsigned long have_lock; +}; + +/* + * Unlock the I/O lock on a vnode. + */ +static void afs_unlock_for_io(struct afs_vnode *vnode) +{ + struct afs_io_locker *locker; + + spin_lock(&vnode->lock); + locker = list_first_entry_or_null(&vnode->io_lock_waiters, + struct afs_io_locker, link); + if (locker) { + list_del(&locker->link); + smp_store_release(&locker->have_lock, 1); + smp_mb__after_atomic(); /* Store have_lock before task state */ + wake_up_process(locker->task); + } else { + clear_bit(AFS_VNODE_IO_LOCK, &vnode->flags); + } + spin_unlock(&vnode->lock); +} + +/* + * Lock the I/O lock on a vnode uninterruptibly. We can't use an ordinary + * mutex as lockdep will complain if we unlock it in the wrong thread. + */ +static void afs_lock_for_io(struct afs_vnode *vnode) +{ + struct afs_io_locker myself = { .task = current, }; + + spin_lock(&vnode->lock); + + if (!test_and_set_bit(AFS_VNODE_IO_LOCK, &vnode->flags)) { + spin_unlock(&vnode->lock); + return; + } + + list_add_tail(&myself.link, &vnode->io_lock_waiters); + spin_unlock(&vnode->lock); + + for (;;) { + set_current_state(TASK_UNINTERRUPTIBLE); + if (smp_load_acquire(&myself.have_lock)) + break; + schedule(); + } + __set_current_state(TASK_RUNNING); +} + +/* + * Lock the I/O lock on a vnode interruptibly. We can't use an ordinary mutex + * as lockdep will complain if we unlock it in the wrong thread. + */ +static int afs_lock_for_io_interruptible(struct afs_vnode *vnode) +{ + struct afs_io_locker myself = { .task = current, }; + int ret = 0; + + spin_lock(&vnode->lock); + + if (!test_and_set_bit(AFS_VNODE_IO_LOCK, &vnode->flags)) { + spin_unlock(&vnode->lock); + return 0; + } + + list_add_tail(&myself.link, &vnode->io_lock_waiters); + spin_unlock(&vnode->lock); + + for (;;) { + set_current_state(TASK_INTERRUPTIBLE); + if (smp_load_acquire(&myself.have_lock) || + signal_pending(current)) + break; + schedule(); + } + __set_current_state(TASK_RUNNING); + + /* If we got a signal, try to transfer the lock onto the next + * waiter. + */ + if (unlikely(signal_pending(current))) { + spin_lock(&vnode->lock); + if (myself.have_lock) { + spin_unlock(&vnode->lock); + afs_unlock_for_io(vnode); + } else { + list_del(&myself.link); + spin_unlock(&vnode->lock); + } + ret = -ERESTARTSYS; + } + return ret; +} + /* * Lock the vnode(s) being operated upon. */ @@ -60,7 +159,7 @@ static bool afs_get_io_locks(struct afs_operation *op) _enter(""); if (op->flags & AFS_OPERATION_UNINTR) { - mutex_lock(&vnode->io_lock); + afs_lock_for_io(vnode); op->flags |= AFS_OPERATION_LOCK_0; _leave(" = t [1]"); return true; @@ -72,7 +171,7 @@ static bool afs_get_io_locks(struct afs_operation *op) if (vnode2 > vnode) swap(vnode, vnode2); - if (mutex_lock_interruptible(&vnode->io_lock) < 0) { + if (afs_lock_for_io_interruptible(vnode) < 0) { afs_op_set_error(op, -ERESTARTSYS); op->flags |= AFS_OPERATION_STOP; _leave(" = f [I 0]"); @@ -81,10 +180,10 @@ static bool afs_get_io_locks(struct afs_operation *op) op->flags |= AFS_OPERATION_LOCK_0; if (vnode2) { - if (mutex_lock_interruptible_nested(&vnode2->io_lock, 1) < 0) { + if (afs_lock_for_io_interruptible(vnode2) < 0) { afs_op_set_error(op, -ERESTARTSYS); op->flags |= AFS_OPERATION_STOP; - mutex_unlock(&vnode->io_lock); + afs_unlock_for_io(vnode); op->flags &= ~AFS_OPERATION_LOCK_0; _leave(" = f [I 1]"); return false; @@ -104,9 +203,9 @@ static void afs_drop_io_locks(struct afs_operation *op) _enter(""); if (op->flags & AFS_OPERATION_LOCK_1) - mutex_unlock(&vnode2->io_lock); + afs_unlock_for_io(vnode2); if (op->flags & AFS_OPERATION_LOCK_0) - mutex_unlock(&vnode->io_lock); + afs_unlock_for_io(vnode); } static void afs_prepare_vnode(struct afs_operation *op, struct afs_vnode_param *vp, diff --git a/fs/afs/internal.h b/fs/afs/internal.h index c9d620175e80..07b8f7083e73 100644 --- a/fs/afs/internal.h +++ b/fs/afs/internal.h @@ -702,13 +702,14 @@ struct afs_vnode { struct afs_file_status status; /* AFS status info for this file */ afs_dataversion_t invalid_before; /* Child dentries are invalid before this */ struct afs_permits __rcu *permit_cache; /* cache of permits so far obtained */ - struct mutex io_lock; /* Lock for serialising I/O on this mutex */ + struct list_head io_lock_waiters; /* Threads waiting for the I/O lock */ struct rw_semaphore validate_lock; /* lock for validating this vnode */ struct rw_semaphore rmdir_lock; /* Lock for rmdir vs sillyrename */ struct key *silly_key; /* Silly rename key */ spinlock_t wb_lock; /* lock for wb_keys */ spinlock_t lock; /* waitqueue/flags lock */ unsigned long flags; +#define AFS_VNODE_IO_LOCK 0 /* Set if the I/O serialisation lock is held */ #define AFS_VNODE_UNSET 1 /* set if vnode attributes not yet set */ #define AFS_VNODE_DIR_VALID 2 /* Set if dir contents are valid */ #define AFS_VNODE_ZAP_DATA 3 /* set if vnode's data should be invalidated */ diff --git a/fs/afs/super.c b/fs/afs/super.c index f3ba1c3e72f5..7631302c1984 100644 --- a/fs/afs/super.c +++ b/fs/afs/super.c @@ -663,7 +663,7 @@ static void afs_i_init_once(void *_vnode) memset(vnode, 0, sizeof(*vnode)); inode_init_once(&vnode->netfs.inode); - mutex_init(&vnode->io_lock); + INIT_LIST_HEAD(&vnode->io_lock_waiters); init_rwsem(&vnode->validate_lock); spin_lock_init(&vnode->wb_lock); spin_lock_init(&vnode->lock);