From patchwork Tue Nov 19 00:27:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 13879258 Received: from 009.lax.mailroute.net (009.lax.mailroute.net [199.89.1.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 821231AAC4; Tue, 19 Nov 2024 00:28:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=199.89.1.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976116; cv=none; b=nLU5jFqHumVHnBH0wHQXdTh+Dw697XDitju+41UGt8EslI47vfMPqVVYo7sJdkSqW8R3UjslEtX4bB58dDBgv79GnYb2FTvBcjLaII7DZ7eV9FjzZaB3kVzGBQsrfSsrGNJKRhgRv/srA98IwpLNyYzPXFD8zurguKcVjonWrwg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976116; c=relaxed/simple; bh=Bh5/UK4HYerQo8SKIvp0FyEnvJpwkWV1/JntUGWwrZA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gos4t/kclqhLJyBNhjCHpz1xtZmtAQGACF7A+4W94FWE/sVL3IWCzB8vXEhfko25/x9ZsiJBfZli7laT5ec3qWI17ksZF+umDXYVL+hWgdRca1wsTam0D7d4ersGnq3+OGA3QXfyyDhW1+dJl/jG+kT9Kq+0n/yPFcRdfZDumfI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org; spf=pass smtp.mailfrom=acm.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b=f2R/m5Aj; arc=none smtp.client-ip=199.89.1.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=acm.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b="f2R/m5Aj" Received: from localhost (localhost [127.0.0.1]) by 009.lax.mailroute.net (Postfix) with ESMTP id 4Xslhx5GwkzlgT1K; Tue, 19 Nov 2024 00:28:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=acm.org; h= content-transfer-encoding:mime-version:references:in-reply-to :x-mailer:message-id:date:date:subject:subject:from:from :received:received; s=mr01; t=1731976111; x=1734568112; bh=hBVx3 Uq/JzR2vP6uuT26+jeD5kDe4DFlRoEVPfAYVNo=; b=f2R/m5Aj8Fxa3P9hoyZs7 20+xFMaFY+oUcuxm12PM44O8BBAzz2VBEewcyk7y2fUu2qxehJs6KlHHmY1jBRb5 K3DQc9PHT87Q8msuL+jEKXulcrpbE2rJ25Jm95juBQe+8zTbZTItMgioyljMd5MJ eJyh2mCBC3Je1iWHQydN1TauGH98+PxkJ/82S0x6RVwcMX6VXdde/tOhISaLRFI0 w6P+QqQUoBEjtwBHslG0dV5QBw4+tSylwGYr6B/LL33ofShh1er5tQNjnI/gjKdj jzF/QyY+4fEWQ8thfc0jdf0TZ29yT6rnuz1mBdsQ7Po/fRsdeS+WrrvTZs8q6rAU g== X-Virus-Scanned: by MailRoute Received: from 009.lax.mailroute.net ([127.0.0.1]) by localhost (009.lax [127.0.0.1]) (mroute_mailscanner, port 10029) with LMTP id fwipjbrtLXOb; Tue, 19 Nov 2024 00:28:31 +0000 (UTC) Received: from bvanassche.mtv.corp.google.com (unknown [104.135.204.82]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: bvanassche@acm.org) by 009.lax.mailroute.net (Postfix) with ESMTPSA id 4Xslhs5tYPzlgVXv; Tue, 19 Nov 2024 00:28:29 +0000 (UTC) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, Christoph Hellwig , Damien Le Moal , Jaegeuk Kim , Bart Van Assche Subject: [PATCH v16 01/26] blk-zoned: Fix a reference count leak Date: Mon, 18 Nov 2024 16:27:50 -0800 Message-ID: <20241119002815.600608-2-bvanassche@acm.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241119002815.600608-1-bvanassche@acm.org> References: <20241119002815.600608-1-bvanassche@acm.org> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Fix a reference count leak in disk_zone_wplug_handle_error() Fixes: dd291d77cc90 ("block: Introduce zone write plugging") Signed-off-by: Bart Van Assche --- block/blk-zoned.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/block/blk-zoned.c b/block/blk-zoned.c index 70211751df16..3346b8c53605 100644 --- a/block/blk-zoned.c +++ b/block/blk-zoned.c @@ -1337,6 +1337,8 @@ static void disk_zone_wplug_handle_error(struct gendisk *disk, unlock: spin_unlock_irqrestore(&zwplug->lock, flags); + + disk_put_zone_wplug(zwplug); } static void disk_zone_wplugs_work(struct work_struct *work) From patchwork Tue Nov 19 00:27:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 13879259 Received: from 009.lax.mailroute.net (009.lax.mailroute.net [199.89.1.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 79734175B1; Tue, 19 Nov 2024 00:28:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=199.89.1.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976119; cv=none; b=aV/8A9RSj+RzbPpEC+iZYGO23LGy5VwDvT8uAiBaiqwbdykLJ25q+bPw16Qcm/1OcIZ4rQ/JQWNUGZ+URFYpEME7lMmwgjR02ANvZjUEjN5x7WcT2yvOtW2Frb2Aunn0AaMryXU9HD5kqd3K8hSp9kHHPU5+sqUZFIZD0n3Qies= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976119; c=relaxed/simple; bh=ryj1LrIqHXicjHqRFWaCVhEPe6TL2T4m6y6qWp3jn6U=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DI9IDAhIXw8DYMsxnIukN8G5S4ydnFjxxwrWUXWW4eCG+RBE3SYXGpIvEafXzpqEniwuv0PlBr5jOSmYBklp1Ib3OU8+exxBCgBsNjtD1fIJAP0Xq3f2V4x0uZ483uGhJrwozJ9Jni/2mtOWqmxvSL42QPh2Dym+XjPK8clixJc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org; spf=pass smtp.mailfrom=acm.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b=wkVLnOY4; arc=none smtp.client-ip=199.89.1.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=acm.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b="wkVLnOY4" Received: from localhost (localhost [127.0.0.1]) by 009.lax.mailroute.net (Postfix) with ESMTP id 4Xslj06pjfzlgMVN; Tue, 19 Nov 2024 00:28:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=acm.org; h= content-transfer-encoding:mime-version:references:in-reply-to :x-mailer:message-id:date:date:subject:subject:from:from :received:received; s=mr01; t=1731976113; x=1734568114; bh=HyWas zR82zexjAxfSqNY6XuWsqw1mUnzF27e1iD6/9Q=; b=wkVLnOY4bU6Elq87TiVOb /d+fBcpawWBEEDneaLD9DCiM9ac/s9QoaNNHkVu1MNMznLkm64zZnQynHpPoJ4k2 ooP2T3b1sskr3C8O2c+RDUPqXOakGCwYvYICYcjcby1y5N4Xrt/zUkSmdbeIXoQq OopJBQz/MyGoisa01pLk8AzQxQkWKmT+3FxzKd35aXIPOHcUKbndOQg7fw/F2MKa WoZqXA6e4vg3Ghq28nc7Abb398Gu9sKvEdktYPTXsJjn5SQE/SHiID2MTuPrDVbE Indrm7V+zxAGtsZOeebsfIqKPVNr3GzFtSPDDjqMdHi9pJI6VyiZrU81BthTgyyf w== X-Virus-Scanned: by MailRoute Received: from 009.lax.mailroute.net ([127.0.0.1]) by localhost (009.lax [127.0.0.1]) (mroute_mailscanner, port 10029) with LMTP id waDMZ7pLB9wz; Tue, 19 Nov 2024 00:28:33 +0000 (UTC) Received: from bvanassche.mtv.corp.google.com (unknown [104.135.204.82]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: bvanassche@acm.org) by 009.lax.mailroute.net (Postfix) with ESMTPSA id 4Xslhv3Wj6zlgVnN; Tue, 19 Nov 2024 00:28:31 +0000 (UTC) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, Christoph Hellwig , Damien Le Moal , Jaegeuk Kim , Bart Van Assche Subject: [PATCH v16 02/26] blk-zoned: Split disk_zone_wplugs_work() Date: Mon, 18 Nov 2024 16:27:51 -0800 Message-ID: <20241119002815.600608-3-bvanassche@acm.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241119002815.600608-1-bvanassche@acm.org> References: <20241119002815.600608-1-bvanassche@acm.org> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Prepare for adding a second disk_zone_process_err_list() call. Signed-off-by: Bart Van Assche --- block/blk-zoned.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/block/blk-zoned.c b/block/blk-zoned.c index 3346b8c53605..e5c8ab1eab9f 100644 --- a/block/blk-zoned.c +++ b/block/blk-zoned.c @@ -1341,10 +1341,8 @@ static void disk_zone_wplug_handle_error(struct gendisk *disk, disk_put_zone_wplug(zwplug); } -static void disk_zone_wplugs_work(struct work_struct *work) +static void disk_zone_process_err_list(struct gendisk *disk) { - struct gendisk *disk = - container_of(work, struct gendisk, zone_wplugs_work); struct blk_zone_wplug *zwplug; unsigned long flags; @@ -1365,6 +1363,14 @@ static void disk_zone_wplugs_work(struct work_struct *work) spin_unlock_irqrestore(&disk->zone_wplugs_lock, flags); } +static void disk_zone_wplugs_work(struct work_struct *work) +{ + struct gendisk *disk = + container_of(work, struct gendisk, zone_wplugs_work); + + disk_zone_process_err_list(disk); +} + static inline unsigned int disk_zone_wplugs_hash_size(struct gendisk *disk) { return 1U << disk->zone_wplugs_hash_bits; From patchwork Tue Nov 19 00:27:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 13879261 Received: from 009.lax.mailroute.net (009.lax.mailroute.net [199.89.1.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B6DBE1AAC4; Tue, 19 Nov 2024 00:28:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=199.89.1.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976121; cv=none; b=K7NcoLzyREFwO1LBNfz1xwf96pfFOgeQr6MH3lslLgr6heJ7cDSbiOh/btQ2BoREqDCsQnaJzfuDQFcHrxxiFFme9ZdJGTXhIfhtqswHcnpLAg5HaiF10W6TaKz996MgKKiRyp8qtRPJPLIYaL8WWLFECwgNhFAtlE2W1ToDfMw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976121; c=relaxed/simple; bh=PjxHhccj2+TAGbA3DmBSp7d2J57ecWn+uCXAsj6mh4k=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=BdiE9PX8xeTq+29X38YsvBlA7NwN7UyOk0PDUdTIWMVeYl+h4odeRQwNOekx4DBTcjgHeNvkeEMk+Q1SzXFgUUXYJvBjxisQC6L7VSatQUg2DNlsQY9nBEA2P36K2Eqfzxx/1PMusYOzbAioCrfjEuhkrmXWccMD0Le8/tCLExA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org; spf=pass smtp.mailfrom=acm.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b=ohkNRHiF; arc=none smtp.client-ip=199.89.1.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=acm.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b="ohkNRHiF" Received: from localhost (localhost [127.0.0.1]) by 009.lax.mailroute.net (Postfix) with ESMTP id 4Xslj21vMpzlgMVP; Tue, 19 Nov 2024 00:28:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=acm.org; h= content-transfer-encoding:mime-version:references:in-reply-to :x-mailer:message-id:date:date:subject:subject:from:from :received:received; s=mr01; t=1731976114; x=1734568115; bh=5tCYU FxX5QB6W+S/nygz6fD4vmkqD7ZMkhY3othO4o4=; b=ohkNRHiFdrU5i+gPhGegX tWt8tAYeezAKqW8Yd5+DgbURaHFF3wz0A2oN+xZkh6cRbBD1JDKpqfns8TOcx1PJ 2NaI7Yt3DGTABgUFMqUIP5iewQse9+B7rToBGYskGhcbNERHIgsD9B68Zy8ezNqd 5fRorevHsugHCU2IIjNOajr+e4IXRHmnBSYoKkCaZ1qH8k2K8GgsSdWmjQ8xVhZL x/AReUOtTqqMDN7KpYbtSDwcAoTUe9xqBQrRenFMS6TpKayAIo3xSlw1rRYLD+gr yFvwGo9Wb0r9GJF6AZzzc15pYpV5393kRMg1B1fwUzdl5mHpHap5jtnrPG26fRS4 g== X-Virus-Scanned: by MailRoute Received: from 009.lax.mailroute.net ([127.0.0.1]) by localhost (009.lax [127.0.0.1]) (mroute_mailscanner, port 10029) with LMTP id noDKav8J8OUX; Tue, 19 Nov 2024 00:28:34 +0000 (UTC) Received: from bvanassche.mtv.corp.google.com (unknown [104.135.204.82]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: bvanassche@acm.org) by 009.lax.mailroute.net (Postfix) with ESMTPSA id 4Xslhx0j5pzlgVnY; Tue, 19 Nov 2024 00:28:32 +0000 (UTC) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, Christoph Hellwig , Damien Le Moal , Jaegeuk Kim , Bart Van Assche Subject: [PATCH v16 03/26] blk-zoned: Split queue_zone_wplugs_show() Date: Mon, 18 Nov 2024 16:27:52 -0800 Message-ID: <20241119002815.600608-4-bvanassche@acm.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241119002815.600608-1-bvanassche@acm.org> References: <20241119002815.600608-1-bvanassche@acm.org> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Reduce the indentation level of the code in queue_zone_wplugs_show() by moving the body of the loop in that function into a new function. Signed-off-by: Bart Van Assche Reviewed-by: Damien Le Moal --- block/blk-zoned.c | 44 ++++++++++++++++++++++++-------------------- 1 file changed, 24 insertions(+), 20 deletions(-) diff --git a/block/blk-zoned.c b/block/blk-zoned.c index e5c8ab1eab9f..7e6e6ebb6235 100644 --- a/block/blk-zoned.c +++ b/block/blk-zoned.c @@ -1838,37 +1838,41 @@ int blk_revalidate_disk_zones(struct gendisk *disk) EXPORT_SYMBOL_GPL(blk_revalidate_disk_zones); #ifdef CONFIG_BLK_DEBUG_FS +static void queue_zone_wplug_show(struct blk_zone_wplug *zwplug, + struct seq_file *m) +{ + unsigned int zwp_wp_offset, zwp_flags; + unsigned int zwp_zone_no, zwp_ref; + unsigned int zwp_bio_list_size; + unsigned long flags; + + spin_lock_irqsave(&zwplug->lock, flags); + zwp_zone_no = zwplug->zone_no; + zwp_flags = zwplug->flags; + zwp_ref = refcount_read(&zwplug->ref); + zwp_wp_offset = zwplug->wp_offset; + zwp_bio_list_size = bio_list_size(&zwplug->bio_list); + spin_unlock_irqrestore(&zwplug->lock, flags); + + seq_printf(m, "%u 0x%x %u %u %u\n", zwp_zone_no, zwp_flags, zwp_ref, + zwp_wp_offset, zwp_bio_list_size); +} int queue_zone_wplugs_show(void *data, struct seq_file *m) { struct request_queue *q = data; struct gendisk *disk = q->disk; struct blk_zone_wplug *zwplug; - unsigned int zwp_wp_offset, zwp_flags; - unsigned int zwp_zone_no, zwp_ref; - unsigned int zwp_bio_list_size, i; - unsigned long flags; + unsigned int i; if (!disk->zone_wplugs_hash) return 0; rcu_read_lock(); - for (i = 0; i < disk_zone_wplugs_hash_size(disk); i++) { - hlist_for_each_entry_rcu(zwplug, - &disk->zone_wplugs_hash[i], node) { - spin_lock_irqsave(&zwplug->lock, flags); - zwp_zone_no = zwplug->zone_no; - zwp_flags = zwplug->flags; - zwp_ref = refcount_read(&zwplug->ref); - zwp_wp_offset = zwplug->wp_offset; - zwp_bio_list_size = bio_list_size(&zwplug->bio_list); - spin_unlock_irqrestore(&zwplug->lock, flags); - - seq_printf(m, "%u 0x%x %u %u %u\n", - zwp_zone_no, zwp_flags, zwp_ref, - zwp_wp_offset, zwp_bio_list_size); - } - } + for (i = 0; i < disk_zone_wplugs_hash_size(disk); i++) + hlist_for_each_entry_rcu(zwplug, &disk->zone_wplugs_hash[i], + node) + queue_zone_wplug_show(zwplug, m); rcu_read_unlock(); return 0; From patchwork Tue Nov 19 00:27:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 13879266 Received: from 009.lax.mailroute.net (009.lax.mailroute.net [199.89.1.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9E75138FA6; Tue, 19 Nov 2024 00:28:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=199.89.1.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976130; cv=none; b=FYv91CZ/Q7PrJSeLL3bykLyH0fYh645/QNCM1DmzNMmi/8Bm36OFi/1iysqPzWQoHy3S4iWGCeBctsN/K13ZV2QXD/prpmGjYb3/KGttYBt1qzwzMm3JfEeFxhCAn1X1PZ+et/O0F4YiKjkAL9sdoV95qKMd/FaGg1qqFEtLq3c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976130; c=relaxed/simple; bh=2HcQmdgi8D0gbNiFbXQIisgIrCR0OGjMywbyhS/VZ+M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=p+bYqlXrI52AaZ4wrUy1mkQSDXKT1f/vSHU42LgC7cpsU1mtLrghy+wuXnclVSbfper/OBdUEQiASNwY3i5rjUQQJIhySNmfzaqvaCmGs7tDd/cUXMGtIJ9y5q2TFGjI2AU3qTd54cfOxUMADC64W3Dn5oXuzBc6jNH7FFhk78g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org; spf=pass smtp.mailfrom=acm.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b=ey/eLbQg; arc=none smtp.client-ip=199.89.1.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=acm.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b="ey/eLbQg" Received: from localhost (localhost [127.0.0.1]) by 009.lax.mailroute.net (Postfix) with ESMTP id 4XsljB4HhyzlgVnY; Tue, 19 Nov 2024 00:28:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=acm.org; h= content-transfer-encoding:mime-version:references:in-reply-to :x-mailer:message-id:date:date:subject:subject:from:from :received:received; s=mr01; t=1731976116; x=1734568117; bh=ApTxr KLlqtUOcY6Efmc/C8s1lOktZwRB41eCGj+9khg=; b=ey/eLbQgNVjWJwXGnTn4v mKxiTwTXMCZFqGcEPNYYv6XJhlFTwE/Kvlh9Eku6pjLUWbfsUvrM13+xv46hyX0U CcTb4ZqAdQBPwLjeAJNqS9xEN5mohIuf1F+2TSg1KArm0Y5lbiPxpG1V9aJIoldj draoHqLVbrpMBWa8pAmiF6PpwRPC0JgrbaWfVMZeTvd0tu/+nzC+KJ0c0ETpg0U/ kiJUoeh5JUAmXnTn5Kozo9Wc4gIkwxfxnw3QDnUs3I0bYJ2h44EaRHOz1QnDAACP kP7t3gukqyQX7y2tgk0lxj9gZbPzY8dQwRi+F3bsUygOBZUzYtrMpakn2iCkS+xt w== X-Virus-Scanned: by MailRoute Received: from 009.lax.mailroute.net ([127.0.0.1]) by localhost (009.lax [127.0.0.1]) (mroute_mailscanner, port 10029) with LMTP id RUa2xrTqqH-7; Tue, 19 Nov 2024 00:28:36 +0000 (UTC) Received: from bvanassche.mtv.corp.google.com (unknown [104.135.204.82]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: bvanassche@acm.org) by 009.lax.mailroute.net (Postfix) with ESMTPSA id 4Xslhy57BMzlgVXv; Tue, 19 Nov 2024 00:28:34 +0000 (UTC) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, Christoph Hellwig , Damien Le Moal , Jaegeuk Kim , Bart Van Assche Subject: [PATCH v16 04/26] blk-zoned: Only handle errors after pending zoned writes have completed Date: Mon, 18 Nov 2024 16:27:53 -0800 Message-ID: <20241119002815.600608-5-bvanassche@acm.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241119002815.600608-1-bvanassche@acm.org> References: <20241119002815.600608-1-bvanassche@acm.org> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Instead of handling write errors immediately, only handle these after all pending zoned write requests have completed or have been requeued. This patch prepares for changing the zone write pointer tracking approach. Signed-off-by: Bart Van Assche --- block/blk-mq.c | 9 +++ block/blk-zoned.c | 154 +++++++++++++++++++++++++++++++++++++++-- block/blk.h | 29 ++++++++ include/linux/blk-mq.h | 18 +++++ 4 files changed, 203 insertions(+), 7 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 270cfd9fc6b0..a45077e187b5 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -793,6 +793,9 @@ void blk_mq_free_request(struct request *rq) rq_qos_done(q, rq); WRITE_ONCE(rq->state, MQ_RQ_IDLE); + + blk_zone_free_request(rq); + if (req_ref_put_and_test(rq)) __blk_mq_free_request(rq); } @@ -1189,6 +1192,9 @@ void blk_mq_end_request_batch(struct io_comp_batch *iob) continue; WRITE_ONCE(rq->state, MQ_RQ_IDLE); + + blk_zone_free_request(rq); + if (!req_ref_put_and_test(rq)) continue; @@ -1507,6 +1513,7 @@ static void __blk_mq_requeue_request(struct request *rq) if (blk_mq_request_started(rq)) { WRITE_ONCE(rq->state, MQ_RQ_IDLE); rq->rq_flags &= ~RQF_TIMED_OUT; + blk_zone_requeue_work(q); } } @@ -1542,6 +1549,8 @@ static void blk_mq_requeue_work(struct work_struct *work) list_splice_init(&q->flush_list, &flush_list); spin_unlock_irq(&q->requeue_lock); + blk_zone_requeue_work(q); + while (!list_empty(&rq_list)) { rq = list_entry(rq_list.next, struct request, queuelist); /* diff --git a/block/blk-zoned.c b/block/blk-zoned.c index 7e6e6ebb6235..b570b773e65f 100644 --- a/block/blk-zoned.c +++ b/block/blk-zoned.c @@ -608,6 +608,8 @@ static inline void disk_zone_wplug_set_error(struct gendisk *disk, if (zwplug->flags & BLK_ZONE_WPLUG_ERROR) return; + zwplug->flags |= BLK_ZONE_WPLUG_PLUGGED; + zwplug->flags |= BLK_ZONE_WPLUG_ERROR; /* * At this point, we already have a reference on the zone write plug. * However, since we are going to add the plug to the disk zone write @@ -616,7 +618,6 @@ static inline void disk_zone_wplug_set_error(struct gendisk *disk, * handled, or in disk_zone_wplug_clear_error() if the zone is reset or * finished. */ - zwplug->flags |= BLK_ZONE_WPLUG_ERROR; refcount_inc(&zwplug->ref); spin_lock_irqsave(&disk->zone_wplugs_lock, flags); @@ -642,6 +643,7 @@ static inline void disk_zone_wplug_clear_error(struct gendisk *disk, spin_lock_irqsave(&disk->zone_wplugs_lock, flags); if (!list_empty(&zwplug->link)) { list_del_init(&zwplug->link); + zwplug->flags &= ~BLK_ZONE_WPLUG_PLUGGED; zwplug->flags &= ~BLK_ZONE_WPLUG_ERROR; disk_put_zone_wplug(zwplug); } @@ -746,6 +748,70 @@ static bool blk_zone_wplug_handle_reset_all(struct bio *bio) return false; } +struct all_zwr_inserted_data { + struct blk_zone_wplug *zwplug; + bool res; +}; + +/* + * Changes @data->res to %false if and only if @rq is a zoned write for the + * given zone and if it is owned by the block driver. + * + * @rq members may change while this function is in progress. Hence, use + * READ_ONCE() to read @rq members. + */ +static bool blk_zwr_inserted(struct request *rq, void *data) +{ + struct all_zwr_inserted_data *d = data; + struct blk_zone_wplug *zwplug = d->zwplug; + struct request_queue *q = zwplug->disk->queue; + struct bio *bio = READ_ONCE(rq->bio); + + if (rq->q == q && READ_ONCE(rq->state) != MQ_RQ_IDLE && + blk_rq_is_seq_zoned_write(rq) && bio && + bio_zone_no(bio) == zwplug->zone_no) { + d->res = false; + return false; + } + + return true; +} + +/* + * Report whether or not all zoned writes for a zone have been inserted into a + * software queue, elevator queue or hardware queue. + */ +static bool blk_zone_all_zwr_inserted(struct blk_zone_wplug *zwplug) +{ + struct gendisk *disk = zwplug->disk; + struct request_queue *q = disk->queue; + struct all_zwr_inserted_data d = { .zwplug = zwplug, .res = true }; + struct blk_mq_hw_ctx *hctx; + long unsigned int i; + struct request *rq; + + scoped_guard(spinlock_irqsave, &q->requeue_lock) { + list_for_each_entry(rq, &q->requeue_list, queuelist) + if (blk_rq_is_seq_zoned_write(rq) && + bio_zone_no(rq->bio) == zwplug->zone_no) + return false; + list_for_each_entry(rq, &q->flush_list, queuelist) + if (blk_rq_is_seq_zoned_write(rq) && + bio_zone_no(rq->bio) == zwplug->zone_no) + return false; + } + + queue_for_each_hw_ctx(q, hctx, i) { + struct blk_mq_tags *tags = hctx->sched_tags ?: hctx->tags; + + blk_mq_all_tag_iter(tags, blk_zwr_inserted, &d); + if (!d.res || blk_mq_is_shared_tags(q->tag_set->flags)) + break; + } + + return d.res; +} + static inline void blk_zone_wplug_add_bio(struct blk_zone_wplug *zwplug, struct bio *bio, unsigned int nr_segs) { @@ -1096,6 +1162,29 @@ static void disk_zone_wplug_schedule_bio_work(struct gendisk *disk, queue_work(disk->zone_wplugs_wq, &zwplug->bio_work); } +/* + * Change the zone state to "error" if a request is requeued to postpone + * processing of requeued requests until all pending requests have either + * completed or have been requeued. + */ +void blk_zone_write_plug_requeue_request(struct request *rq) +{ + struct gendisk *disk = rq->q->disk; + struct blk_zone_wplug *zwplug; + + if (!disk->zone_wplugs_hash_bits || !blk_rq_is_seq_zoned_write(rq)) + return; + + zwplug = disk_get_zone_wplug(disk, blk_rq_pos(rq)); + if (WARN_ON_ONCE(!zwplug)) + return; + + scoped_guard(spinlock_irqsave, &zwplug->lock) + disk_zone_wplug_set_error(disk, zwplug); + + disk_put_zone_wplug(zwplug); +} + static void disk_zone_wplug_unplug_bio(struct gendisk *disk, struct blk_zone_wplug *zwplug) { @@ -1202,6 +1291,33 @@ void blk_zone_write_plug_finish_request(struct request *req) disk_put_zone_wplug(zwplug); } +/* + * Schedule zone_plugs_work if a zone is in the error state and if no requests + * are in flight. Called from blk_mq_free_request(). + */ +void blk_zone_write_plug_free_request(struct request *rq) +{ + struct gendisk *disk = rq->q->disk; + struct blk_zone_wplug *zwplug; + + /* + * Do nothing if this function is called before the zone information + * has been initialized. + */ + if (!disk->zone_wplugs_hash_bits) + return; + + zwplug = disk_get_zone_wplug(disk, blk_rq_pos(rq)); + + if (!zwplug) + return; + + if (zwplug->flags & BLK_ZONE_WPLUG_ERROR) + kblockd_schedule_work(&disk->zone_wplugs_work); + + disk_put_zone_wplug(zwplug); +} + static void blk_zone_wplug_bio_work(struct work_struct *work) { struct blk_zone_wplug *zwplug = @@ -1343,14 +1459,15 @@ static void disk_zone_wplug_handle_error(struct gendisk *disk, static void disk_zone_process_err_list(struct gendisk *disk) { - struct blk_zone_wplug *zwplug; + struct blk_zone_wplug *zwplug, *next; unsigned long flags; spin_lock_irqsave(&disk->zone_wplugs_lock, flags); - while (!list_empty(&disk->zone_wplugs_err_list)) { - zwplug = list_first_entry(&disk->zone_wplugs_err_list, - struct blk_zone_wplug, link); + list_for_each_entry_safe(zwplug, next, &disk->zone_wplugs_err_list, + link) { + if (!blk_zone_all_zwr_inserted(zwplug)) + continue; list_del_init(&zwplug->link); spin_unlock_irqrestore(&disk->zone_wplugs_lock, flags); @@ -1361,6 +1478,12 @@ static void disk_zone_process_err_list(struct gendisk *disk) } spin_unlock_irqrestore(&disk->zone_wplugs_lock, flags); + + /* + * If one or more zones have been skipped, this work will be requeued + * when a request is requeued (blk_zone_requeue_work()) or freed + * (blk_zone_write_plug_free_request()). + */ } static void disk_zone_wplugs_work(struct work_struct *work) @@ -1371,6 +1494,20 @@ static void disk_zone_wplugs_work(struct work_struct *work) disk_zone_process_err_list(disk); } +/* May be called from interrupt context and hence must not sleep. */ +void blk_zone_requeue_work(struct request_queue *q) +{ + struct gendisk *disk = q->disk; + + if (!disk) + return; + + if (in_interrupt()) + kblockd_schedule_work(&disk->zone_wplugs_work); + else + disk_zone_process_err_list(disk); +} + static inline unsigned int disk_zone_wplugs_hash_size(struct gendisk *disk) { return 1U << disk->zone_wplugs_hash_bits; @@ -1854,8 +1991,11 @@ static void queue_zone_wplug_show(struct blk_zone_wplug *zwplug, zwp_bio_list_size = bio_list_size(&zwplug->bio_list); spin_unlock_irqrestore(&zwplug->lock, flags); - seq_printf(m, "%u 0x%x %u %u %u\n", zwp_zone_no, zwp_flags, zwp_ref, - zwp_wp_offset, zwp_bio_list_size); + bool all_zwr_inserted = blk_zone_all_zwr_inserted(zwplug); + + seq_printf(m, "zone_no %u flags 0x%x ref %u wp_offset %u bio_list_size %u all_zwr_inserted %d\n", + zwp_zone_no, zwp_flags, zwp_ref, zwp_wp_offset, + zwp_bio_list_size, all_zwr_inserted); } int queue_zone_wplugs_show(void *data, struct seq_file *m) diff --git a/block/blk.h b/block/blk.h index 2c26abf505b8..be945db6298d 100644 --- a/block/blk.h +++ b/block/blk.h @@ -473,6 +473,18 @@ static inline void blk_zone_update_request_bio(struct request *rq, if (req_op(rq) == REQ_OP_ZONE_APPEND || bio_zone_write_plugging(bio)) bio->bi_iter.bi_sector = rq->__sector; } + +void blk_zone_write_plug_requeue_request(struct request *rq); +static inline void blk_zone_requeue_request(struct request *rq) +{ + if (!blk_rq_is_seq_zoned_write(rq)) + return; + + blk_zone_write_plug_requeue_request(rq); +} + +void blk_zone_requeue_work(struct request_queue *q); + void blk_zone_write_plug_bio_endio(struct bio *bio); static inline void blk_zone_bio_endio(struct bio *bio) { @@ -490,6 +502,14 @@ static inline void blk_zone_finish_request(struct request *rq) if (rq->rq_flags & RQF_ZONE_WRITE_PLUGGING) blk_zone_write_plug_finish_request(rq); } + +void blk_zone_write_plug_free_request(struct request *rq); +static inline void blk_zone_free_request(struct request *rq) +{ + if (blk_queue_is_zoned(rq->q)) + blk_zone_write_plug_free_request(rq); +} + int blkdev_report_zones_ioctl(struct block_device *bdev, unsigned int cmd, unsigned long arg); int blkdev_zone_mgmt_ioctl(struct block_device *bdev, blk_mode_t mode, @@ -515,12 +535,21 @@ static inline void blk_zone_update_request_bio(struct request *rq, struct bio *bio) { } +static inline void blk_zone_requeue_request(struct request *rq) +{ +} +static inline void blk_zone_requeue_work(struct request_queue *q) +{ +} static inline void blk_zone_bio_endio(struct bio *bio) { } static inline void blk_zone_finish_request(struct request *rq) { } +static inline void blk_zone_free_request(struct request *rq) +{ +} static inline int blkdev_report_zones_ioctl(struct block_device *bdev, unsigned int cmd, unsigned long arg) { diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index c596e0e4cb75..ac05974f08f9 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -1169,4 +1169,22 @@ static inline int blk_rq_map_sg(struct request_queue *q, struct request *rq, } void blk_dump_rq_flags(struct request *, char *); +#ifdef CONFIG_BLK_DEV_ZONED +static inline bool blk_rq_is_seq_zoned_write(struct request *rq) +{ + switch (req_op(rq)) { + case REQ_OP_WRITE: + case REQ_OP_WRITE_ZEROES: + return bdev_zone_is_seq(rq->q->disk->part0, blk_rq_pos(rq)); + default: + return false; + } +} +#else /* CONFIG_BLK_DEV_ZONED */ +static inline bool blk_rq_is_seq_zoned_write(struct request *rq) +{ + return false; +} +#endif /* CONFIG_BLK_DEV_ZONED */ + #endif /* BLK_MQ_H */ From patchwork Tue Nov 19 00:27:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 13879263 Received: from 009.lax.mailroute.net (009.lax.mailroute.net [199.89.1.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D0488A93D; Tue, 19 Nov 2024 00:28:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=199.89.1.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976126; cv=none; b=JKezkuEeRMjouFC6tmHXykhqduZ5AOMJaXnhXwryDp2oGDGI4WN83qMC+QedlGoV1lcoA1NVz7v000q8kbxRBN0Dvtm3WWRsh6G8Ff7mQCfWiMYTq8y9HFtgBxsdH114j6WnP+2hK556VUObjhrfa+Npj+vEwAN5lqLAIPqbA2g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976126; c=relaxed/simple; bh=55CfKNm0m6d840UXU3+xco443Ng4vf0jVKA4Rb3WTKw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DJ2T6gBLVb461UZrQbul2VHtC8PXsrFt5GG1znpFMTN0pI4q8Av+53JsnIO8hRTM60wQIJxlTUHfSpswVA4fQ1PiuJAbVo0YMHRVGFihnT994iocInvlM19LPw6BMuoyjN1HmcKAwF/LTx/8aWJ4lvGgbVSfdkz/nTd6C5nXr1g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org; spf=pass smtp.mailfrom=acm.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b=JbHXPfCA; arc=none smtp.client-ip=199.89.1.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=acm.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b="JbHXPfCA" Received: from localhost (localhost [127.0.0.1]) by 009.lax.mailroute.net (Postfix) with ESMTP id 4Xslj829NkzlgMVQ; Tue, 19 Nov 2024 00:28:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=acm.org; h= content-transfer-encoding:mime-version:references:in-reply-to :x-mailer:message-id:date:date:subject:subject:from:from :received:received; s=mr01; t=1731976117; x=1734568118; bh=stSQS fUIRjbG+lH//eK5AJt7SRvcFvSCywnt/x9hkjU=; b=JbHXPfCATXNWsFYvfo9Lq ES687hZ/4/9llslZl/pC+oD/NL3d9QCyFWVPYbdq8/tQzrYu5jF+4Qt2AaPP5L6g +qk09N3nEmAR6fGs0H4+gNNJCE3n61RlY3jnXeWDFyvQP69Nh3PBH1U6ttwxGYuJ 6OCI+ZlG5V8utbUfcjpuiRjehA1UC4haM7KFrZum9HaWt6DFdcVVz4GCkgwfYVsc zIfzzHniu6zNuXa0WAttG7/RJEZMvAo2xcAWf35lg35Klbnpfz8yeyfGQSidXyHu MFt1kBdqq1GDIBktCM8U/QGe4J5wJ4CNJ1Vo/jsUh4L/S21tYhjdRdcxMWLc+kBO A== X-Virus-Scanned: by MailRoute Received: from 009.lax.mailroute.net ([127.0.0.1]) by localhost (009.lax [127.0.0.1]) (mroute_mailscanner, port 10029) with LMTP id uiu7UU1BmcJ3; Tue, 19 Nov 2024 00:28:37 +0000 (UTC) Received: from bvanassche.mtv.corp.google.com (unknown [104.135.204.82]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: bvanassche@acm.org) by 009.lax.mailroute.net (Postfix) with ESMTPSA id 4Xslj02MD2zlgT1M; Tue, 19 Nov 2024 00:28:36 +0000 (UTC) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, Christoph Hellwig , Damien Le Moal , Jaegeuk Kim , Bart Van Assche Subject: [PATCH v16 05/26] blk-zoned: Fix a deadlock triggered by unaligned writes Date: Mon, 18 Nov 2024 16:27:54 -0800 Message-ID: <20241119002815.600608-6-bvanassche@acm.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241119002815.600608-1-bvanassche@acm.org> References: <20241119002815.600608-1-bvanassche@acm.org> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 If the queue is filled with unaligned writes then the following deadlock occurs: Call Trace: __schedule+0x8cc/0x2190 schedule+0xdd/0x2b0 blk_queue_enter+0x2ce/0x4f0 blk_mq_alloc_request+0x303/0x810 scsi_execute_cmd+0x3f4/0x7b0 sd_zbc_do_report_zones+0x19e/0x4c0 sd_zbc_report_zones+0x304/0x920 disk_zone_wplug_handle_error+0x237/0x920 disk_zone_wplugs_work+0x17e/0x430 process_one_work+0xdd0/0x1490 worker_thread+0x5eb/0x1010 kthread+0x2e5/0x3b0 ret_from_fork+0x3a/0x80 Fix this deadlock by removing the disk->fops->report_zones() call and by deriving the write pointer information from successfully completed zoned writes. Signed-off-by: Bart Van Assche --- block/blk-zoned.c | 69 +++++++++++++++++++---------------------------- block/blk.h | 4 ++- 2 files changed, 31 insertions(+), 42 deletions(-) diff --git a/block/blk-zoned.c b/block/blk-zoned.c index b570b773e65f..284820b29285 100644 --- a/block/blk-zoned.c +++ b/block/blk-zoned.c @@ -56,6 +56,8 @@ static const char *const zone_cond_name[] = { * @zone_no: The number of the zone the plug is managing. * @wp_offset: The zone write pointer location relative to the start of the zone * as a number of 512B sectors. + * @wp_offset_compl: End offset for completed zoned writes as a number of 512 + * byte sectors. * @bio_list: The list of BIOs that are currently plugged. * @bio_work: Work struct to handle issuing of plugged BIOs * @rcu_head: RCU head to free zone write plugs with an RCU grace period. @@ -69,6 +71,7 @@ struct blk_zone_wplug { unsigned int flags; unsigned int zone_no; unsigned int wp_offset; + unsigned int wp_offset_compl; struct bio_list bio_list; struct work_struct bio_work; struct rcu_head rcu_head; @@ -531,6 +534,7 @@ static struct blk_zone_wplug *disk_get_and_lock_zone_wplug(struct gendisk *disk, zwplug->flags = 0; zwplug->zone_no = zno; zwplug->wp_offset = sector & (disk->queue->limits.chunk_sectors - 1); + zwplug->wp_offset_compl = 0; bio_list_init(&zwplug->bio_list); INIT_WORK(&zwplug->bio_work, blk_zone_wplug_bio_work); zwplug->disk = disk; @@ -1226,6 +1230,7 @@ void blk_zone_write_plug_bio_endio(struct bio *bio) struct gendisk *disk = bio->bi_bdev->bd_disk; struct blk_zone_wplug *zwplug = disk_get_zone_wplug(disk, bio->bi_iter.bi_sector); + unsigned int end_sector; unsigned long flags; if (WARN_ON_ONCE(!zwplug)) @@ -1243,11 +1248,24 @@ void blk_zone_write_plug_bio_endio(struct bio *bio) bio->bi_opf |= REQ_OP_ZONE_APPEND; } - /* - * If the BIO failed, mark the plug as having an error to trigger - * recovery. - */ - if (bio->bi_status != BLK_STS_OK) { + if (bio->bi_status == BLK_STS_OK) { + switch (bio_op(bio)) { + case REQ_OP_WRITE: + case REQ_OP_ZONE_APPEND: + case REQ_OP_WRITE_ZEROES: + end_sector = bdev_offset_from_zone_start(disk->part0, + bio->bi_iter.bi_sector + bio_sectors(bio)); + if (end_sector > zwplug->wp_offset_compl) + zwplug->wp_offset_compl = end_sector; + break; + default: + break; + } + } else { + /* + * If the BIO failed, mark the plug as having an error to + * trigger recovery. + */ spin_lock_irqsave(&zwplug->lock, flags); disk_zone_wplug_set_error(disk, zwplug); spin_unlock_irqrestore(&zwplug->lock, flags); @@ -1388,30 +1406,10 @@ static unsigned int blk_zone_wp_offset(struct blk_zone *zone) } } -static int blk_zone_wplug_report_zone_cb(struct blk_zone *zone, - unsigned int idx, void *data) -{ - struct blk_zone *zonep = data; - - *zonep = *zone; - return 0; -} - static void disk_zone_wplug_handle_error(struct gendisk *disk, struct blk_zone_wplug *zwplug) { - sector_t zone_start_sector = - bdev_zone_sectors(disk->part0) * zwplug->zone_no; - unsigned int noio_flag; - struct blk_zone zone; unsigned long flags; - int ret; - - /* Get the current zone information from the device. */ - noio_flag = memalloc_noio_save(); - ret = disk->fops->report_zones(disk, zone_start_sector, 1, - blk_zone_wplug_report_zone_cb, &zone); - memalloc_noio_restore(noio_flag); spin_lock_irqsave(&zwplug->lock, flags); @@ -1425,19 +1423,8 @@ static void disk_zone_wplug_handle_error(struct gendisk *disk, zwplug->flags &= ~BLK_ZONE_WPLUG_ERROR; - if (ret != 1) { - /* - * We failed to get the zone information, meaning that something - * is likely really wrong with the device. Abort all remaining - * plugged BIOs as otherwise we could endup waiting forever on - * plugged BIOs to complete if there is a queue freeze on-going. - */ - disk_zone_wplug_abort(zwplug); - goto unplug; - } - /* Update the zone write pointer offset. */ - zwplug->wp_offset = blk_zone_wp_offset(&zone); + zwplug->wp_offset = zwplug->wp_offset_compl; disk_zone_wplug_abort_unaligned(disk, zwplug); /* Restart BIO submission if we still have any BIO left. */ @@ -1446,7 +1433,6 @@ static void disk_zone_wplug_handle_error(struct gendisk *disk, goto unlock; } -unplug: zwplug->flags &= ~BLK_ZONE_WPLUG_PLUGGED; if (disk_should_remove_zone_wplug(disk, zwplug)) disk_remove_zone_wplug(disk, zwplug); @@ -1978,7 +1964,7 @@ EXPORT_SYMBOL_GPL(blk_revalidate_disk_zones); static void queue_zone_wplug_show(struct blk_zone_wplug *zwplug, struct seq_file *m) { - unsigned int zwp_wp_offset, zwp_flags; + unsigned int zwp_wp_offset, zwp_wp_offset_compl, zwp_flags; unsigned int zwp_zone_no, zwp_ref; unsigned int zwp_bio_list_size; unsigned long flags; @@ -1988,14 +1974,15 @@ static void queue_zone_wplug_show(struct blk_zone_wplug *zwplug, zwp_flags = zwplug->flags; zwp_ref = refcount_read(&zwplug->ref); zwp_wp_offset = zwplug->wp_offset; + zwp_wp_offset_compl = zwplug->wp_offset_compl; zwp_bio_list_size = bio_list_size(&zwplug->bio_list); spin_unlock_irqrestore(&zwplug->lock, flags); bool all_zwr_inserted = blk_zone_all_zwr_inserted(zwplug); - seq_printf(m, "zone_no %u flags 0x%x ref %u wp_offset %u bio_list_size %u all_zwr_inserted %d\n", + seq_printf(m, "zone_no %u flags 0x%x ref %u wp_offset %u wp_offset_compl %u bio_list_size %u all_zwr_inserted %d\n", zwp_zone_no, zwp_flags, zwp_ref, zwp_wp_offset, - zwp_bio_list_size, all_zwr_inserted); + zwp_wp_offset_compl, zwp_bio_list_size, all_zwr_inserted); } int queue_zone_wplugs_show(void *data, struct seq_file *m) diff --git a/block/blk.h b/block/blk.h index be945db6298d..88a6e258eafe 100644 --- a/block/blk.h +++ b/block/blk.h @@ -470,8 +470,10 @@ static inline void blk_zone_update_request_bio(struct request *rq, * the original BIO sector so that blk_zone_write_plug_bio_endio() can * lookup the zone write plug. */ - if (req_op(rq) == REQ_OP_ZONE_APPEND || bio_zone_write_plugging(bio)) + if (req_op(rq) == REQ_OP_ZONE_APPEND || bio_zone_write_plugging(bio)) { bio->bi_iter.bi_sector = rq->__sector; + bio->bi_iter.bi_size = rq->__data_len; + } } void blk_zone_write_plug_requeue_request(struct request *rq); From patchwork Tue Nov 19 00:27:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 13879262 Received: from 009.lax.mailroute.net (009.lax.mailroute.net [199.89.1.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 76C75610B; Tue, 19 Nov 2024 00:28:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=199.89.1.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976124; cv=none; b=oGOOKlO6clDWBYYsM6nYubmiEZcvmGdMMzHmQkEIAkOYI6DigyEEqpU8l9jz8S0JxTPWeTkuJY/Ku5fiUDFC/YL54Y0tg3EcbWRnB9dr9Dt4Q1cwK7Yq+2TZ9hpIt+g6WxVIIUU+t10NNLrSlLYlZO7DLLmNq8F5rJVqvYhGiVM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976124; c=relaxed/simple; bh=dx+OfJ7J7hMc9ed8wbmXnUYBuEtk3Cw23r4yFfCtulA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Ti/nuy3GsCNdiEK1LBLz80fNTBZwMcGRNlZu8qmfLjhYXMVGF0XdCO/gTfPNmWjO/1vyoJ0MbewR+GfiryJxlsuR+CImv3B9qnoAH5TJfgeaEKOLUYWcYuFYybNzhwzWFWtYB7flw4mTt2A0RVJ/PMZSxWtzzdD8zJ4ssCAU51U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org; spf=pass smtp.mailfrom=acm.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b=aNrbEK3d; arc=none smtp.client-ip=199.89.1.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=acm.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b="aNrbEK3d" Received: from localhost (localhost [127.0.0.1]) by 009.lax.mailroute.net (Postfix) with ESMTP id 4Xslj71ZxdzlgVnN; Tue, 19 Nov 2024 00:28:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=acm.org; h= content-transfer-encoding:mime-version:references:in-reply-to :x-mailer:message-id:date:date:subject:subject:from:from :received:received; s=mr01; t=1731976119; x=1734568120; bh=mI8SH BvMkiBxhT++X6Sv40oLp6cQRq+UeWcjKu1tp9I=; b=aNrbEK3dwBafDI9vRt3ZC e22roAu06AEDefe/e4tQM6RkTBaRaDJ0HyWJSm37faiv5XKuDfDoTgHOf2utrwPS k469Dz7TA68o84QHoVp/Y6wBpkPNPHRbKqpcuJbQkxAuvE9/tFWrgSgC2j8gwbOn 6z4UZs4AlcCMc323GJHOtjOIgEL5e2EMvsImcfvjWSoO+rI2uOphAVdtPaQqC48T lLdVO+mC6CFWfqFcg/akXl8UOoLlTkGJHO2G2RGlyz82Bo4B0CCnS2ot9tGQ7b9g /1aVj6wLjSjkwBlNI6mpi+RzCNRYs0eK3wa3hIVVoX+KB2L3rKPwYZBNW2Rx/eqi Q== X-Virus-Scanned: by MailRoute Received: from 009.lax.mailroute.net ([127.0.0.1]) by localhost (009.lax [127.0.0.1]) (mroute_mailscanner, port 10029) with LMTP id RgX2fBwLongY; Tue, 19 Nov 2024 00:28:39 +0000 (UTC) Received: from bvanassche.mtv.corp.google.com (unknown [104.135.204.82]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: bvanassche@acm.org) by 009.lax.mailroute.net (Postfix) with ESMTPSA id 4Xslj16gFvzlgTWQ; Tue, 19 Nov 2024 00:28:37 +0000 (UTC) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, Christoph Hellwig , Damien Le Moal , Jaegeuk Kim , Bart Van Assche Subject: [PATCH v16 06/26] blk-zoned: Fix requeuing of zoned writes Date: Mon, 18 Nov 2024 16:27:55 -0800 Message-ID: <20241119002815.600608-7-bvanassche@acm.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241119002815.600608-1-bvanassche@acm.org> References: <20241119002815.600608-1-bvanassche@acm.org> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Make sure that unaligned writes are sent to the block driver. This allows the block driver to limit the number of retries of unaligned writes. Remove disk_zone_wplug_abort_unaligned() because it only examines the bio plug list. Pending writes can occur either on the bio plug list or on the request queue requeue list. Signed-off-by: Bart Van Assche --- block/blk-zoned.c | 36 ------------------------------------ 1 file changed, 36 deletions(-) diff --git a/block/blk-zoned.c b/block/blk-zoned.c index 284820b29285..ded38fa9ae3d 100644 --- a/block/blk-zoned.c +++ b/block/blk-zoned.c @@ -577,33 +577,6 @@ static void disk_zone_wplug_abort(struct blk_zone_wplug *zwplug) blk_zone_wplug_bio_io_error(zwplug, bio); } -/* - * Abort (fail) all plugged BIOs of a zone write plug that are not aligned - * with the assumed write pointer location of the zone when the BIO will - * be unplugged. - */ -static void disk_zone_wplug_abort_unaligned(struct gendisk *disk, - struct blk_zone_wplug *zwplug) -{ - unsigned int wp_offset = zwplug->wp_offset; - struct bio_list bl = BIO_EMPTY_LIST; - struct bio *bio; - - while ((bio = bio_list_pop(&zwplug->bio_list))) { - if (disk_zone_is_full(disk, zwplug->zone_no, wp_offset) || - (bio_op(bio) != REQ_OP_ZONE_APPEND && - bio_offset_from_zone_start(bio) != wp_offset)) { - blk_zone_wplug_bio_io_error(zwplug, bio); - continue; - } - - wp_offset += bio_sectors(bio); - bio_list_add(&bl, bio); - } - - bio_list_merge(&zwplug->bio_list, &bl); -} - static inline void disk_zone_wplug_set_error(struct gendisk *disk, struct blk_zone_wplug *zwplug) { @@ -982,14 +955,6 @@ static bool blk_zone_wplug_prepare_bio(struct blk_zone_wplug *zwplug, * so that we can restore its operation code on completion. */ bio_set_flag(bio, BIO_EMULATES_ZONE_APPEND); - } else { - /* - * Check for non-sequential writes early because we avoid a - * whole lot of error handling trouble if we don't send it off - * to the driver. - */ - if (bio_offset_from_zone_start(bio) != zwplug->wp_offset) - goto err; } /* Advance the zone write pointer offset. */ @@ -1425,7 +1390,6 @@ static void disk_zone_wplug_handle_error(struct gendisk *disk, /* Update the zone write pointer offset. */ zwplug->wp_offset = zwplug->wp_offset_compl; - disk_zone_wplug_abort_unaligned(disk, zwplug); /* Restart BIO submission if we still have any BIO left. */ if (!bio_list_empty(&zwplug->bio_list)) { From patchwork Tue Nov 19 00:27:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 13879265 Received: from 009.lax.mailroute.net (009.lax.mailroute.net [199.89.1.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 13E3533991; Tue, 19 Nov 2024 00:28:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=199.89.1.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976128; cv=none; b=ev1C9dK+PFUPw/pxThCG+jgxaZ5BwxrxroUZi3Z1jIBMBlfw5FHATBUGYLLob5VQeH4deyAZqnQGdO1lPvMDc0k1fEgA/q/ZZ0Ev/q8UItwDAfFtMJmLTodnqGAbgvssTvRVp9vBn+GubkMyg5Y0aerc3DHrsoBJs4sx16xt8IE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976128; c=relaxed/simple; bh=bMQkL84IxjkXmmPRY6YMAlz/aLP/ZnxOX+IgNNanzjM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cPcQH05OTEGkGILhKrlDdE6dTRZPKkvXIusbG01yDyRnsyn8O3RjAl1E3/pQCiXfDdLDoNcbe/lLGOXz3iKDeD8uZMgF1HfC23GWlnyS9STTWdf1tCAfbvwCEd/Hz0mWAY6pR4/QfC2BJjZTBAE5ouZcZBAQAzrZQN2m6MN5C1s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org; spf=pass smtp.mailfrom=acm.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b=0UwpC/BD; arc=none smtp.client-ip=199.89.1.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=acm.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b="0UwpC/BD" Received: from localhost (localhost [127.0.0.1]) by 009.lax.mailroute.net (Postfix) with ESMTP id 4XsljB2GQLzlgMVR; Tue, 19 Nov 2024 00:28:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=acm.org; h= content-transfer-encoding:mime-version:references:in-reply-to :x-mailer:message-id:date:date:subject:subject:from:from :received:received; s=mr01; t=1731976121; x=1734568122; bh=5SJWf 5W2tyHPoFL/qgha2Dp6Z4cagmvWNWMma2DsGxg=; b=0UwpC/BDj5q586/vN759a lgzzxOC+jI1NvbHM8aloF/ErAqvbLbeyJjraaINqDLRmydAvYvqjhpwz/Q+DMu3J 7WsDYxVza+ha3ILmmYvAdZFw9LzV9yy+MHwNeSiMyCb5CcmSKdLluFg4g9NwqAET AnX3rk8HfOtDskVRoOW/HvPsKVCWoFAvUeFLF74FYXB/xSW0mmdhhPTdeG+R4HBG fjf7U6HbOO/TRX5YCZcQNvFsofHc0K6wju9QdICDIlTSpDDu//Am4w/H1UwzHNzH JTLAmJwtr95G8Tt6Kq2D/FJDIXGpXTeQYV7t+YLPbXScjP0VU6hEVozokkAxs9Eq A== X-Virus-Scanned: by MailRoute Received: from 009.lax.mailroute.net ([127.0.0.1]) by localhost (009.lax [127.0.0.1]) (mroute_mailscanner, port 10029) with LMTP id f4KXGpne-O9z; Tue, 19 Nov 2024 00:28:41 +0000 (UTC) Received: from bvanassche.mtv.corp.google.com (unknown [104.135.204.82]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: bvanassche@acm.org) by 009.lax.mailroute.net (Postfix) with ESMTPSA id 4Xslj33qnnzlgVnY; Tue, 19 Nov 2024 00:28:39 +0000 (UTC) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, Christoph Hellwig , Damien Le Moal , Jaegeuk Kim , Bart Van Assche , Hannes Reinecke , Nitesh Shetty , Ming Lei Subject: [PATCH v16 07/26] block: Support block drivers that preserve the order of write requests Date: Mon, 18 Nov 2024 16:27:56 -0800 Message-ID: <20241119002815.600608-8-bvanassche@acm.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241119002815.600608-1-bvanassche@acm.org> References: <20241119002815.600608-1-bvanassche@acm.org> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Many but not all storage controllers require serialization of zoned writes. Introduce a new request queue limit member variable related to write serialization. 'driver_preserves_write_order' allows block drivers to indicate that the order of write commands is preserved per hardware queue and hence that serialization of writes per zone is not required if all pending writes are submitted to the same hardware queue. Cc: Damien Le Moal Cc: Hannes Reinecke Cc: Nitesh Shetty Cc: Christoph Hellwig Cc: Ming Lei Signed-off-by: Bart Van Assche --- block/blk-settings.c | 2 ++ include/linux/blkdev.h | 5 +++++ 2 files changed, 7 insertions(+) diff --git a/block/blk-settings.c b/block/blk-settings.c index f1d4dfdc37a7..329d8b65a8d7 100644 --- a/block/blk-settings.c +++ b/block/blk-settings.c @@ -633,6 +633,8 @@ int blk_stack_limits(struct queue_limits *t, struct queue_limits *b, } t->max_secure_erase_sectors = min_not_zero(t->max_secure_erase_sectors, b->max_secure_erase_sectors); + t->driver_preserves_write_order = t->driver_preserves_write_order && + b->driver_preserves_write_order; t->zone_write_granularity = max(t->zone_write_granularity, b->zone_write_granularity); if (!(t->features & BLK_FEAT_ZONED)) { diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index a1fd0ddce5cf..72be33d02d1f 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -397,6 +397,11 @@ struct queue_limits { unsigned int max_open_zones; unsigned int max_active_zones; + /* + * Whether or not the block driver preserves the order of write + * requests. Set by the block driver. + */ + bool driver_preserves_write_order; /* * Drivers that set dma_alignment to less than 511 must be prepared to From patchwork Tue Nov 19 00:27:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 13879264 Received: from 009.lax.mailroute.net (009.lax.mailroute.net [199.89.1.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BADCC2BAF4; Tue, 19 Nov 2024 00:28:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=199.89.1.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976128; cv=none; b=GW5ApUuYYpcbUqXJmtQDrqag9PU0jaFUehlyj99cLldrPwSiMkCJf1A10m+rBg80ocBUy3gyAumkNP4fBRbHYT55cEaIzfxKcKesUpBrhelzaVGyRjdUusi65nF8lhBLewAzgOW4ceaK6C6rGz0GU1FRArrJ09cNAJHD23a3TGQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976128; c=relaxed/simple; bh=0aO/tJS1Fk2M9sXF8jIs8qBBa5ynBt417BVu3n/cZIM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rOn59MwPGTVEp6U68bFGgd8wqZl+kqRiCN9YtW2nY0ZocONFxJtCiLGwCrXMF6NtKsIJ4mYRzTJVoHRRhcd0n2R3g7bbV8LgJqMcdjn3+29TetuHJRdBCT889WHmvUKLTwx1prnV4Dq7W12zDJ+efd3xPQzxFozvwPrMNX2PWDE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org; spf=pass smtp.mailfrom=acm.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b=ZNf9kpMW; arc=none smtp.client-ip=199.89.1.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=acm.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b="ZNf9kpMW" Received: from localhost (localhost [127.0.0.1]) by 009.lax.mailroute.net (Postfix) with ESMTP id 4XsljB2C6mzlgMVN; Tue, 19 Nov 2024 00:28:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=acm.org; h= content-transfer-encoding:mime-version:references:in-reply-to :x-mailer:message-id:date:date:subject:subject:from:from :received:received; s=mr01; t=1731976123; x=1734568124; bh=NS2JS s8lR/V97xTiYuhrMQeh4/Cxjgl/9C2s/Uc66TA=; b=ZNf9kpMWmMMGbyu7biT5Y IqjRNnT/s9PcSooJ/3XQvq4WCYeWiWb7tbllvEdeLwfGvz8MXmlS2B3lBJvrY9c6 ce/0HA/dXGt8LJ5e9QNz9ruXqAqhuVrfETgWf6DjozkM8tLfnKxE3R7hdCgMy+Yf YuCh/DdBpy4c9J1UJ82FS7lkNbr15HPojmV+BqKgFfLZUGVkscfpjwXf+80Hy4Y7 Mn144lBp/SSXAxEvoQeQztSpTMEcEQAoOMorgap5I9q62Afbsqg0Tv4n+nphHVg+ SIYupk8POAP63WDYIOsa1eqkBMFtTT86CjFdqXYrh/uXIz1pcDF8dSp0g0RLqXoJ w== X-Virus-Scanned: by MailRoute Received: from 009.lax.mailroute.net ([127.0.0.1]) by localhost (009.lax [127.0.0.1]) (mroute_mailscanner, port 10029) with LMTP id 3ZypB5pine44; Tue, 19 Nov 2024 00:28:43 +0000 (UTC) Received: from bvanassche.mtv.corp.google.com (unknown [104.135.204.82]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: bvanassche@acm.org) by 009.lax.mailroute.net (Postfix) with ESMTPSA id 4Xslj55fvbzlgT1K; Tue, 19 Nov 2024 00:28:41 +0000 (UTC) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, Christoph Hellwig , Damien Le Moal , Jaegeuk Kim , Bart Van Assche Subject: [PATCH v16 08/26] dm-linear: Report to the block layer that the write order is preserved Date: Mon, 18 Nov 2024 16:27:57 -0800 Message-ID: <20241119002815.600608-9-bvanassche@acm.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241119002815.600608-1-bvanassche@acm.org> References: <20241119002815.600608-1-bvanassche@acm.org> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Enable write pipelining if dm-linear is stacked on top of a driver that supports write pipelining. Signed-off-by: Bart Van Assche --- drivers/md/dm-linear.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/drivers/md/dm-linear.c b/drivers/md/dm-linear.c index 49fb0f684193..967fbf856abc 100644 --- a/drivers/md/dm-linear.c +++ b/drivers/md/dm-linear.c @@ -148,6 +148,11 @@ static int linear_report_zones(struct dm_target *ti, #define linear_report_zones NULL #endif +static void linear_io_hints(struct dm_target *ti, struct queue_limits *limits) +{ + limits->driver_preserves_write_order = true; +} + static int linear_iterate_devices(struct dm_target *ti, iterate_devices_callout_fn fn, void *data) { @@ -209,6 +214,7 @@ static struct target_type linear_target = { .map = linear_map, .status = linear_status, .prepare_ioctl = linear_prepare_ioctl, + .io_hints = linear_io_hints, .iterate_devices = linear_iterate_devices, .direct_access = linear_dax_direct_access, .dax_zero_page_range = linear_dax_zero_page_range, From patchwork Tue Nov 19 00:27:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 13879267 Received: from 009.lax.mailroute.net (009.lax.mailroute.net [199.89.1.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2226743AA9; Tue, 19 Nov 2024 00:28:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=199.89.1.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976130; cv=none; b=ZU0XmmZ71qnHpv/A6DJCA9+WV0LAuR+1p0BzTQGA1dI+bSimweTCdulNjhY1xGOjRwkG3r7wfSEq04innosq6On1QVPPl2mu6ZtNCiadHs/j4mYdYwqPnfEXXFSFAXvb/m44snh6CIYz0Vf0uiMS7+yC44aNgDjvydmjSxnioWI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976130; c=relaxed/simple; bh=mdIxfJhSFQPyVT3iSEGDD2aJ3TX4T6DDjjaCvbgFg6w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=J3PNhAiesJlszCFTcXbriMBpXw+sDcbfSbxMaI2q4QeMRvpoCpxjjPCinSnfMOOia/LE2dQGk/HTHFujCmuAEpgYMSKkHGP//To1NMRNU1x7e/FjLe5yl++gdjoOujEjG7RdphOcd3XlK9E5OBH+2VK7fok7/KFiMLTljS/HaZ8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org; spf=pass smtp.mailfrom=acm.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b=KdH5/LU0; arc=none smtp.client-ip=199.89.1.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=acm.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b="KdH5/LU0" Received: from localhost (localhost [127.0.0.1]) by 009.lax.mailroute.net (Postfix) with ESMTP id 4XsljD50SYzlgMVS; Tue, 19 Nov 2024 00:28:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=acm.org; h= content-transfer-encoding:mime-version:references:in-reply-to :x-mailer:message-id:date:date:subject:subject:from:from :received:received; s=mr01; t=1731976124; x=1734568125; bh=QmQfh cd5qbOt9dLDyXYo6dHH1hE32kIzV5l++g1im1s=; b=KdH5/LU0JMXJQ/+XEuRuA VYjk3xHApK4ebWBIMLU18k/oJyBUE55Pgb902y3P2vPF7QWj42r1Of8rLaPB9kDP NyCyfCce4pf0q/7vtg8wJ5puCAYNuK104nZ93hThVohEsNVMpcUqO5SqyUvVYPuf 1LQnMfuTyWoqCi1rNLJfngh6mlCfuK6FaKsrLicys6xTI58uxenWEfxiU75VPKSG /iLkiHb2abUuWFAgDcoM9i1XwlcF3HnukLJ4xkr7pPXP+0p1C2LD+cHaIA4XF83p Bkh9xXQzlcPwuTEBvYFfqd4soPMN1+x3o3xfjAsQwrB6T7P9+ZiXxxv0C4SCf4uK g== X-Virus-Scanned: by MailRoute Received: from 009.lax.mailroute.net ([127.0.0.1]) by localhost (009.lax [127.0.0.1]) (mroute_mailscanner, port 10029) with LMTP id Pvnirn-g1omd; Tue, 19 Nov 2024 00:28:44 +0000 (UTC) Received: from bvanassche.mtv.corp.google.com (unknown [104.135.204.82]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: bvanassche@acm.org) by 009.lax.mailroute.net (Postfix) with ESMTPSA id 4Xslj72vS7zlgTWQ; Tue, 19 Nov 2024 00:28:43 +0000 (UTC) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, Christoph Hellwig , Damien Le Moal , Jaegeuk Kim , Bart Van Assche Subject: [PATCH v16 09/26] mq-deadline: Remove a local variable Date: Mon, 18 Nov 2024 16:27:58 -0800 Message-ID: <20241119002815.600608-10-bvanassche@acm.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241119002815.600608-1-bvanassche@acm.org> References: <20241119002815.600608-1-bvanassche@acm.org> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Since commit fde02699c242 ("block: mq-deadline: Remove support for zone write locking"), the local variable 'insert_before' is assigned once and is used once. Hence remove this local variable. Signed-off-by: Bart Van Assche Reviewed-by: Damien Le Moal --- block/mq-deadline.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/block/mq-deadline.c b/block/mq-deadline.c index acdc28756d9d..2edf84b1bc2a 100644 --- a/block/mq-deadline.c +++ b/block/mq-deadline.c @@ -699,8 +699,6 @@ static void dd_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, list_add(&rq->queuelist, &per_prio->dispatch); rq->fifo_time = jiffies; } else { - struct list_head *insert_before; - deadline_add_rq_rb(per_prio, rq); if (rq_mergeable(rq)) { @@ -713,8 +711,7 @@ static void dd_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, * set expire time and add to fifo list */ rq->fifo_time = jiffies + dd->fifo_expire[data_dir]; - insert_before = &per_prio->fifo_list[data_dir]; - list_add_tail(&rq->queuelist, insert_before); + list_add_tail(&rq->queuelist, &per_prio->fifo_list[data_dir]); } } From patchwork Tue Nov 19 00:27:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 13879268 Received: from 009.lax.mailroute.net (009.lax.mailroute.net [199.89.1.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A69D518638; Tue, 19 Nov 2024 00:28:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=199.89.1.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976132; cv=none; b=D4VYiqEOdJidSms8tEJWo+b9jex2hyud4gOmp7TgoUSAKbyXC2qoe4P/67sJtmOO/yoF5h9SmpfgYjl7SGJl2ypy3gipzkWrYgAXVp4d5BmRHcZMkAPU4nlyrACtoieRTafnA5WbhZ2UMOaRFb8yL0JhdEAhhInRarFVvD6hnjE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976132; c=relaxed/simple; bh=mJ4y40DOicBQ5BujFGiQk4Pp8sj5xwksoDUXKuUiBCo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=p4thoRo0uoXO3nfzPCg1xSEFE4qrXW1kFU8xKGGMRAApWlRpwE/sVQTvHpjmbih8NjWiqD/AKYXV7LpDGLv9xCEEBUKFdWktVmYaWij7Yaq5fesM6lRIunmXdn7H7clEoG4qOsTKjqdXUncASlb704323DOqAV1GFRuyel/XT/k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org; spf=pass smtp.mailfrom=acm.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b=RnxWcMof; arc=none smtp.client-ip=199.89.1.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=acm.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b="RnxWcMof" Received: from localhost (localhost [127.0.0.1]) by 009.lax.mailroute.net (Postfix) with ESMTP id 4XsljF2pf3zlgTWQ; Tue, 19 Nov 2024 00:28:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=acm.org; h= content-transfer-encoding:mime-version:references:in-reply-to :x-mailer:message-id:date:date:subject:subject:from:from :received:received; s=mr01; t=1731976126; x=1734568127; bh=T6k78 f57hdyUFAJFLQGLZF6bJ5r0i2KCdQTyofDwiRI=; b=RnxWcMof5R5aHXMdrMgDy Z/lee+6Ge3ZpSoL8+flrLUir8JxpVHFxwIUtBjyMXXbcD6fZvSHMXD3UXTK/Ga/O PVhQpiCTuWXs28AJiA+4cXnFizqCXtnnBJ3b2P+cPGQOL66ryqjHd5vV+QPpewAN KprnYfNFdJPNy5VSnDUKfyxAMfA25Edr6xDPJ/8BxmkQaFTwlLftH/q7bqd3oMFY 0QmzuBslXcntjb8IvlFt97zpMNdqdphwax8gk3tE/AVauKxWUQChL4gzQFLoffHv luVxDpSfhpF1oWu1mQxgp+FaxEfwUKQR1znvf7fQnxPmoMzEtP4cILHDut6OXMLY g== X-Virus-Scanned: by MailRoute Received: from 009.lax.mailroute.net ([127.0.0.1]) by localhost (009.lax [127.0.0.1]) (mroute_mailscanner, port 10029) with LMTP id p-pf_4pXcS1X; Tue, 19 Nov 2024 00:28:46 +0000 (UTC) Received: from bvanassche.mtv.corp.google.com (unknown [104.135.204.82]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: bvanassche@acm.org) by 009.lax.mailroute.net (Postfix) with ESMTPSA id 4Xslj902nnzlgT1M; Tue, 19 Nov 2024 00:28:44 +0000 (UTC) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, Christoph Hellwig , Damien Le Moal , Jaegeuk Kim , Bart Van Assche Subject: [PATCH v16 10/26] blk-mq: Clean up blk_mq_requeue_work() Date: Mon, 18 Nov 2024 16:27:59 -0800 Message-ID: <20241119002815.600608-11-bvanassche@acm.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241119002815.600608-1-bvanassche@acm.org> References: <20241119002815.600608-1-bvanassche@acm.org> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Move a statement that occurs in both branches of an if-statement in front of the if-statement. Fix a typo in a source code comment. No functionality has been changed. Signed-off-by: Bart Van Assche Reviewed-by: Damien Le Moal --- block/blk-mq.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index a45077e187b5..ece567b1b6be 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1553,19 +1553,17 @@ static void blk_mq_requeue_work(struct work_struct *work) while (!list_empty(&rq_list)) { rq = list_entry(rq_list.next, struct request, queuelist); + list_del_init(&rq->queuelist); /* - * If RQF_DONTPREP ist set, the request has been started by the + * If RQF_DONTPREP is set, the request has been started by the * driver already and might have driver-specific data allocated * already. Insert it into the hctx dispatch list to avoid * block layer merges for the request. */ - if (rq->rq_flags & RQF_DONTPREP) { - list_del_init(&rq->queuelist); + if (rq->rq_flags & RQF_DONTPREP) blk_mq_request_bypass_insert(rq, 0); - } else { - list_del_init(&rq->queuelist); + else blk_mq_insert_request(rq, BLK_MQ_INSERT_AT_HEAD); - } } while (!list_empty(&flush_list)) { From patchwork Tue Nov 19 00:28:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 13879269 Received: from 009.lax.mailroute.net (009.lax.mailroute.net [199.89.1.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ADBBE18E25; Tue, 19 Nov 2024 00:28:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=199.89.1.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976133; cv=none; b=ZsYGQbropjQF9OJ2jjbsX8n9406v2gzG7OXbdv7sFF8I3945MFZq3E98DNFaFYnEyYA3ascY8aiCnMeCr468gYbaVPbJuY3tTqpD0xqmYk2UOBfl276yHvgM/ZV4d9ETI+q4dN7qWHoG2rbgI7dUoeBWQBQxMzjGhuJ9fCf1pQE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976133; c=relaxed/simple; bh=QFQoLX2vfi6NRsME0p2vQj6OMR+B0rwhneMp/2n/bMQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZoKk2E+VSU2hTQ0JZAfw3yc5U31Fg8fPpZK6kMximlUdI44GVrbDhyx6N4ISZ9uoVZrzGebystMsWPt0qM/Zf+daUCaxuqV/gMbq2yf7+dao65KAmfotbuhfmtsCflPzxpp9QMF4wLyrBr8/edskfxtqg0Typs3ABvM7AtDC80Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org; spf=pass smtp.mailfrom=acm.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b=MDJfSRNG; arc=none smtp.client-ip=199.89.1.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=acm.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b="MDJfSRNG" Received: from localhost (localhost [127.0.0.1]) by 009.lax.mailroute.net (Postfix) with ESMTP id 4XsljH1kJSzlgMVT; Tue, 19 Nov 2024 00:28:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=acm.org; h= content-transfer-encoding:mime-version:references:in-reply-to :x-mailer:message-id:date:date:subject:subject:from:from :received:received; s=mr01; t=1731976128; x=1734568129; bh=/hnkz coA8QROymJcNm6a8ampF92BXmIteYFFiUE/6Qw=; b=MDJfSRNGgZprxvgm8HYgn G2Njs8Hiq9gplkDPxPrwJnOcPDsGgWI6cCXG9cxIwgQQXed8DPG+EBPfS/A6OwvN T/qJtD85LDI5xKsp4J8Hxcn02bqXbpSFFpLNlV7YD1zdNaQNjiEZ0Goa0sd2l+yt i3uQH9MWjuG83EyDdQT+A6dGC0c8ZhEOSmrmcISVHd1fTO4CizjDnUOEe+7ymzsM lozw4LCmSMJ+Ta/iAucInONi3SzfIyCINCdz7C3XoUvRFftF+/Y51L8RXnGMFsp7 Z2ITL7uP5xFWf48kCLsFDV7KqI01i5yJ01i7lyXNu7T5Wc/Kl8h/oGWnNXuzc6nl A== X-Virus-Scanned: by MailRoute Received: from 009.lax.mailroute.net ([127.0.0.1]) by localhost (009.lax [127.0.0.1]) (mroute_mailscanner, port 10029) with LMTP id Vngy7NXZOmoJ; Tue, 19 Nov 2024 00:28:48 +0000 (UTC) Received: from bvanassche.mtv.corp.google.com (unknown [104.135.204.82]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: bvanassche@acm.org) by 009.lax.mailroute.net (Postfix) with ESMTPSA id 4XsljB4MRrzlgT1K; Tue, 19 Nov 2024 00:28:46 +0000 (UTC) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, Christoph Hellwig , Damien Le Moal , Jaegeuk Kim , Bart Van Assche Subject: [PATCH v16 11/26] block: Optimize blk_mq_submit_bio() for the cache hit scenario Date: Mon, 18 Nov 2024 16:28:00 -0800 Message-ID: <20241119002815.600608-12-bvanassche@acm.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241119002815.600608-1-bvanassche@acm.org> References: <20241119002815.600608-1-bvanassche@acm.org> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Help the CPU branch predictor in case of a cache hit by handling the cache hit scenario first. Signed-off-by: Bart Van Assche Reviewed-by: Damien Le Moal --- block/blk-mq.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index ece567b1b6be..56a6b5bef39f 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -3123,12 +3123,12 @@ void blk_mq_submit_bio(struct bio *bio) goto queue_exit; new_request: - if (!rq) { + if (rq) { + blk_mq_use_cached_rq(rq, plug, bio); + } else { rq = blk_mq_get_new_requests(q, plug, bio, nr_segs); if (unlikely(!rq)) goto queue_exit; - } else { - blk_mq_use_cached_rq(rq, plug, bio); } trace_block_getrq(bio); From patchwork Tue Nov 19 00:28:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 13879270 Received: from 009.lax.mailroute.net (009.lax.mailroute.net [199.89.1.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EA1DC18E25; Tue, 19 Nov 2024 00:28:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=199.89.1.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976135; cv=none; b=LQWwxarQi409gUjiDdIqyFwsTAoNhATERG5EfAt2SAsVwGFvdo8Iik5rpIoue7i8Ujk4HFxYNFGJ7FtPa5pFqG3uIQq7jm3RtammUf6Q8zDjDiEW6mOq2Fih2PkpmqQpB5NhRlUqOxu0y+X/L3dAqaG6vjZ9sVSYMh9Uv6yi298= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976135; c=relaxed/simple; bh=JF9Y4w5Y21BZ0tYaraMKaxOwF1n2S/yIW/HnTfzVa3c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KDHNmms6HsE6SFX5ix+Am5MBcilAJIDoa2hWVE6POvBVM8/5LncEGto3pqZhGB+ZFGnv9qM5qbqoqyqpo+jPyd9uQ3L42Nh2yttIOq2W3xXyFc3C+Y8RXvmkbhJHGaReQaTxkcQxuZbE5D0tc0hgv6JPs16PjR10xxOpJ9bOFP0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org; spf=pass smtp.mailfrom=acm.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b=tYTbZE/l; arc=none smtp.client-ip=199.89.1.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=acm.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b="tYTbZE/l" Received: from localhost (localhost [127.0.0.1]) by 009.lax.mailroute.net (Postfix) with ESMTP id 4XsljK4zm3zlgMVW; Tue, 19 Nov 2024 00:28:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=acm.org; h= content-transfer-encoding:mime-version:references:in-reply-to :x-mailer:message-id:date:date:subject:subject:from:from :received:received; s=mr01; t=1731976129; x=1734568130; bh=dpl9r r8l6Vc6TiIGWK8Lw33yEmOF46WSghV+C95ot2c=; b=tYTbZE/lEIbcCsVI+D7+8 KmiZVZyGtMZxgsO5a74mROsHhk1vnUFNtvFw1lOlB9uXvxKsLIoByT1jt9u54kUm vz4n1USrrc4dxnG83OqqsKz3ZJv+kZyD/TOn2szuosbC4klNd2+K3IzJXWKLQpBo 2g8Su+ffKtnaohXZAInfX9O6E9moUAut1jnj1f/xdIl+Un8s1a0TY1TPkWbIx5fF CHk6dcbJMFrAi+pAKuwlooA8144o+wJ9Vckbnsa7qEQ5Hrfw+jWK8NZeUdRt/hr1 B4SUTIAgEYk5OuL6cuBmO9MPOOALywenXcZyyLZk4YR90Rh0cLny+9VYf33shS7N g== X-Virus-Scanned: by MailRoute Received: from 009.lax.mailroute.net ([127.0.0.1]) by localhost (009.lax [127.0.0.1]) (mroute_mailscanner, port 10029) with LMTP id ue1ddAi4XD_O; Tue, 19 Nov 2024 00:28:49 +0000 (UTC) Received: from bvanassche.mtv.corp.google.com (unknown [104.135.204.82]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: bvanassche@acm.org) by 009.lax.mailroute.net (Postfix) with ESMTPSA id 4XsljD1pT7zlgVXv; Tue, 19 Nov 2024 00:28:48 +0000 (UTC) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, Christoph Hellwig , Damien Le Moal , Jaegeuk Kim , Bart Van Assche Subject: [PATCH v16 12/26] block: Rework request allocation in blk_mq_submit_bio() Date: Mon, 18 Nov 2024 16:28:01 -0800 Message-ID: <20241119002815.600608-13-bvanassche@acm.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241119002815.600608-1-bvanassche@acm.org> References: <20241119002815.600608-1-bvanassche@acm.org> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Prepare for allocating a request from a specific hctx by making blk_mq_submit_bio() allocate a request later. The performance impact of this patch on the hot path is small: if a request is cached, one percpu_ref_get(&q->q_usage_counter) call and one percpu_ref_put(&q->q_usage_counter) call are added to the hot path. Signed-off-by: Bart Van Assche --- block/blk-mq.c | 36 +++++++++++++----------------------- 1 file changed, 13 insertions(+), 23 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 56a6b5bef39f..5ff124fbc17c 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -3071,11 +3071,6 @@ void blk_mq_submit_bio(struct bio *bio) struct request *rq; blk_status_t ret; - /* - * If the plug has a cached request for this queue, try to use it. - */ - rq = blk_mq_peek_cached_request(plug, q, bio->bi_opf); - /* * A BIO that was released from a zone write plug has already been * through the preparation in this function, already holds a reference @@ -3084,26 +3079,19 @@ void blk_mq_submit_bio(struct bio *bio) */ if (bio_zone_write_plugging(bio)) { nr_segs = bio->__bi_nr_segments; - if (rq) - blk_queue_exit(q); goto new_request; } bio = blk_queue_bounce(bio, q); /* - * The cached request already holds a q_usage_counter reference and we - * don't have to acquire a new one if we use it. + * Increment the queue usage counter before performing any checks + * based on queue properties that may change if the queue is frozen, + * e.g. the logical block size. */ - if (!rq) { - if (unlikely(bio_queue_enter(bio))) - return; - } + if (unlikely(bio_queue_enter(bio))) + return; - /* - * Device reconfiguration may change logical block size, so alignment - * check has to be done with queue usage counter held - */ if (unlikely(bio_unaligned(bio, q))) { bio_io_error(bio); goto queue_exit; @@ -3123,8 +3111,15 @@ void blk_mq_submit_bio(struct bio *bio) goto queue_exit; new_request: + rq = blk_mq_peek_cached_request(plug, q, bio->bi_opf); if (rq) { blk_mq_use_cached_rq(rq, plug, bio); + /* + * Here we hold two references: one because of the + * bio_queue_enter() call and a second one as the result of + * request allocation. Drop one. + */ + blk_queue_exit(q); } else { rq = blk_mq_get_new_requests(q, plug, bio, nr_segs); if (unlikely(!rq)) @@ -3167,12 +3162,7 @@ void blk_mq_submit_bio(struct bio *bio) return; queue_exit: - /* - * Don't drop the queue reference if we were trying to use a cached - * request and thus didn't acquire one. - */ - if (!rq) - blk_queue_exit(q); + blk_queue_exit(q); } #ifdef CONFIG_BLK_MQ_STACKING From patchwork Tue Nov 19 00:28:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 13879271 Received: from 009.lax.mailroute.net (009.lax.mailroute.net [199.89.1.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 147AC1CA81; Tue, 19 Nov 2024 00:28:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=199.89.1.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976137; cv=none; b=n9gRfcoa/kf+zXo83tQ5P1exnIHniC+5co7K/gXhWRfvIKUSmevKvnEW/uorq20UxSZ1dbDnzZc3wO3EE19biEZgvJ8j88cUHDcDVcDc7XZuHWzFB4JP2lyePC9FRmauq1v9umtHZzXuyLL+B817oCgi0FSAT56GMyCLOItZxD4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976137; c=relaxed/simple; bh=rKdB9eAJuqecXPJ1Ip+TFvZjsbRzDHsvQKGOfrBPXSA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rAL+WzIoq3arpWvsVUoX7BLEgGqtVeZGWBxUu7HZqqI9DASP3WudUjuaGD714u8yfySe0WPbJEw9cb6n6G6m9zmR7LquxYneAVQdkHNGSmj2J8R/i0hZQGXYxtP8fJieyMny64vL1UqbifZw62sje36OAFAtMpI1NlZH7IhNUCI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org; spf=pass smtp.mailfrom=acm.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b=nVWwhhCA; arc=none smtp.client-ip=199.89.1.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=acm.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b="nVWwhhCA" Received: from localhost (localhost [127.0.0.1]) by 009.lax.mailroute.net (Postfix) with ESMTP id 4XsljM5slyzlgMVX; Tue, 19 Nov 2024 00:28:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=acm.org; h= content-transfer-encoding:mime-version:references:in-reply-to :x-mailer:message-id:date:date:subject:subject:from:from :received:received; s=mr01; t=1731976131; x=1734568132; bh=mRFhE hPfN0g3PDtdjUeIsCcBOXYtmEpeE7xYsQ1hmxs=; b=nVWwhhCAwcKDxDG+iQNRU zgerqZrXl5obg1jAvS5PXpKE37mGFGYuM9xrauFZtSHSq8kAm4mouqUnkTu/lslh HPtWjjKQq6qBr6C/Hg/FOqm4jB+wri1xn7hASjj8YZIkT3uee/glrGpYenf1DXLe 4H2asKTfoa/YlfSXcIk8ZfUVtk3c5+VhHngDUQl63zTDwPvMAzO89bIyIPRiB4k+ uthqW93F0atMQsXI09Gdj+i6AJ4K1pq+iW2SFVpdHC7tfrMe9eFgyTihqHL60kr2 jSO1vor6Hl6wNMYM5R+n33vKmhylQYwgFrCLp1oINiVaSW5Vd9EF2gOaRgy8UElR A== X-Virus-Scanned: by MailRoute Received: from 009.lax.mailroute.net ([127.0.0.1]) by localhost (009.lax [127.0.0.1]) (mroute_mailscanner, port 10029) with LMTP id 8O4LOpz5ytuX; Tue, 19 Nov 2024 00:28:51 +0000 (UTC) Received: from bvanassche.mtv.corp.google.com (unknown [104.135.204.82]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: bvanassche@acm.org) by 009.lax.mailroute.net (Postfix) with ESMTPSA id 4XsljF68b2zlgT1M; Tue, 19 Nov 2024 00:28:49 +0000 (UTC) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, Christoph Hellwig , Damien Le Moal , Jaegeuk Kim , Bart Van Assche Subject: [PATCH v16 13/26] block: Support allocating from a specific software queue Date: Mon, 18 Nov 2024 16:28:02 -0800 Message-ID: <20241119002815.600608-14-bvanassche@acm.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241119002815.600608-1-bvanassche@acm.org> References: <20241119002815.600608-1-bvanassche@acm.org> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 A later patch will preserve the order of pipelined zoned writes by submitting all zoned writes per zone to the same queue as previously submitted zoned writes. Hence support allocating a request from a specific queue. Signed-off-by: Bart Van Assche --- block/blk-mq.c | 18 ++++++++++++++---- block/blk-mq.h | 1 + 2 files changed, 15 insertions(+), 4 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 5ff124fbc17c..f134d5e1c4a1 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -493,6 +493,7 @@ __blk_mq_alloc_requests_batch(struct blk_mq_alloc_data *data) static struct request *__blk_mq_alloc_requests(struct blk_mq_alloc_data *data) { struct request_queue *q = data->q; + int swq_cpu = data->swq_cpu; u64 alloc_time_ns = 0; struct request *rq; unsigned int tag; @@ -505,7 +506,8 @@ static struct request *__blk_mq_alloc_requests(struct blk_mq_alloc_data *data) data->flags |= BLK_MQ_REQ_NOWAIT; retry: - data->ctx = blk_mq_get_ctx(q); + data->swq_cpu = swq_cpu >= 0 ? swq_cpu : raw_smp_processor_id(); + data->ctx = __blk_mq_get_ctx(q, data->swq_cpu); data->hctx = blk_mq_map_queue(q, data->cmd_flags, data->ctx); if (q->elevator) { @@ -585,6 +587,7 @@ static struct request *blk_mq_rq_cache_fill(struct request_queue *q, .cmd_flags = opf, .nr_tags = plug->nr_ios, .cached_rqs = &plug->cached_rqs, + .swq_cpu = -1, }; struct request *rq; @@ -646,6 +649,7 @@ struct request *blk_mq_alloc_request(struct request_queue *q, blk_opf_t opf, .flags = flags, .cmd_flags = opf, .nr_tags = 1, + .swq_cpu = -1, }; int ret; @@ -2969,12 +2973,14 @@ static bool blk_mq_attempt_bio_merge(struct request_queue *q, } static struct request *blk_mq_get_new_requests(struct request_queue *q, + int swq_cpu, struct blk_plug *plug, struct bio *bio, unsigned int nsegs) { struct blk_mq_alloc_data data = { .q = q, + .swq_cpu = swq_cpu, .nr_tags = 1, .cmd_flags = bio->bi_opf, }; @@ -3001,7 +3007,8 @@ static struct request *blk_mq_get_new_requests(struct request_queue *q, * Check if there is a suitable cached request and return it. */ static struct request *blk_mq_peek_cached_request(struct blk_plug *plug, - struct request_queue *q, blk_opf_t opf) + struct request_queue *q, + int swq_cpu, blk_opf_t opf) { enum hctx_type type = blk_mq_get_hctx_type(opf); struct request *rq; @@ -3011,6 +3018,8 @@ static struct request *blk_mq_peek_cached_request(struct blk_plug *plug, rq = rq_list_peek(&plug->cached_rqs); if (!rq || rq->q != q) return NULL; + if (swq_cpu >= 0 && rq->mq_ctx->cpu != swq_cpu) + return NULL; if (type != rq->mq_hctx->type && (type != HCTX_TYPE_READ || rq->mq_hctx->type != HCTX_TYPE_DEFAULT)) return NULL; @@ -3069,6 +3078,7 @@ void blk_mq_submit_bio(struct bio *bio) struct blk_mq_hw_ctx *hctx; unsigned int nr_segs; struct request *rq; + int swq_cpu = -1; blk_status_t ret; /* @@ -3111,7 +3121,7 @@ void blk_mq_submit_bio(struct bio *bio) goto queue_exit; new_request: - rq = blk_mq_peek_cached_request(plug, q, bio->bi_opf); + rq = blk_mq_peek_cached_request(plug, q, swq_cpu, bio->bi_opf); if (rq) { blk_mq_use_cached_rq(rq, plug, bio); /* @@ -3121,7 +3131,7 @@ void blk_mq_submit_bio(struct bio *bio) */ blk_queue_exit(q); } else { - rq = blk_mq_get_new_requests(q, plug, bio, nr_segs); + rq = blk_mq_get_new_requests(q, swq_cpu, plug, bio, nr_segs); if (unlikely(!rq)) goto queue_exit; } diff --git a/block/blk-mq.h b/block/blk-mq.h index 89a20fffa4b1..309db553aba6 100644 --- a/block/blk-mq.h +++ b/block/blk-mq.h @@ -158,6 +158,7 @@ struct blk_mq_alloc_data { struct rq_list *cached_rqs; /* input & output parameter */ + int swq_cpu; struct blk_mq_ctx *ctx; struct blk_mq_hw_ctx *hctx; }; From patchwork Tue Nov 19 00:28:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 13879273 Received: from 009.lax.mailroute.net (009.lax.mailroute.net [199.89.1.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F1BC01798C; Tue, 19 Nov 2024 00:28:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=199.89.1.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976140; cv=none; b=ramtTgxlrg5isWFt9x7kyqGva5oT8r62VjKx12IysYE+vDFKiPwo7n/rp+VFoeb4nRmXywgHcHB2lbYvfy0mSGdip3manBjFWmOL0Gh+BBh8cy1Ka1afZzQyusGGfVuclUst5+EgF4IQckVSCehgcz+FgIq4pqlv5w9ZNt8Y3hc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976140; c=relaxed/simple; bh=uJbIK0PotaTTcRtYuAvkSM7875tnTVV1J8UYYFYHMtY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DxURig3hL1EbFclOo7Q9sA4v/uxdXDDa17Y069M6GqsfJ7lszkvnyDRm/PHMIxIElrTRWqw2AaH2+jxU/vYhT/NCIOHRv+GUBHrD7vTpCHNQhrGn7dj22bZxlYfGIS+6pj26RgQ8flD5lTKQhxxQvIqDPWT8/wKr6HvBAr1gVmA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org; spf=pass smtp.mailfrom=acm.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b=SsYqyPX6; arc=none smtp.client-ip=199.89.1.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=acm.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b="SsYqyPX6" Received: from localhost (localhost [127.0.0.1]) by 009.lax.mailroute.net (Postfix) with ESMTP id 4XsljQ3rzBzlgMVZ; Tue, 19 Nov 2024 00:28:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=acm.org; h= content-transfer-encoding:mime-version:references:in-reply-to :x-mailer:message-id:date:date:subject:subject:from:from :received:received; s=mr01; t=1731976132; x=1734568133; bh=+JzvP Rq8XzCgGi53f83c3kpTmMgFQVQtENNrxDRXm14=; b=SsYqyPX6j7a7ZM87atznG j0urHLk7aNC4Le/tH8NoZY+7WeePHB7OgJJ2q2Me0ANb69ShdPXhgv6Kfplc0vZW yv6btHAQKzJn+QIvuyoDT29lUULKkjq8gg53y7G3TaM0hZAHQjmbza8f3XDjqHCV f2ZilnoIC86u/R+Prqzi1h/teCa8DlgDeXePEMCYKk/dRY5MmbwsVNfLpvAgg5lw z+efcBgPog8xlJ02vsYZtiy2V2eRJXCy3M8ymSek2TUMUd+QakCCm8OM29/7rBUY TzjCUU3d3rYXogs8ORmNR4MJn9fCSeJm4FIXovwy3iQoO8YdXo3Xx5in80UuvgwP g== X-Virus-Scanned: by MailRoute Received: from 009.lax.mailroute.net ([127.0.0.1]) by localhost (009.lax [127.0.0.1]) (mroute_mailscanner, port 10029) with LMTP id c9EfC0iEQDgu; Tue, 19 Nov 2024 00:28:52 +0000 (UTC) Received: from bvanassche.mtv.corp.google.com (unknown [104.135.204.82]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: bvanassche@acm.org) by 009.lax.mailroute.net (Postfix) with ESMTPSA id 4XsljH3Q46zlgT1K; Tue, 19 Nov 2024 00:28:51 +0000 (UTC) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, Christoph Hellwig , Damien Le Moal , Jaegeuk Kim , Bart Van Assche Subject: [PATCH v16 14/26] blk-mq: Restore the zoned write order when requeuing Date: Mon, 18 Nov 2024 16:28:03 -0800 Message-ID: <20241119002815.600608-15-bvanassche@acm.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241119002815.600608-1-bvanassche@acm.org> References: <20241119002815.600608-1-bvanassche@acm.org> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Zoned writes may be requeued, e.g. if a block driver returns BLK_STS_RESOURCE. Requests may be requeued in another order than submitted. Restore the request order if requests are requeued. Signed-off-by: Bart Van Assche --- block/bfq-iosched.c | 2 ++ block/blk-mq.c | 20 +++++++++++++++++++- block/blk-mq.h | 2 ++ block/kyber-iosched.c | 2 ++ block/mq-deadline.c | 7 ++++++- include/linux/blk-mq.h | 2 +- 6 files changed, 32 insertions(+), 3 deletions(-) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 0747d9d0e48c..13bedbf03bd2 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -6265,6 +6265,8 @@ static void bfq_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, if (flags & BLK_MQ_INSERT_AT_HEAD) { list_add(&rq->queuelist, &bfqd->dispatch); + } else if (flags & BLK_MQ_INSERT_ORDERED) { + blk_mq_insert_ordered(rq, &bfqd->dispatch); } else if (!bfqq) { list_add_tail(&rq->queuelist, &bfqd->dispatch); } else { diff --git a/block/blk-mq.c b/block/blk-mq.c index f134d5e1c4a1..1302ccbf2a7d 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1564,7 +1564,9 @@ static void blk_mq_requeue_work(struct work_struct *work) * already. Insert it into the hctx dispatch list to avoid * block layer merges for the request. */ - if (rq->rq_flags & RQF_DONTPREP) + if (blk_rq_is_seq_zoned_write(rq)) + blk_mq_insert_request(rq, BLK_MQ_INSERT_ORDERED); + else if (rq->rq_flags & RQF_DONTPREP) blk_mq_request_bypass_insert(rq, 0); else blk_mq_insert_request(rq, BLK_MQ_INSERT_AT_HEAD); @@ -2599,6 +2601,20 @@ static void blk_mq_insert_requests(struct blk_mq_hw_ctx *hctx, blk_mq_run_hw_queue(hctx, run_queue_async); } +void blk_mq_insert_ordered(struct request *rq, struct list_head *list) +{ + struct request_queue *q = rq->q; + struct request *rq2; + + list_for_each_entry(rq2, list, queuelist) + if (rq2->q == q && blk_rq_pos(rq2) > blk_rq_pos(rq)) + break; + + /* Insert rq before rq2. If rq2 is the list head, append at the end. */ + list_add_tail(&rq->queuelist, &rq2->queuelist); +} +EXPORT_SYMBOL_GPL(blk_mq_insert_ordered); + static void blk_mq_insert_request(struct request *rq, blk_insert_t flags) { struct request_queue *q = rq->q; @@ -2653,6 +2669,8 @@ static void blk_mq_insert_request(struct request *rq, blk_insert_t flags) spin_lock(&ctx->lock); if (flags & BLK_MQ_INSERT_AT_HEAD) list_add(&rq->queuelist, &ctx->rq_lists[hctx->type]); + else if (flags & BLK_MQ_INSERT_ORDERED) + blk_mq_insert_ordered(rq, &ctx->rq_lists[hctx->type]); else list_add_tail(&rq->queuelist, &ctx->rq_lists[hctx->type]); diff --git a/block/blk-mq.h b/block/blk-mq.h index 309db553aba6..10b9fb3ca762 100644 --- a/block/blk-mq.h +++ b/block/blk-mq.h @@ -40,8 +40,10 @@ enum { typedef unsigned int __bitwise blk_insert_t; #define BLK_MQ_INSERT_AT_HEAD ((__force blk_insert_t)0x01) +#define BLK_MQ_INSERT_ORDERED ((__force blk_insert_t)0x02) void blk_mq_submit_bio(struct bio *bio); +void blk_mq_insert_ordered(struct request *rq, struct list_head *list); int blk_mq_poll(struct request_queue *q, blk_qc_t cookie, struct io_comp_batch *iob, unsigned int flags); void blk_mq_exit_queue(struct request_queue *q); diff --git a/block/kyber-iosched.c b/block/kyber-iosched.c index 4155594aefc6..77bb41bab68d 100644 --- a/block/kyber-iosched.c +++ b/block/kyber-iosched.c @@ -603,6 +603,8 @@ static void kyber_insert_requests(struct blk_mq_hw_ctx *hctx, trace_block_rq_insert(rq); if (flags & BLK_MQ_INSERT_AT_HEAD) list_move(&rq->queuelist, head); + else if (flags & BLK_MQ_INSERT_ORDERED) + blk_mq_insert_ordered(rq, head); else list_move_tail(&rq->queuelist, head); sbitmap_set_bit(&khd->kcq_map[sched_domain], diff --git a/block/mq-deadline.c b/block/mq-deadline.c index 2edf84b1bc2a..200e5a2928ce 100644 --- a/block/mq-deadline.c +++ b/block/mq-deadline.c @@ -711,7 +711,12 @@ static void dd_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, * set expire time and add to fifo list */ rq->fifo_time = jiffies + dd->fifo_expire[data_dir]; - list_add_tail(&rq->queuelist, &per_prio->fifo_list[data_dir]); + if (flags & BLK_MQ_INSERT_ORDERED) + blk_mq_insert_ordered(rq, + &per_prio->fifo_list[data_dir]); + else + list_add_tail(&rq->queuelist, + &per_prio->fifo_list[data_dir]); } } diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index ac05974f08f9..f7514eefccfd 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -85,7 +85,7 @@ enum { /* flags that prevent us from merging requests: */ #define RQF_NOMERGE_FLAGS \ - (RQF_STARTED | RQF_FLUSH_SEQ | RQF_SPECIAL_PAYLOAD) + (RQF_STARTED | RQF_FLUSH_SEQ | RQF_DONTPREP | RQF_SPECIAL_PAYLOAD) enum mq_rq_state { MQ_RQ_IDLE = 0, From patchwork Tue Nov 19 00:28:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 13879272 Received: from 009.lax.mailroute.net (009.lax.mailroute.net [199.89.1.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A6A797711F; Tue, 19 Nov 2024 00:28:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=199.89.1.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976139; cv=none; b=s72rQSsDQqtIV7iavE+cDmLSozCbw41NDMpANvnkFD/l5ihwmjAqMAbwGE85GI70yrV1Aa22k2wCr3zVo1DZCl3/lNHp+JzpYGBUNxejK5QXLnr9+9+nP7r/WTFXq7Uop5H6sOq1hU72yeBU7uafljtx5yLS3vHN2NIjYcvp+Do= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976139; c=relaxed/simple; bh=ZL+N+xB9nnJlFQeOnW84AorIn6wz09Q7mmzClFv5/Xk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dfiag1NyCwPhJGUv8WA1EXQEyqO5EZDY0g35TILe80pG4oa6e7j2MMOmN587oIKt71XOP07GJpentoSvjl9rwzfznR0w1cRTjXj1PVuRyDxl3ylkizWjpMREhvqM1UpLCf6icJZIBL2yYs1bpS892NEe+OmdgmtttnvRVAhk5T8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org; spf=pass smtp.mailfrom=acm.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b=nRPmXmIg; arc=none smtp.client-ip=199.89.1.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=acm.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b="nRPmXmIg" Received: from localhost (localhost [127.0.0.1]) by 009.lax.mailroute.net (Postfix) with ESMTP id 4XsljP2zvTzlgMVY; Tue, 19 Nov 2024 00:28:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=acm.org; h= content-transfer-encoding:mime-version:references:in-reply-to :x-mailer:message-id:date:date:subject:subject:from:from :received:received; s=mr01; t=1731976134; x=1734568135; bh=Nk0D2 kBxuDtV2SY+RnXHM5qpiqmKRg2f69nFTovfeK4=; b=nRPmXmIgcdDQlyDSMXmMO /d++gFKUFFzLKDE0nfs8ycXhrS3f/mhwumTitlyEg369dM8CuNctWXEFugIDJXLx 16dVzubsoHnUsbVYoihjhSb2rGxU5/3+5kIkH8nq81J1bJo2xVlnqHNmVmhqhOJG ffEts8IxZiWSjP64ybIpYlUZ1RhlA3BEQNHlUWEhoOzFV/D0JpiGxJBNd9QlMRXJ muTjVIp9+InHaOBnff4LVJSoQqBFdNPxTSCIxdMrWu0kKKqer4v18hfHHIJarivl kpUlAJ8uo0+CWIi1GpmWQHGSi41BofvlPv/hz9agupFqpjgMlOmcPR1k4KnZoytt A== X-Virus-Scanned: by MailRoute Received: from 009.lax.mailroute.net ([127.0.0.1]) by localhost (009.lax [127.0.0.1]) (mroute_mailscanner, port 10029) with LMTP id 5yC0ktTUZ6Cx; Tue, 19 Nov 2024 00:28:54 +0000 (UTC) Received: from bvanassche.mtv.corp.google.com (unknown [104.135.204.82]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: bvanassche@acm.org) by 009.lax.mailroute.net (Postfix) with ESMTPSA id 4XsljK0dT0zlgMVV; Tue, 19 Nov 2024 00:28:52 +0000 (UTC) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, Christoph Hellwig , Damien Le Moal , Jaegeuk Kim , Bart Van Assche Subject: [PATCH v16 15/26] blk-zoned: Document the locking order Date: Mon, 18 Nov 2024 16:28:04 -0800 Message-ID: <20241119002815.600608-16-bvanassche@acm.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241119002815.600608-1-bvanassche@acm.org> References: <20241119002815.600608-1-bvanassche@acm.org> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Signed-off-by: Bart Van Assche Reviewed-by: Damien Le Moal --- block/blk-zoned.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/block/blk-zoned.c b/block/blk-zoned.c index ded38fa9ae3d..1d0f8fea7516 100644 --- a/block/blk-zoned.c +++ b/block/blk-zoned.c @@ -51,7 +51,8 @@ static const char *const zone_cond_name[] = { * reference is dropped whenever the zone of the zone write plug is reset, * finished and when the zone becomes full (last write BIO to the zone * completes). - * @lock: Spinlock to atomically manipulate the plug. + * @lock: Spinlock to atomically manipulate the plug. Outer lock relative to + * disk->zone_wplugs_lock. * @flags: Flags indicating the plug state. * @zone_no: The number of the zone the plug is managing. * @wp_offset: The zone write pointer location relative to the start of the zone From patchwork Tue Nov 19 00:28:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 13879274 Received: from 009.lax.mailroute.net (009.lax.mailroute.net [199.89.1.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A908282D91; Tue, 19 Nov 2024 00:28:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=199.89.1.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976141; cv=none; b=jk6prQhJ3SCq9A5oeYDB5WEHphARdAh4EUyWDNT9Abki6eAc1baLX8RutlTFZynBsYqZJd7Yi6forBECEuDA+Ry6yX65fQiVnIigV/cQ/maer7tCHUjMR/MrC6mc1L8Qn+AOxukcop1AnBYPxGrSjNRcvOE01pmPKBYYnHk23M8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976141; c=relaxed/simple; bh=fwOdfQ/x2HOgWKEO78/wmXLgfazU3us33yLktN29+6M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=crafIbU9cXhi1kNr3SbEvjhcJYbMMF0NPIjxP5TmZbKLoN8sVP3qkBMlGsD1KnrdzWs5+MhpSnUKHI/Sm8aVCkoPZJHO6gN3brRVcUAl+Ao+p27N4HhQNa3la6rTfjZmxCri1X/j3g9o5TDc3SuT/dIKKxmCVlROWjBr6knOPN0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org; spf=pass smtp.mailfrom=acm.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b=IsOSiKyk; arc=none smtp.client-ip=199.89.1.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=acm.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b="IsOSiKyk" Received: from localhost (localhost [127.0.0.1]) by 009.lax.mailroute.net (Postfix) with ESMTP id 4XsljR1GtqzlgT1K; Tue, 19 Nov 2024 00:28:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=acm.org; h= content-transfer-encoding:mime-version:references:in-reply-to :x-mailer:message-id:date:date:subject:subject:from:from :received:received; s=mr01; t=1731976136; x=1734568137; bh=71v0R Ga9kFJCqJAJTpFKNLCnmQ7dJ7JQaKkZwVvcq1w=; b=IsOSiKykek1MrG8dAjZPv m6uRO4HlSJPIFCmYZF89MWxpMyIow+KLruz+PYsMhdkMU84Ovteo327CbhimroXl 7tND8WMG8macgblyYLuV5cYOOD7okrAAi3C+A3wi94N01IEh6AqeCH24ACLdvMjW kt5TSKG4VRlpBv4c6zXfwnVUzSySc6n3QVGwcGxcAglfkSm6KkMxN072IaqUm4ZO O5ko/fIzbg1WDuPvR50W/Z/dRplc48Y6jFdnDY0boYKk6904EC43seQf+VsFpebY z1aoUvvoCbj7Q5tOxprhD4uht/mVLF3l7aQUhntTfmaAlIDrbFVIuS5fKrU0nLDg w== X-Virus-Scanned: by MailRoute Received: from 009.lax.mailroute.net ([127.0.0.1]) by localhost (009.lax [127.0.0.1]) (mroute_mailscanner, port 10029) with LMTP id A6ExA0re-6iL; Tue, 19 Nov 2024 00:28:56 +0000 (UTC) Received: from bvanassche.mtv.corp.google.com (unknown [104.135.204.82]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: bvanassche@acm.org) by 009.lax.mailroute.net (Postfix) with ESMTPSA id 4XsljL5DTFzlgVXv; Tue, 19 Nov 2024 00:28:54 +0000 (UTC) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, Christoph Hellwig , Damien Le Moal , Jaegeuk Kim , Bart Van Assche Subject: [PATCH v16 16/26] blk-zoned: Document locking assumptions Date: Mon, 18 Nov 2024 16:28:05 -0800 Message-ID: <20241119002815.600608-17-bvanassche@acm.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241119002815.600608-1-bvanassche@acm.org> References: <20241119002815.600608-1-bvanassche@acm.org> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Signed-off-by: Bart Van Assche Reviewed-by: Damien Le Moal --- block/blk-zoned.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/block/blk-zoned.c b/block/blk-zoned.c index 1d0f8fea7516..17fe40db1888 100644 --- a/block/blk-zoned.c +++ b/block/blk-zoned.c @@ -441,6 +441,8 @@ static inline void disk_put_zone_wplug(struct blk_zone_wplug *zwplug) static inline bool disk_should_remove_zone_wplug(struct gendisk *disk, struct blk_zone_wplug *zwplug) { + lockdep_assert_held(&zwplug->lock); + /* If the zone write plug was already removed, we are done. */ if (zwplug->flags & BLK_ZONE_WPLUG_UNHASHED) return false; @@ -583,6 +585,8 @@ static inline void disk_zone_wplug_set_error(struct gendisk *disk, { unsigned long flags; + lockdep_assert_held(&zwplug->lock); + if (zwplug->flags & BLK_ZONE_WPLUG_ERROR) return; @@ -933,6 +937,8 @@ static bool blk_zone_wplug_prepare_bio(struct blk_zone_wplug *zwplug, { struct gendisk *disk = bio->bi_bdev->bd_disk; + lockdep_assert_held(&zwplug->lock); + /* * Check that the user is not attempting to write to a full zone. * We know such BIO will fail, and that would potentially overflow our From patchwork Tue Nov 19 00:28:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 13879275 Received: from 009.lax.mailroute.net (009.lax.mailroute.net [199.89.1.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 74EA485956; Tue, 19 Nov 2024 00:29:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=199.89.1.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976142; cv=none; b=DT0fQ5zvN8xHnybkSTt13aL0AMeuq+56NDdWX46Dx3s6kC7aW7qHSCHOck16R2XQwcgJdW3gDMx16JTEN6Y2JKPpcquSA9EZx2m29gG/MGQQeUFy22HmefJv+yzSJjMFaviokl5M2Dev8iFqxz5ZkeQBthcChmZ8r254qFPbeUc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976142; c=relaxed/simple; bh=+xK2oaLrr8HO08NFs9MHzCcsEr4cVuqvKd/tWQ0+RvA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qHrE1acuEhFvFaQfUSPd4cJk6hviaabl2KXny7v91BLidUpcFhYpcnee8WqcyaViPv5dTBjXp51XdxeeHwhMi2BWjW7xcdLi3mrTRrekMhrokxn5FL3v0Sg/K1H9+cmVGbOP2jMIUbyyijJPoHkNUIpikiufwqpH1Lqs7kAZWK8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org; spf=pass smtp.mailfrom=acm.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b=n15Isiem; arc=none smtp.client-ip=199.89.1.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=acm.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b="n15Isiem" Received: from localhost (localhost [127.0.0.1]) by 009.lax.mailroute.net (Postfix) with ESMTP id 4XsljT1flxzlgMVc; Tue, 19 Nov 2024 00:29:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=acm.org; h= content-transfer-encoding:mime-version:references:in-reply-to :x-mailer:message-id:date:date:subject:subject:from:from :received:received; s=mr01; t=1731976137; x=1734568138; bh=SCPA9 PUI3otf9aArN2z/q/hRgSOZLD5iRBbmbUow0HQ=; b=n15Isiemzh8xZkjuAiK+z 4MvM9bSX+fhuvn2Ek81AgAELdJowsJZq1jgBbc6CCpDsx8QX867bFHjhOGeQayoV PDajjeEjIZvBaaabrcMHl3VwohRNA2TcKXkY9BP7u3SYafcZ4pWil2XgM67+O1Au PV/alWxwG7WzpgQOghu6osN9x+5NL1W1PpZ1B//0nZ2J3JNN5RhvsUmlCdvRjw5m qGpQ+owF6Bxh1AoldGZVUfnICn+sgcihGLasUMWFjya9CBom/hUvvUKqQd14fozv d4tVII+viHzKAyHJPeSdKgbJwmOFCCuKXtgRLqbvhiNpCELcPvoyGdrckaZvzwQF A== X-Virus-Scanned: by MailRoute Received: from 009.lax.mailroute.net ([127.0.0.1]) by localhost (009.lax [127.0.0.1]) (mroute_mailscanner, port 10029) with LMTP id esQLrL8NlFr4; Tue, 19 Nov 2024 00:28:57 +0000 (UTC) Received: from bvanassche.mtv.corp.google.com (unknown [104.135.204.82]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: bvanassche@acm.org) by 009.lax.mailroute.net (Postfix) with ESMTPSA id 4XsljN2THGzlgT1M; Tue, 19 Nov 2024 00:28:56 +0000 (UTC) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, Christoph Hellwig , Damien Le Moal , Jaegeuk Kim , Bart Van Assche Subject: [PATCH v16 17/26] blk-zoned: Uninline functions that are not in the hot path Date: Mon, 18 Nov 2024 16:28:06 -0800 Message-ID: <20241119002815.600608-18-bvanassche@acm.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241119002815.600608-1-bvanassche@acm.org> References: <20241119002815.600608-1-bvanassche@acm.org> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Apply the convention that is followed elsewhere in the block layer: only declare functions inline if these are in the hot path. Signed-off-by: Bart Van Assche Reviewed-by: Damien Le Moal --- block/blk-zoned.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/block/blk-zoned.c b/block/blk-zoned.c index 17fe40db1888..2b4783026450 100644 --- a/block/blk-zoned.c +++ b/block/blk-zoned.c @@ -558,8 +558,8 @@ static struct blk_zone_wplug *disk_get_and_lock_zone_wplug(struct gendisk *disk, return zwplug; } -static inline void blk_zone_wplug_bio_io_error(struct blk_zone_wplug *zwplug, - struct bio *bio) +static void blk_zone_wplug_bio_io_error(struct blk_zone_wplug *zwplug, + struct bio *bio) { struct request_queue *q = zwplug->disk->queue; @@ -580,8 +580,8 @@ static void disk_zone_wplug_abort(struct blk_zone_wplug *zwplug) blk_zone_wplug_bio_io_error(zwplug, bio); } -static inline void disk_zone_wplug_set_error(struct gendisk *disk, - struct blk_zone_wplug *zwplug) +static void disk_zone_wplug_set_error(struct gendisk *disk, + struct blk_zone_wplug *zwplug) { unsigned long flags; @@ -607,8 +607,8 @@ static inline void disk_zone_wplug_set_error(struct gendisk *disk, spin_unlock_irqrestore(&disk->zone_wplugs_lock, flags); } -static inline void disk_zone_wplug_clear_error(struct gendisk *disk, - struct blk_zone_wplug *zwplug) +static void disk_zone_wplug_clear_error(struct gendisk *disk, + struct blk_zone_wplug *zwplug) { unsigned long flags; @@ -1597,7 +1597,7 @@ void disk_free_zone_resources(struct gendisk *disk) disk->nr_zones = 0; } -static inline bool disk_need_zone_resources(struct gendisk *disk) +static bool disk_need_zone_resources(struct gendisk *disk) { /* * All mq zoned devices need zone resources so that the block layer From patchwork Tue Nov 19 00:28:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 13879276 Received: from 009.lax.mailroute.net (009.lax.mailroute.net [199.89.1.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7CA0525632; Tue, 19 Nov 2024 00:29:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=199.89.1.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976143; cv=none; b=SfuVhhmhI2Zuq2aD4fGkviEilaX+P+qGYVW0aj4PQJJBmN7lR/Oj/TQ5PMkxmx1GjL5pWiiUQY5rDDZsoGaZxuAe1kaiIgW16RWH85sfSoXMacIebUmOC9xrrlSyPIaUcwM7JDNOq7wi0RMkm7/iVVL0gvbbQN3oxZWE05fYKTk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976143; c=relaxed/simple; bh=CE632OLHkRVhdjSCwwDmV09yw1UiRoQewtJ+mADvMkc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=f/0ohiNPj+a1149QNYLuWCFLKGGCgQDTeiHCreukp+YkaB12yeJk2ksEkwuXKbwbZBa9mqFsx8L+WttYsMVjBt4QXW0QRqqsO8dVf8pnCR+fzqBfYyrXRWd9YBlOZWwf080xyKKv46VGQdq+cZF0x+GboEqKxxxXwcs1t2yB5Y8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org; spf=pass smtp.mailfrom=acm.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b=jQ10niYo; arc=none smtp.client-ip=199.89.1.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=acm.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b="jQ10niYo" Received: from localhost (localhost [127.0.0.1]) by 009.lax.mailroute.net (Postfix) with ESMTP id 4XsljV1TD6zlgT1M; Tue, 19 Nov 2024 00:29:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=acm.org; h= content-transfer-encoding:mime-version:references:in-reply-to :x-mailer:message-id:date:date:subject:subject:from:from :received:received; s=mr01; t=1731976139; x=1734568140; bh=OHDmu ir9C2Gpk82PnlQKADcYSuAXlDxCFRO42HlpWPo=; b=jQ10niYoH/12bdZMfXfeU wtkSe40/c4r9bB4vw+GO5KM3gpaDGrhbU2t8FKhcf2IC/Q4zGJYa8erlih7Vhhel iq3Xo6lanYVsDJdnOPFTs+wZo3r0i18L3sxc9tbc/3ZFtHJOymuamtNS5y2vSjxs vnxReqYO2UC/iqln5cgDKsfZBdBpN7ct0v8KasHQ9zyLHuggOXlfA0rsaDexrcae mwcfjmFq+LoKPE6FFhFVC88swFVz2tc29P1Gpeps2Gxs31kZ82ryIl24ndozHj5B MZjRDKDf/WMugdfqGkCR1AjMXMBSg+Lk+sjHfFWdMf44N4vUjQHPqoiF1ASWPqks g== X-Virus-Scanned: by MailRoute Received: from 009.lax.mailroute.net ([127.0.0.1]) by localhost (009.lax [127.0.0.1]) (mroute_mailscanner, port 10029) with LMTP id V83q5IkwGgpO; Tue, 19 Nov 2024 00:28:59 +0000 (UTC) Received: from bvanassche.mtv.corp.google.com (unknown [104.135.204.82]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: bvanassche@acm.org) by 009.lax.mailroute.net (Postfix) with ESMTPSA id 4XsljP6ks0zlgMVV; Tue, 19 Nov 2024 00:28:57 +0000 (UTC) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, Christoph Hellwig , Damien Le Moal , Jaegeuk Kim , Bart Van Assche Subject: [PATCH v16 18/26] blk-zoned: Make disk_should_remove_zone_wplug() more robust Date: Mon, 18 Nov 2024 16:28:07 -0800 Message-ID: <20241119002815.600608-19-bvanassche@acm.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241119002815.600608-1-bvanassche@acm.org> References: <20241119002815.600608-1-bvanassche@acm.org> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Make the disk_should_remove_zone_wplug() behavior independent of the number of zwplug references held by the caller. Signed-off-by: Bart Van Assche --- block/blk-zoned.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/block/blk-zoned.c b/block/blk-zoned.c index 2b4783026450..59f6559b94da 100644 --- a/block/blk-zoned.c +++ b/block/blk-zoned.c @@ -457,12 +457,9 @@ static inline bool disk_should_remove_zone_wplug(struct gendisk *disk, * blk_zone_write_plug_finish_request() (e.g. with split BIOs * that are chained). In such case, disk_zone_wplug_unplug_bio() * should not attempt to remove the zone write plug until all BIO - * completions are seen. Check by looking at the zone write plug - * reference count, which is 2 when the plug is unused (one reference - * taken when the plug was allocated and another reference taken by the - * caller context). + * completions are seen. */ - if (refcount_read(&zwplug->ref) > 2) + if (zwplug->wp_offset != zwplug->wp_offset_compl) return false; /* We can remove zone write plugs for zones that are empty or full. */ From patchwork Tue Nov 19 00:28:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 13879277 Received: from 009.lax.mailroute.net (009.lax.mailroute.net [199.89.1.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7DB8CDDD9; Tue, 19 Nov 2024 00:29:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=199.89.1.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976146; cv=none; b=UI/vo0YFrwjy+UYY3Y0YTpekZGftpo1h7nzxQIs1vm2+5tWBUf7EFuea32s4a060+07vu9AGlNot49BKs3mTbHz/Ebhrbn6kaEUCkM+cHfMO+8rqhhc5fh68RhspnJm3XdwVzhJEh4b4jFAmYjYki9zsanzNkP88xTfZDjsnYoQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976146; c=relaxed/simple; bh=nH7YAVHzBh202POY8G4tJIN0Gy33p/of5ZpcPsHrbZ0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gnbHZyeaTETKmBOwLsu+Cmqj9JABtP/OQ8Cgtnq9B8tIrWBOtWCCK2i4pZJaTq13/1oBt5RcQOyysJNzEtQY1iN6cLjcy8Dfjsg9+utsszoIrpgF0LVcSDezVXhge/9csDi9nkaGxl1Iuk1Q/94eThCyf9ehdYijIRoDQeKqWUE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org; spf=pass smtp.mailfrom=acm.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b=LKrJoeBD; arc=none smtp.client-ip=199.89.1.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=acm.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b="LKrJoeBD" Received: from localhost (localhost [127.0.0.1]) by 009.lax.mailroute.net (Postfix) with ESMTP id 4XsljY1vpFzlgMVg; Tue, 19 Nov 2024 00:29:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=acm.org; h= content-transfer-encoding:mime-version:references:in-reply-to :x-mailer:message-id:date:date:subject:subject:from:from :received:received; s=mr01; t=1731976140; x=1734568141; bh=icrDL w2/tDGWLUCZD68srtLa4ZCNc3u6D6f3ezs2BU8=; b=LKrJoeBDbcKpF2LVaseQR euGezVCZGdu7tBSqPDfntcgTBOgZFd+9lMZm+TQ/zoR3XftAiXppGJdjtUQHu023 TfXnVxAfSWUcqx3e1lwQWrahGQKN7woLpBK2e/AxFMxJnwRoH3z8Di29fim7jwu7 RagnfaN+9mgnuzu67FeZndkztWbKVXsG7b3shJK/ESRSQP9mbtGBI7jHzzLV0ieR 8vuP4nM24AOdONf4qq7ptB5fQj4yE/Pty1MBhkkLELLKb1mbYXTRsK6Y76BhFAtu tr9gxoRIsKpeh+725BS6SX4u+xLl5uAhtLBeFZ3k/smh83qTpvbL/Q+C5bgTAEaL A== X-Virus-Scanned: by MailRoute Received: from 009.lax.mailroute.net ([127.0.0.1]) by localhost (009.lax [127.0.0.1]) (mroute_mailscanner, port 10029) with LMTP id tRdbjoKXvXAK; Tue, 19 Nov 2024 00:29:00 +0000 (UTC) Received: from bvanassche.mtv.corp.google.com (unknown [104.135.204.82]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: bvanassche@acm.org) by 009.lax.mailroute.net (Postfix) with ESMTPSA id 4XsljR3wzKzlgVXv; Tue, 19 Nov 2024 00:28:59 +0000 (UTC) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, Christoph Hellwig , Damien Le Moal , Jaegeuk Kim , Bart Van Assche Subject: [PATCH v16 19/26] blk-zoned: Add an argument to blk_zone_plug_bio() Date: Mon, 18 Nov 2024 16:28:08 -0800 Message-ID: <20241119002815.600608-20-bvanassche@acm.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241119002815.600608-1-bvanassche@acm.org> References: <20241119002815.600608-1-bvanassche@acm.org> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Prepare for preserving the order of pipelined zoned writes per zone. Signed-off-by: Bart Van Assche --- block/blk-mq.c | 2 +- block/blk-zoned.c | 3 ++- drivers/md/dm.c | 2 +- include/linux/blkdev.h | 6 ++++-- 4 files changed, 8 insertions(+), 5 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 1302ccbf2a7d..b396ddfe7165 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -3135,7 +3135,7 @@ void blk_mq_submit_bio(struct bio *bio) if (blk_mq_attempt_bio_merge(q, bio, nr_segs)) goto queue_exit; - if (blk_queue_is_zoned(q) && blk_zone_plug_bio(bio, nr_segs)) + if (blk_queue_is_zoned(q) && blk_zone_plug_bio(bio, nr_segs, &swq_cpu)) goto queue_exit; new_request: diff --git a/block/blk-zoned.c b/block/blk-zoned.c index 59f6559b94da..8d0fac14c837 100644 --- a/block/blk-zoned.c +++ b/block/blk-zoned.c @@ -1049,6 +1049,7 @@ static bool blk_zone_wplug_handle_write(struct bio *bio, unsigned int nr_segs) * blk_zone_plug_bio - Handle a zone write BIO with zone write plugging * @bio: The BIO being submitted * @nr_segs: The number of physical segments of @bio + * @swq_cpu: [out] CPU of the software queue to which the bio should be queued * * Handle write, write zeroes and zone append operations requiring emulation * using zone write plugging. @@ -1057,7 +1058,7 @@ static bool blk_zone_wplug_handle_write(struct bio *bio, unsigned int nr_segs) * write plug. Otherwise, return false to let the submission path process * @bio normally. */ -bool blk_zone_plug_bio(struct bio *bio, unsigned int nr_segs) +bool blk_zone_plug_bio(struct bio *bio, unsigned int nr_segs, int *swq_cpu) { struct block_device *bdev = bio->bi_bdev; diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 19230404d8c2..cd85c9b6a463 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -1796,7 +1796,7 @@ static inline bool dm_zone_bio_needs_split(struct mapped_device *md, } static inline bool dm_zone_plug_bio(struct mapped_device *md, struct bio *bio) { - return dm_emulate_zone_append(md) && blk_zone_plug_bio(bio, 0); + return dm_emulate_zone_append(md) && blk_zone_plug_bio(bio, 0, NULL); } static blk_status_t __send_zone_reset_all_emulated(struct clone_info *ci, diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 72be33d02d1f..1c28ec5b75ce 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -694,13 +694,15 @@ static inline unsigned int disk_nr_zones(struct gendisk *disk) { return disk->nr_zones; } -bool blk_zone_plug_bio(struct bio *bio, unsigned int nr_segs); +struct blk_mq_ctx; +bool blk_zone_plug_bio(struct bio *bio, unsigned int nr_segs, int *swq_cpu); #else /* CONFIG_BLK_DEV_ZONED */ static inline unsigned int disk_nr_zones(struct gendisk *disk) { return 0; } -static inline bool blk_zone_plug_bio(struct bio *bio, unsigned int nr_segs) +static inline bool blk_zone_plug_bio(struct bio *bio, unsigned int nr_segs, + int *swq_cpu) { return false; } From patchwork Tue Nov 19 00:28:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 13879279 Received: from 009.lax.mailroute.net (009.lax.mailroute.net [199.89.1.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2CCA2610B; Tue, 19 Nov 2024 00:29:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=199.89.1.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976150; cv=none; b=V9mfJ5gRO1FkqUN9O/Fa306zEBXwa8CkADh8GZPQjyYt8oP5ToqmIji3SnGKH8g07+7z+DwSbEcu0LFS61QZVrRJaOKUs+/HwU/vIQ+jIjUh5fhSZRGy/Ed3hjZFQRhGLYHBsyMpeJDOTaw+QD2ZbUTSzCIzX7NAoX7wlOUxVzo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976150; c=relaxed/simple; bh=Q6QmAIcqJC5cupCYeyNx1mlrrc+kt+kKNMpwXvrx8wo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=k2ziHRCSdasexwbrikiYjFWWzRt+k+4atQIT2uYPageR0cgRR2skqiD4ISd6cgy5EoEEz7LZu4xA6gKxvA74joSMbLE0yvTyS1ohc3IbwN4xI2OO5VPJ+If5aHnL3O2U7viJpyc8/DGDlXpWmaAkbDcJ8+x2Ulbg2W1EbK+pZsw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org; spf=pass smtp.mailfrom=acm.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b=tj7a5UYk; arc=none smtp.client-ip=199.89.1.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=acm.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b="tj7a5UYk" Received: from localhost (localhost [127.0.0.1]) by 009.lax.mailroute.net (Postfix) with ESMTP id 4Xsljc6dGmzlgMVV; Tue, 19 Nov 2024 00:29:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=acm.org; h= content-transfer-encoding:mime-version:references:in-reply-to :x-mailer:message-id:date:date:subject:subject:from:from :received:received; s=mr01; t=1731976142; x=1734568143; bh=Tt1/D 0e9LV64r7vKCUB7bjczn1bJ4ahHGvzAEU7qeKI=; b=tj7a5UYks/qR+DuOGqqww nOj4d2sgL9q7K7q9RTla54V348Z7Y0qIBmyIlLxv2bXQdoncFq0otReMIpJpwRdZ njF2eExnzAkGnqW3yar+4uM1Oq8J+fkbjGqq3+2qfdaHep1NcghljY0Oh1IOE+h/ aFzGXs0dDZxJxLRICUDz5f9aw197wP/WUBibLDA50Mn5yzS7RmwPaKbjLiQlv30e ZEsC3uXcNWBcTvXAnjdn5tE1x0pUdUqvw1C/PcHFO4wp7RhOF9oR9Vw+mewJtODB qazftFw5K+COy3THxEX+y9qYk6yPWwS3BPRU3mp72ss/ZsrBNWZjFVfx+b0DArSg Q== X-Virus-Scanned: by MailRoute Received: from 009.lax.mailroute.net ([127.0.0.1]) by localhost (009.lax [127.0.0.1]) (mroute_mailscanner, port 10029) with LMTP id HRXXmZYV1hA7; Tue, 19 Nov 2024 00:29:02 +0000 (UTC) Received: from bvanassche.mtv.corp.google.com (unknown [104.135.204.82]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: bvanassche@acm.org) by 009.lax.mailroute.net (Postfix) with ESMTPSA id 4XsljT162yzlgMVb; Tue, 19 Nov 2024 00:29:00 +0000 (UTC) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, Christoph Hellwig , Damien Le Moal , Jaegeuk Kim , Bart Van Assche Subject: [PATCH v16 20/26] blk-zoned: Support pipelining of zoned writes Date: Mon, 18 Nov 2024 16:28:09 -0800 Message-ID: <20241119002815.600608-21-bvanassche@acm.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241119002815.600608-1-bvanassche@acm.org> References: <20241119002815.600608-1-bvanassche@acm.org> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Support pipelining of zoned writes if the block driver preserves the write order per hardware queue. Track per zone to which software queue writes have been queued. If zoned writes are pipelined, submit new writes to the same software queue as the writes that are already in progress. This prevents reordering by submitting requests for the same zone to different software or hardware queues. Signed-off-by: Bart Van Assche --- block/blk-zoned.c | 40 ++++++++++++++++++++++++++++------------ 1 file changed, 28 insertions(+), 12 deletions(-) diff --git a/block/blk-zoned.c b/block/blk-zoned.c index 8d0fac14c837..f934c0cb5fdd 100644 --- a/block/blk-zoned.c +++ b/block/blk-zoned.c @@ -8,6 +8,7 @@ * Copyright (c) 2016, Damien Le Moal * Copyright (c) 2016, Western Digital * Copyright (c) 2024, Western Digital Corporation or its affiliates. + * Copyright 2024 Google LLC */ #include @@ -59,6 +60,8 @@ static const char *const zone_cond_name[] = { * as a number of 512B sectors. * @wp_offset_compl: End offset for completed zoned writes as a number of 512 * byte sectors. + * @swq_cpu: Software queue to submit writes to for drivers that preserve the + * write order. * @bio_list: The list of BIOs that are currently plugged. * @bio_work: Work struct to handle issuing of plugged BIOs * @rcu_head: RCU head to free zone write plugs with an RCU grace period. @@ -73,6 +76,7 @@ struct blk_zone_wplug { unsigned int zone_no; unsigned int wp_offset; unsigned int wp_offset_compl; + int swq_cpu; struct bio_list bio_list; struct work_struct bio_work; struct rcu_head rcu_head; @@ -82,8 +86,7 @@ struct blk_zone_wplug { /* * Zone write plug flags bits: * - BLK_ZONE_WPLUG_PLUGGED: Indicates that the zone write plug is plugged, - * that is, that write BIOs are being throttled due to a write BIO already - * being executed or the zone write plug bio list is not empty. + * that is, that write BIOs are being throttled. * - BLK_ZONE_WPLUG_ERROR: Indicates that a write error happened which will be * recovered with a report zone to update the zone write pointer offset. * - BLK_ZONE_WPLUG_UNHASHED: Indicates that the zone write plug was removed @@ -535,6 +538,7 @@ static struct blk_zone_wplug *disk_get_and_lock_zone_wplug(struct gendisk *disk, zwplug->zone_no = zno; zwplug->wp_offset = sector & (disk->queue->limits.chunk_sectors - 1); zwplug->wp_offset_compl = 0; + zwplug->swq_cpu = -1; bio_list_init(&zwplug->bio_list); INIT_WORK(&zwplug->bio_work, blk_zone_wplug_bio_work); zwplug->disk = disk; @@ -973,7 +977,8 @@ static bool blk_zone_wplug_prepare_bio(struct blk_zone_wplug *zwplug, return false; } -static bool blk_zone_wplug_handle_write(struct bio *bio, unsigned int nr_segs) +static bool blk_zone_wplug_handle_write(struct bio *bio, unsigned int nr_segs, + int *swq_cpu) { struct gendisk *disk = bio->bi_bdev->bd_disk; sector_t sector = bio->bi_iter.bi_sector; @@ -1017,11 +1022,19 @@ static bool blk_zone_wplug_handle_write(struct bio *bio, unsigned int nr_segs) bio_set_flag(bio, BIO_ZONE_WRITE_PLUGGING); /* - * If the zone is already plugged or has a pending error, add the BIO - * to the plug BIO list. Otherwise, plug and let the BIO execute. + * If the zone has a pending error or is already plugged, add the BIO + * to the plug BIO list. Otherwise, execute the BIO and plug if not yet + * plugged and if the write order is not preserved. */ - if (zwplug->flags & BLK_ZONE_WPLUG_BUSY) + if (zwplug->flags & BLK_ZONE_WPLUG_BUSY) { goto plug; + } else if (disk->queue->limits.driver_preserves_write_order) { + if (zwplug->swq_cpu < 0) + zwplug->swq_cpu = raw_smp_processor_id(); + *swq_cpu = zwplug->swq_cpu; + } else { + zwplug->flags |= BLK_ZONE_WPLUG_PLUGGED; + } /* * If an error is detected when preparing the BIO, add it to the BIO @@ -1030,8 +1043,6 @@ static bool blk_zone_wplug_handle_write(struct bio *bio, unsigned int nr_segs) if (!blk_zone_wplug_prepare_bio(zwplug, bio)) goto plug; - zwplug->flags |= BLK_ZONE_WPLUG_PLUGGED; - spin_unlock_irqrestore(&zwplug->lock, flags); return false; @@ -1107,7 +1118,7 @@ bool blk_zone_plug_bio(struct bio *bio, unsigned int nr_segs, int *swq_cpu) fallthrough; case REQ_OP_WRITE: case REQ_OP_WRITE_ZEROES: - return blk_zone_wplug_handle_write(bio, nr_segs); + return blk_zone_wplug_handle_write(bio, nr_segs, swq_cpu); case REQ_OP_ZONE_RESET: return blk_zone_wplug_handle_reset_or_finish(bio, 0); case REQ_OP_ZONE_FINISH: @@ -1131,7 +1142,6 @@ static void disk_zone_wplug_schedule_bio_work(struct gendisk *disk, * of the next plugged BIO. blk_zone_wplug_bio_work() will release the * reference we take here. */ - WARN_ON_ONCE(!(zwplug->flags & BLK_ZONE_WPLUG_PLUGGED)); refcount_inc(&zwplug->ref); queue_work(disk->zone_wplugs_wq, &zwplug->bio_work); } @@ -1185,6 +1195,9 @@ static void disk_zone_wplug_unplug_bio(struct gendisk *disk, zwplug->flags &= ~BLK_ZONE_WPLUG_PLUGGED; + if (refcount_read(&zwplug->ref) == 2) + zwplug->swq_cpu = -1; + /* * If the zone is full (it was fully written or finished, or empty * (it was reset), remove its zone write plug from the hash table. @@ -1937,6 +1950,7 @@ static void queue_zone_wplug_show(struct blk_zone_wplug *zwplug, unsigned int zwp_zone_no, zwp_ref; unsigned int zwp_bio_list_size; unsigned long flags; + int swq_cpu; spin_lock_irqsave(&zwplug->lock, flags); zwp_zone_no = zwplug->zone_no; @@ -1945,13 +1959,15 @@ static void queue_zone_wplug_show(struct blk_zone_wplug *zwplug, zwp_wp_offset = zwplug->wp_offset; zwp_wp_offset_compl = zwplug->wp_offset_compl; zwp_bio_list_size = bio_list_size(&zwplug->bio_list); + swq_cpu = zwplug->swq_cpu; spin_unlock_irqrestore(&zwplug->lock, flags); bool all_zwr_inserted = blk_zone_all_zwr_inserted(zwplug); - seq_printf(m, "zone_no %u flags 0x%x ref %u wp_offset %u wp_offset_compl %u bio_list_size %u all_zwr_inserted %d\n", + seq_printf(m, "zone_no %u flags 0x%x ref %u wp_offset %u wp_offset_compl %u bio_list_size %u all_zwr_inserted %d swq_cpu %d\n", zwp_zone_no, zwp_flags, zwp_ref, zwp_wp_offset, - zwp_wp_offset_compl, zwp_bio_list_size, all_zwr_inserted); + zwp_wp_offset_compl, zwp_bio_list_size, all_zwr_inserted, + swq_cpu); } int queue_zone_wplugs_show(void *data, struct seq_file *m) From patchwork Tue Nov 19 00:28:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 13879278 Received: from 009.lax.mailroute.net (009.lax.mailroute.net [199.89.1.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 09388DDD9; Tue, 19 Nov 2024 00:29:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=199.89.1.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976150; cv=none; b=m4ymKYP2PrYOhwqADtRE9c1+WPogR4w3wIUtN7bLlAzqHVsKfPFEo92XDbgLjo58WzTnyOpag6kaY7ZE6g/hfChWnNFvbBLNyzMYg+La7KxnSFttVzPZHuPi/1YHV4zAPof0QI1ho9KllkwOzfLXgU7jolVj7EgwWD7KzbmOg18= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976150; c=relaxed/simple; bh=kLLkt4Fs0e4m0K2fT+/PzOsB07T/v4pilQwrgq8wzs0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rE1lsH0RHDl9d7A53kJNAnkpg04+70XB1/SGh81vxgdrg+1hDG4w16/mqgmSO7PX3ySe2VZA+7H1Gip7oWSx0GnTN3acy37GJ12WobaZ3xv7Sto5gSOXP09dqADvFqecYYtry2Q4t74LfmNWpl5cmeWneHOeB9OL1gQmrDuniAA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org; spf=pass smtp.mailfrom=acm.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b=t2LccxBM; arc=none smtp.client-ip=199.89.1.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=acm.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b="t2LccxBM" Received: from localhost (localhost [127.0.0.1]) by 009.lax.mailroute.net (Postfix) with ESMTP id 4Xsljc5dj7zlgMVP; Tue, 19 Nov 2024 00:29:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=acm.org; h= content-transfer-encoding:mime-version:references:in-reply-to :x-mailer:message-id:date:date:subject:subject:from:from :received:received; s=mr01; t=1731976144; x=1734568145; bh=nx4KB RxMaCf8LrtAVRN8jammCUAU/H3BD9B61OciwYM=; b=t2LccxBMxnagLI07+TFDz MpuXTCRcaMkJNi0AkSFi7NrcNgn2uwQnNfLBWgy02AhfssCyByRoYPvL61j48WIg I+pp1KRrF5AdRGB4n0cuhU0V+F5DDwP1TxkZvc0tlX1zJC3xR/CrKmJQmtICcB59 Gw9rclRJwWVdKXpbGCAQ1q+yLwUmLzqYquuyGExGXPeXgIzZRxiJAp8UEaNYioL/ hOt895qOyq9TH5qbPBP3aTYmxPLVw47WQINPbJ+X0w0c9RX+WAnuniszMWtSZijc 4sLNDCgz6k+IOsYvZeAp+zWCi/JnR6cWpB27Rrh/MlaLPEsGWf6/IbmKatQRfoyy A== X-Virus-Scanned: by MailRoute Received: from 009.lax.mailroute.net ([127.0.0.1]) by localhost (009.lax [127.0.0.1]) (mroute_mailscanner, port 10029) with LMTP id MVrqmIfq7uqQ; Tue, 19 Nov 2024 00:29:04 +0000 (UTC) Received: from bvanassche.mtv.corp.google.com (unknown [104.135.204.82]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: bvanassche@acm.org) by 009.lax.mailroute.net (Postfix) with ESMTPSA id 4XsljV5PW1zlgMVV; Tue, 19 Nov 2024 00:29:02 +0000 (UTC) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, Christoph Hellwig , Damien Le Moal , Jaegeuk Kim , Bart Van Assche , "Martin K. Petersen" , Ming Lei Subject: [PATCH v16 21/26] scsi: core: Retry unaligned zoned writes Date: Mon, 18 Nov 2024 16:28:10 -0800 Message-ID: <20241119002815.600608-22-bvanassche@acm.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241119002815.600608-1-bvanassche@acm.org> References: <20241119002815.600608-1-bvanassche@acm.org> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 If zoned writes (REQ_OP_WRITE) for a sequential write required zone have a starting LBA that differs from the write pointer, e.g. because zoned writes have been reordered, then the storage device will respond with an UNALIGNED WRITE COMMAND error. Retry commands that failed with an unaligned write error. The block layer core will sort the SCSI commands per LBA before these are resubmitted. If the write order is preserved, increase the number of retries for write commands sent to a sequential zone to the maximum number of outstanding commands because in the worst case the number of times reordered zoned writes have to be retried is (number of outstanding writes per sequential zone) - 1. Reviewed-by: Damien Le Moal Cc: Martin K. Petersen Cc: Christoph Hellwig Cc: Ming Lei Signed-off-by: Bart Van Assche --- drivers/scsi/scsi_error.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/drivers/scsi/scsi_error.c b/drivers/scsi/scsi_error.c index 10154d78e336..24cd8e8538e5 100644 --- a/drivers/scsi/scsi_error.c +++ b/drivers/scsi/scsi_error.c @@ -700,6 +700,22 @@ enum scsi_disposition scsi_check_sense(struct scsi_cmnd *scmd) fallthrough; case ILLEGAL_REQUEST: + /* + * Unaligned write command. This may indicate that zoned writes + * have been received by the device in the wrong order. If write + * pipelining is enabled, retry. + */ + if (sshdr.asc == 0x21 && sshdr.ascq == 0x04 && + req->q->limits.driver_preserves_write_order && + blk_rq_is_seq_zoned_write(req) && + scsi_cmd_retry_allowed(scmd)) { + SCSI_LOG_ERROR_RECOVERY(1, + sdev_printk(KERN_WARNING, scmd->device, + "Retrying unaligned write at LBA %#llx.\n", + scsi_get_lba(scmd))); + return NEEDS_RETRY; + } + if (sshdr.asc == 0x20 || /* Invalid command operation code */ sshdr.asc == 0x21 || /* Logical block address out of range */ sshdr.asc == 0x22 || /* Invalid function */ From patchwork Tue Nov 19 00:28:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 13879280 Received: from 009.lax.mailroute.net (009.lax.mailroute.net [199.89.1.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EF22F136337; Tue, 19 Nov 2024 00:29:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=199.89.1.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976152; cv=none; b=fCP9bfid1zBOX8kYIpHX89y6mR/QnmM3u2aaDOROkcBWPsqM+/k5eH1UtHd22zk+jDcrRMn9jo6vUyZ/13Wk3oa/iU8yncXvjRvO9ikdlfYQmqYurFX71Yq14HBnCRAj0VgVDISRyFfyv1eOp9HqdSp6Bm202PjwZ55UQ4wr7wo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976152; c=relaxed/simple; bh=QkqlwuuNYl4VVEnHfU3DWlFW7oW9V0AIugQBhAHScm8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CNiuwuRAoYbfCPFm+LGzsuwEaLqVGvvo5OrOondKVDh/NpEdrsWBzLCbUsBdXbe3HS02HLtQTr84zmbHHD8MwMkUzci2I2T6o9/W7pixsR9PuoR0HRStRACmIJ1lolpChuiFTPcQFm0Mwiv724upeNOIZGbPeT7yFqQiqC2bIwQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org; spf=pass smtp.mailfrom=acm.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b=TxF25nA+; arc=none smtp.client-ip=199.89.1.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=acm.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b="TxF25nA+" Received: from localhost (localhost [127.0.0.1]) by 009.lax.mailroute.net (Postfix) with ESMTP id 4Xsljf36xwzlgMVj; Tue, 19 Nov 2024 00:29:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=acm.org; h= content-transfer-encoding:mime-version:references:in-reply-to :x-mailer:message-id:date:date:subject:subject:from:from :received:received; s=mr01; t=1731976146; x=1734568147; bh=Q5vhR 0hiuCOQ1QAEux5brr8vJyia4iEYYO5oL+JQtLk=; b=TxF25nA+F7GnthN1miS21 3ZKAeWSrW6sFwc42w8cz50cApIeFOreu2HZ1rMaBN2uQL5TiDpq/Rp2s4oUWmAhA blkz536QOkGfZ1Qvt8VMDi5oujNlWuCdrRPui5knQcX91wck9HLmnqwlIP0iH5O3 61M4o+pmwUUyV5L61D0M9hOxDE8gffq+0/kNT3Xp8wBK/duIk/NMS+hhtRS23s4B qhXmBsljKkk1+eMOJS3x5tCAfP2V4u1MrVQ4PXjc8a/RTJw+3Lv59JQZlsPIm2K6 wkKkxWd8cIeixTC5sudX1tUKhkmk/telxRBPl79VNoqqvc8h19IeyV9LAQyffeb0 Q== X-Virus-Scanned: by MailRoute Received: from 009.lax.mailroute.net ([127.0.0.1]) by localhost (009.lax [127.0.0.1]) (mroute_mailscanner, port 10029) with LMTP id kL_jt8GtTVzZ; Tue, 19 Nov 2024 00:29:06 +0000 (UTC) Received: from bvanassche.mtv.corp.google.com (unknown [104.135.204.82]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: bvanassche@acm.org) by 009.lax.mailroute.net (Postfix) with ESMTPSA id 4XsljX5jSLzlgMVd; Tue, 19 Nov 2024 00:29:04 +0000 (UTC) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, Christoph Hellwig , Damien Le Moal , Jaegeuk Kim , Bart Van Assche , "Martin K. Petersen" , Ming Lei Subject: [PATCH v16 22/26] scsi: sd: Increase retry count for zoned writes Date: Mon, 18 Nov 2024 16:28:11 -0800 Message-ID: <20241119002815.600608-23-bvanassche@acm.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241119002815.600608-1-bvanassche@acm.org> References: <20241119002815.600608-1-bvanassche@acm.org> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 If the write order is preserved, increase the number of retries for write commands sent to a sequential zone to the maximum number of outstanding commands because in the worst case the number of times reordered zoned writes have to be retried is (number of outstanding writes per sequential zone) - 1. Cc: Damien Le Moal Cc: Martin K. Petersen Cc: Christoph Hellwig Cc: Ming Lei Signed-off-by: Bart Van Assche --- drivers/scsi/sd.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/drivers/scsi/sd.c b/drivers/scsi/sd.c index 8947dab132d7..117fce3b1c7b 100644 --- a/drivers/scsi/sd.c +++ b/drivers/scsi/sd.c @@ -1408,6 +1408,13 @@ static blk_status_t sd_setup_read_write_cmnd(struct scsi_cmnd *cmd) cmd->transfersize = sdp->sector_size; cmd->underflow = nr_blocks << 9; cmd->allowed = sdkp->max_retries; + /* + * Increase the number of allowed retries for zoned writes if the driver + * preserves the command order. + */ + if (rq->q->limits.driver_preserves_write_order && + blk_rq_is_seq_zoned_write(rq)) + cmd->allowed += rq->q->nr_requests; cmd->sdb.length = nr_blocks * sdp->sector_size; SCSI_LOG_HLQUEUE(1, From patchwork Tue Nov 19 00:28:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 13879281 Received: from 009.lax.mailroute.net (009.lax.mailroute.net [199.89.1.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 17A38A93D; Tue, 19 Nov 2024 00:29:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=199.89.1.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976156; cv=none; b=DIcwMOhYb0hwIpsZN2ah34/MAIVwSf607QPbMcKcoROpbJX4PSUgOsL+9q4GpneZXdelR9W6qnfDRW5dvpzwofVFSbRFwUjakLujWjRmoFyW4Iaot0cahb5smZuJwy1Ym508dexEt6T5NsM2eGtCdrP3izgwh2+KRAzRVnZoawM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976156; c=relaxed/simple; bh=j4sz/qBNMekuofU1ktxXDEUv8Uh022s1uRb3hgBcN9s=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=g1e4DCybiakeQfiT5MHyJ0x4t78Uf+izjjBZXWH0IYVJsb3qG2EY1Do8dgnBfp3O8sfqS/Fl9EAQh9Z7eI9GXyo8WiAsaJ/1YhORUJn8Ir6Dxr4OKd/qMWKGsYLN8nVRktzcEoURLhU6NRwq7/HPJwHKLvHgVNOBwAA4ashD6J8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org; spf=pass smtp.mailfrom=acm.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b=fEL64GDZ; arc=none smtp.client-ip=199.89.1.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=acm.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b="fEL64GDZ" Received: from localhost (localhost [127.0.0.1]) by 009.lax.mailroute.net (Postfix) with ESMTP id 4Xsljk618fzlgMVl; Tue, 19 Nov 2024 00:29:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=acm.org; h= content-transfer-encoding:mime-version:references:in-reply-to :x-mailer:message-id:date:date:subject:subject:from:from :received:received; s=mr01; t=1731976148; x=1734568149; bh=HSoRR Z7v34DsckbfLaZU4jruxRl81/x/BHT3EsdK8So=; b=fEL64GDZdMKon94/empXM bSjxEMZocC93mnYGVsdngexs8/XTxygonzv1vZuW99GSKM/VxZfLWMMzzl10PFRk +MKZYdoAOtm6AUCa2YyVwN/xOaGebbuWpsW9IKoPQu0SINEusZWhs7C3p0DK2T4K JtxzAV4jpSOUDldPpqjmQ2khAMaaFmWtJIO0Te6g2vUtnbTXCOyVqTL33/p3vnNw X2mI1QAq/myRAhe148XFLxian5I2Kb1oyaOpjknRpUXXbOYPQY40YwT+4ILMX7w3 953ROHJ0FTTjYbEQXth0Yg7uJ2GBtvYwi8JMtn/xzh6qQHHH37pVBDB79mdELTin g== X-Virus-Scanned: by MailRoute Received: from 009.lax.mailroute.net ([127.0.0.1]) by localhost (009.lax [127.0.0.1]) (mroute_mailscanner, port 10029) with LMTP id 6khAYI1PbBwx; Tue, 19 Nov 2024 00:29:08 +0000 (UTC) Received: from bvanassche.mtv.corp.google.com (unknown [104.135.204.82]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: bvanassche@acm.org) by 009.lax.mailroute.net (Postfix) with ESMTPSA id 4XsljZ5zNmzlgVXv; Tue, 19 Nov 2024 00:29:06 +0000 (UTC) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, Christoph Hellwig , Damien Le Moal , Jaegeuk Kim , Bart Van Assche , Douglas Gilbert , "Martin K. Petersen" , Ming Lei Subject: [PATCH v16 23/26] scsi: scsi_debug: Add the preserves_write_order module parameter Date: Mon, 18 Nov 2024 16:28:12 -0800 Message-ID: <20241119002815.600608-24-bvanassche@acm.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241119002815.600608-1-bvanassche@acm.org> References: <20241119002815.600608-1-bvanassche@acm.org> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Zone write locking is not used for zoned devices if the block driver reports that it preserves the order of write commands. Make it easier to test not using zone write locking by adding support for setting the driver_preserves_write_order flag. Acked-by: Douglas Gilbert Reviewed-by: Damien Le Moal Cc: Martin K. Petersen Cc: Christoph Hellwig Cc: Ming Lei Signed-off-by: Bart Van Assche --- drivers/scsi/scsi_debug.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/drivers/scsi/scsi_debug.c b/drivers/scsi/scsi_debug.c index b52513eeeafa..1b5a9cc27fbd 100644 --- a/drivers/scsi/scsi_debug.c +++ b/drivers/scsi/scsi_debug.c @@ -927,6 +927,7 @@ static int dix_reads; static int dif_errors; /* ZBC global data */ +static bool sdeb_preserves_write_order; static bool sdeb_zbc_in_use; /* true for host-aware and host-managed disks */ static int sdeb_zbc_zone_cap_mb; static int sdeb_zbc_zone_size_mb; @@ -5881,10 +5882,14 @@ static struct sdebug_dev_info *find_build_dev_info(struct scsi_device *sdev) static int scsi_debug_slave_alloc(struct scsi_device *sdp) { + struct request_queue *q = sdp->request_queue; + if (sdebug_verbose) pr_info("slave_alloc <%u %u %u %llu>\n", sdp->host->host_no, sdp->channel, sdp->id, sdp->lun); + q->limits.driver_preserves_write_order = sdeb_preserves_write_order; + return 0; } @@ -6620,6 +6625,8 @@ module_param_named(statistics, sdebug_statistics, bool, S_IRUGO | S_IWUSR); module_param_named(strict, sdebug_strict, bool, S_IRUGO | S_IWUSR); module_param_named(submit_queues, submit_queues, int, S_IRUGO); module_param_named(poll_queues, poll_queues, int, S_IRUGO); +module_param_named(preserves_write_order, sdeb_preserves_write_order, bool, + S_IRUGO); module_param_named(tur_ms_to_ready, sdeb_tur_ms_to_ready, int, S_IRUGO); module_param_named(unmap_alignment, sdebug_unmap_alignment, int, S_IRUGO); module_param_named(unmap_granularity, sdebug_unmap_granularity, int, S_IRUGO); @@ -6692,6 +6699,8 @@ MODULE_PARM_DESC(opts, "1->noise, 2->medium_err, 4->timeout, 8->recovered_err... MODULE_PARM_DESC(per_host_store, "If set, next positive add_host will get new store (def=0)"); MODULE_PARM_DESC(physblk_exp, "physical block exponent (def=0)"); MODULE_PARM_DESC(poll_queues, "support for iouring iopoll queues (1 to max(submit_queues - 1))"); +MODULE_PARM_DESC(preserves_write_order, + "Whether or not to inform the block layer that this driver preserves the order of WRITE commands (def=0)"); MODULE_PARM_DESC(ptype, "SCSI peripheral type(def=0[disk])"); MODULE_PARM_DESC(random, "If set, uniformly randomize command duration between 0 and delay_in_ns"); MODULE_PARM_DESC(removable, "claim to have removable media (def=0)"); From patchwork Tue Nov 19 00:28:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 13879283 Received: from 009.lax.mailroute.net (009.lax.mailroute.net [199.89.1.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1C3D5136337; Tue, 19 Nov 2024 00:29:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=199.89.1.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976158; cv=none; b=fAK0JC5AkTjnjkyHqLxmLvgCscQBZWFVkZxhZ4HmUVU4W73uJhhdw95AuAnbccDU2nKgxS8SnDyNOk3OL/G5qNpZMFuD7GkTJ2D7zc5Qi5jTpDxZ/23bswaIdRkA0/Myt6GDvBHjRU3/AEKYFrpuWDMDV6oYw39xAtmWMQ5ShAM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976158; c=relaxed/simple; bh=s4x8TqaOOtoOVatOrd6FZFtmAncBHnf2VpTw8g2RbDA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=c8LygWSUeLPz7rkzng1yHLy/Hef2OLe+zw06MBAMiVSIpNPW99eyvUbznr+CRlz/xLClcxjgJop+8E2Ql9b1GDkcWz2C+7OqgIc5UOvpyGlEqRg8r9Fli74i+XgUvkOCuNSoTbvGob5wvXOGAqth3BnpwsVRkYUy9sthXViG3Tw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org; spf=pass smtp.mailfrom=acm.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b=rZKHRyEl; arc=none smtp.client-ip=199.89.1.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=acm.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b="rZKHRyEl" Received: from localhost (localhost [127.0.0.1]) by 009.lax.mailroute.net (Postfix) with ESMTP id 4Xsljm3w4RzlgVnN; Tue, 19 Nov 2024 00:29:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=acm.org; h= content-transfer-encoding:mime-version:references:in-reply-to :x-mailer:message-id:date:date:subject:subject:from:from :received:received; s=mr01; t=1731976151; x=1734568152; bh=nW0Ev n6u34f29zcjp/DABwnQSxUaNNmA/7rGD/r3KBE=; b=rZKHRyEl26UUTVyk0V06i oCjv8cd5GrEEk4H3c58McF9B28TG429biWMNkeASiLQ0rprq5i4oayzEkFAl6+eP CW3awSix1U8PD+xUgmAuuqVrJkUWe36Ze5eKUtCcnptzczfHBzkc8SXZ4QYpXPVo i9Y+zjwDGyLjRgLGBZ6S8MWmxHDyDhb2tFbdGR+08ETdzgHwwCSVFrIRvgDxp4Rx 6qyt+2kRIujBXDpj8lvo6+OhK6jlrTIUzMmgUfoLUxjyY2ZYrBON96zqzdmCGD2G 5400SuYu7JqScF6LLjJFiReWcbIHHku5+i1hSOLPExHbrN58aWQfCspmEkDV02iu A== X-Virus-Scanned: by MailRoute Received: from 009.lax.mailroute.net ([127.0.0.1]) by localhost (009.lax [127.0.0.1]) (mroute_mailscanner, port 10029) with LMTP id 5toeUiHcoIq6; Tue, 19 Nov 2024 00:29:11 +0000 (UTC) Received: from bvanassche.mtv.corp.google.com (unknown [104.135.204.82]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: bvanassche@acm.org) by 009.lax.mailroute.net (Postfix) with ESMTPSA id 4Xsljd0kvdzlgMVb; Tue, 19 Nov 2024 00:29:08 +0000 (UTC) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, Christoph Hellwig , Damien Le Moal , Jaegeuk Kim , Bart Van Assche , Douglas Gilbert , "Martin K. Petersen" , Ming Lei Subject: [PATCH v16 24/26] scsi: scsi_debug: Support injecting unaligned write errors Date: Mon, 18 Nov 2024 16:28:13 -0800 Message-ID: <20241119002815.600608-25-bvanassche@acm.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241119002815.600608-1-bvanassche@acm.org> References: <20241119002815.600608-1-bvanassche@acm.org> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Allow user space software, e.g. a blktests test, to inject unaligned write errors. Acked-by: Douglas Gilbert Reviewed-by: Damien Le Moal Cc: Martin K. Petersen Cc: Christoph Hellwig Cc: Ming Lei Signed-off-by: Bart Van Assche --- drivers/scsi/scsi_debug.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/drivers/scsi/scsi_debug.c b/drivers/scsi/scsi_debug.c index 1b5a9cc27fbd..2ff3a24d791e 100644 --- a/drivers/scsi/scsi_debug.c +++ b/drivers/scsi/scsi_debug.c @@ -193,6 +193,7 @@ static const char *sdebug_version_date = "20210520"; #define SDEBUG_OPT_NO_CDB_NOISE 0x4000 #define SDEBUG_OPT_HOST_BUSY 0x8000 #define SDEBUG_OPT_CMD_ABORT 0x10000 +#define SDEBUG_OPT_UNALIGNED_WRITE 0x20000 #define SDEBUG_OPT_ALL_NOISE (SDEBUG_OPT_NOISE | SDEBUG_OPT_Q_NOISE | \ SDEBUG_OPT_RESET_NOISE) #define SDEBUG_OPT_ALL_INJECTING (SDEBUG_OPT_RECOVERED_ERR | \ @@ -200,7 +201,8 @@ static const char *sdebug_version_date = "20210520"; SDEBUG_OPT_DIF_ERR | SDEBUG_OPT_DIX_ERR | \ SDEBUG_OPT_SHORT_TRANSFER | \ SDEBUG_OPT_HOST_BUSY | \ - SDEBUG_OPT_CMD_ABORT) + SDEBUG_OPT_CMD_ABORT | \ + SDEBUG_OPT_UNALIGNED_WRITE) #define SDEBUG_OPT_RECOV_DIF_DIX (SDEBUG_OPT_RECOVERED_ERR | \ SDEBUG_OPT_DIF_ERR | SDEBUG_OPT_DIX_ERR) @@ -4191,6 +4193,14 @@ static int resp_write_dt0(struct scsi_cmnd *scp, struct sdebug_dev_info *devip) u8 *cmd = scp->cmnd; bool meta_data_locked = false; + if (unlikely(sdebug_opts & SDEBUG_OPT_UNALIGNED_WRITE && + atomic_read(&sdeb_inject_pending))) { + atomic_set(&sdeb_inject_pending, 0); + mk_sense_buffer(scp, ILLEGAL_REQUEST, LBA_OUT_OF_RANGE, + UNALIGNED_WRITE_ASCQ); + return check_condition_result; + } + switch (cmd[0]) { case WRITE_16: ei_lba = 0; From patchwork Tue Nov 19 00:28:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 13879282 Received: from 009.lax.mailroute.net (009.lax.mailroute.net [199.89.1.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CB4AB17557; Tue, 19 Nov 2024 00:29:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=199.89.1.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976157; cv=none; b=txVzk4EcFD2Q9vn1WwKc0sKIz/uJTvwzwupH05s28x+CYAXjJDgDeAmQf+bo5HckX7+oqbQ2jwGjC8lqZYR7c1sMKyJVvmxJLDV9BloZzkkoHl47gMcjdC8OwNmxE6uOM2c4EngFO0qP82y4/c75xZmlHPTYkIVL9IJ2hlLNNnY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976157; c=relaxed/simple; bh=bk9zvHFPBucCSXhMT9N1+kTQJs/laUB3R64x/Vx3Okg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ctdfXqbHxw45FZO1GmAC1CA/uKqwwimr1UGxdYhqaQWe2+vEAcfCXOeurpiXNwAHrhW5ZAbHRdNyi06GVJqXOV9S6m1J90PbnrBEZjcM2aIQBmZ+vRgve0OjLcfBD6XUgydEQieORR1ydefRcg6++bUpQvmIx0sLkk4BWCgfWr8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org; spf=pass smtp.mailfrom=acm.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b=wg+OGLnm; arc=none smtp.client-ip=199.89.1.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=acm.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b="wg+OGLnm" Received: from localhost (localhost [127.0.0.1]) by 009.lax.mailroute.net (Postfix) with ESMTP id 4Xsljl44sBzlgVXv; Tue, 19 Nov 2024 00:29:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=acm.org; h= content-transfer-encoding:mime-version:references:in-reply-to :x-mailer:message-id:date:date:subject:subject:from:from :received:received; s=mr01; t=1731976152; x=1734568153; bh=sdYDv 5rHj+PM9ZB2zWOgqt4k0jVu9cQDS7ai8yA3A5E=; b=wg+OGLnm5RqOnb4NFV0YC mqb36NkLq/Hc+lbFBbLjSc/auJP9JIsK0vCkKLrAdXq1ep2ColFW8GYVrJnk910W HUz6DEj5vtXYxwip0fU6s2+TRFW2Bz/B/c9RBtD5L9uttccEYyxpakbZgYbCkpSf SxHlgk++gDFQpp4M1Zo0AigZynkzpNqwYIhiZQIr9/IW4A3d1C035Sf+/xGzpILW Q8g4s+JG2BKo7nKXgEVw/PQuthyBWZmwZPOVR/oKSBpcPIWDml1WXzmVPmvWpx50 0IxKaqniV0qoSOb9EijV8h7dNcgoBSC4P3KLjoVyU8dJ7Bt3cb0mdxy57y529K/K g== X-Virus-Scanned: by MailRoute Received: from 009.lax.mailroute.net ([127.0.0.1]) by localhost (009.lax [127.0.0.1]) (mroute_mailscanner, port 10029) with LMTP id Jx4RXGKLqb2m; Tue, 19 Nov 2024 00:29:12 +0000 (UTC) Received: from bvanassche.mtv.corp.google.com (unknown [104.135.204.82]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: bvanassche@acm.org) by 009.lax.mailroute.net (Postfix) with ESMTPSA id 4Xsljg2ZkMzlgVnN; Tue, 19 Nov 2024 00:29:11 +0000 (UTC) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, Christoph Hellwig , Damien Le Moal , Jaegeuk Kim , Bart Van Assche Subject: [PATCH v16 25/26] scsi: scsi_debug: Skip host/bus reset settle delay Date: Mon, 18 Nov 2024 16:28:14 -0800 Message-ID: <20241119002815.600608-26-bvanassche@acm.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241119002815.600608-1-bvanassche@acm.org> References: <20241119002815.600608-1-bvanassche@acm.org> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Skip the reset settle delay during error handling since the scsi_debug driver doesn't need this delay. Signed-off-by: Bart Van Assche --- drivers/scsi/scsi_debug.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/scsi/scsi_debug.c b/drivers/scsi/scsi_debug.c index 2ff3a24d791e..791c085917bc 100644 --- a/drivers/scsi/scsi_debug.c +++ b/drivers/scsi/scsi_debug.c @@ -8751,6 +8751,7 @@ static struct scsi_host_template sdebug_driver_template = { .max_sectors = -1U, .max_segment_size = -1U, .module = THIS_MODULE, + .skip_settle_delay = 1, .track_queue_depth = 1, .cmd_size = sizeof(struct sdebug_scsi_cmd), .init_cmd_priv = sdebug_init_cmd_priv, From patchwork Tue Nov 19 00:28:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 13879284 Received: from 009.lax.mailroute.net (009.lax.mailroute.net [199.89.1.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 469633CF73; Tue, 19 Nov 2024 00:29:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=199.89.1.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976162; cv=none; b=NIuG2E4FkRExgjFaVc3+IzBmIUAkRwGXrZK7UlKLniiX0kHKVUDmeanJRBoXndOsXes23ZmDFALDln/OZ93P1J750IRWrhXbeZMFkfuqHf+0ec/R3ErOWIo6wCNM5oIyJr6ys3HxBx4pxFLrEgo2vCC8oooxMliavnkY1SVKG2c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731976162; c=relaxed/simple; bh=tzk6wFCiz+nISJZzyxVRfy5Zdhci77ozeEvvnQn+Hek=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PGq5GApZ9lc83Xgl9DCHJftnReM0cnzTMHN3LmYgUGU4Qiy4FgQFTU/uFNYqYo3UEJoHsc5pkYrTwMuEdQl5v++0L39QjcmctAeaVlIl617jfggwV3YPab/tpw5MaGtX6K0K6AzAmZCwi+TYkRVCczlzatCFSBa4X4pK5r/Qqzg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org; spf=pass smtp.mailfrom=acm.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b=bp9uB69n; arc=none smtp.client-ip=199.89.1.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=acm.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b="bp9uB69n" Received: from localhost (localhost [127.0.0.1]) by 009.lax.mailroute.net (Postfix) with ESMTP id 4Xsljs0KrRzlgMVN; Tue, 19 Nov 2024 00:29:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=acm.org; h= content-transfer-encoding:mime-version:references:in-reply-to :x-mailer:message-id:date:date:subject:subject:from:from :received:received; s=mr01; t=1731976155; x=1734568156; bh=CgpLe vxFIdXXnVWzkP3TJcVOLg1Ot6lzGaTdNt7Ung8=; b=bp9uB69nNYvZAYHbZoHo2 DI0g/vUc5fcfQsfqsscR/npmARt4YgVDFgTdvD38VBAwsz/JPGTQonJgBK5roLKY DBK109pnhl4uM02w47Cz0MSmaaOyBpnDfmxe5HwyI7Fff15GvGXJsgWEC6aHhL3R vkiHvt6UWumwTfR8rt+eHOPH/FjB6ZEhC/vqE+QeLtBq/DXODyXPbi6+BtagNTch C1zIBqIETxPbRWKfZWtu6dGc6B457oNTvYYhWviMePrAnSPsfxEe0mN2Fo9rhVrP VvO5d3kq6Pe79XpecbRDm0uyfZSB8C02BpKkNQBeGj0eNP88hNwhJEVL72qBqIIq A== X-Virus-Scanned: by MailRoute Received: from 009.lax.mailroute.net ([127.0.0.1]) by localhost (009.lax [127.0.0.1]) (mroute_mailscanner, port 10029) with LMTP id Fi0VJmnZPCh2; Tue, 19 Nov 2024 00:29:15 +0000 (UTC) Received: from bvanassche.mtv.corp.google.com (unknown [104.135.204.82]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: bvanassche@acm.org) by 009.lax.mailroute.net (Postfix) with ESMTPSA id 4Xsljj04sCzlgMVd; Tue, 19 Nov 2024 00:29:12 +0000 (UTC) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, Christoph Hellwig , Damien Le Moal , Jaegeuk Kim , Bart Van Assche , "Bao D. Nguyen" , Can Guo , "Martin K. Petersen" , Avri Altman Subject: [PATCH v16 26/26] scsi: ufs: Inform the block layer about write ordering Date: Mon, 18 Nov 2024 16:28:15 -0800 Message-ID: <20241119002815.600608-27-bvanassche@acm.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241119002815.600608-1-bvanassche@acm.org> References: <20241119002815.600608-1-bvanassche@acm.org> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From the UFSHCI 4.0 specification, about the legacy (single queue) mode: "The host controller always process transfer requests in-order according to the order submitted to the list. In case of multiple commands with single doorbell register ringing (batch mode), The dispatch order for these transfer requests by host controller will base on their index in the List. A transfer request with lower index value will be executed before a transfer request with higher index value." From the UFSHCI 4.0 specification, about the MCQ mode: "Command Submission 1. Host SW writes an Entry to SQ 2. Host SW updates SQ doorbell tail pointer Command Processing 3. After fetching the Entry, Host Controller updates SQ doorbell head pointer 4. Host controller sends COMMAND UPIU to UFS device" In other words, for both legacy and MCQ mode, UFS controllers are required to forward commands to the UFS device in the order these commands have been received from the host. Notes: - For legacy mode this is only correct if the host submits one command at a time. The UFS driver does this. - Also in legacy mode, the command order is not preserved if auto-hibernation is enabled in the UFS controller. This patch improves performance as follows on a test setup with UFSHCI 3.0 controller: - With the mq-deadline scheduler: 2.5x more IOPS for small writes. - When not using an I/O scheduler compared to using mq-deadline with zone locking: 4x more IOPS for small writes. Cc: Bao D. Nguyen Cc: Can Guo Cc: Martin K. Petersen Cc: Avri Altman Signed-off-by: Bart Van Assche --- drivers/ufs/core/ufshcd.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/drivers/ufs/core/ufshcd.c b/drivers/ufs/core/ufshcd.c index abbe7135a977..a6dec3b7e3fd 100644 --- a/drivers/ufs/core/ufshcd.c +++ b/drivers/ufs/core/ufshcd.c @@ -5241,6 +5241,13 @@ static int ufshcd_device_configure(struct scsi_device *sdev, struct ufs_hba *hba = shost_priv(sdev->host); struct request_queue *q = sdev->request_queue; + /* + * With auto-hibernation disabled, the write order is preserved per + * MCQ. Auto-hibernation may cause write reordering that results in + * unaligned write errors. The SCSI core will retry the failed writes. + */ + lim->driver_preserves_write_order = true; + lim->dma_pad_mask = PRDT_DATA_BYTE_COUNT_PAD - 1; /*