From patchwork Fri Jul 21 16:16:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13322263 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C15FEEB64DD for ; Fri, 21 Jul 2023 16:18:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232505AbjGUQSh (ORCPT ); Fri, 21 Jul 2023 12:18:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46192 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232677AbjGUQSD (ORCPT ); Fri, 21 Jul 2023 12:18:03 -0400 Received: from mail-io1-xd2a.google.com (mail-io1-xd2a.google.com [IPv6:2607:f8b0:4864:20::d2a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 449544203 for ; Fri, 21 Jul 2023 09:17:01 -0700 (PDT) Received: by mail-io1-xd2a.google.com with SMTP id ca18e2360f4ac-760dff4b701so30950239f.0 for ; Fri, 21 Jul 2023 09:17:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20221208.gappssmtp.com; s=20221208; t=1689956217; x=1690561017; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=A/R5UlJihGOa6+mEJqx8lHR7UM+4ST5J1wIdWJwVmsw=; b=A7DoNGrfaYn++dwRA606+rOPTEW3O5nyCKrQ3PA16H7RWwMk15a8TV/rL+a97766FS y59U3Zo/XrwDvrmNmZGUm38ljXOmxpzQBIIGPVPTiuy2YwiZzNNp+mUdP6RkWi5G5j09 BUaBuFHj6ZkGVwgIz93hFvlswj5WI+ffRJksgS/ZyQ9pKpJlQYNK4nea6guTCpJh5btx RQIdFU+3Hmm7d4gIRKA1YP/2m+y3HhHf9x3ZM/DnA0tlFATPD5FeOPHhdtS3xVC2Bikg cISPDlji3YfONNX0D1UORBioMP4a1eVUMBGkmEhiyCvoRBTafsho1xWFcIgHpjr4M3BK ETbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689956217; x=1690561017; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=A/R5UlJihGOa6+mEJqx8lHR7UM+4ST5J1wIdWJwVmsw=; b=fX14mkj/2tA7jWJQE7qnkpzNlC2qKXDtn+FxNiDUqGAuG3x2t9z3J2Q4o72SKVeg1H CKMPgExWEQD5d71jlBCWvKRKQjXmxL/c2yw7Y1C3BRjXQODjKeoeukuSF7nLh3VDAf4E MS/AxPxk6eM79u7Jw8QjqxUtEzHMbs7HNpsB/p+sTfh9Qj6Ql/U30Gzo/TIRF/eXULOF BdP8y+Sju2/+298uqp6HWBcLMP5xF2/z2qNzk2c11fWFmY34rhLwZxOgq+gvVz91RmUJ ueoOxeZGnEx3U0PSNoppnT7nyWrZrLhQ8LjNdLCMzRXd0cueCIVZMPGifM+JYWeUlx4A FjWw== X-Gm-Message-State: ABy/qLaNdTieMYxmGnmr5OZV5zpdpi8v5cqZ/4EWzg0pJ+xAogj03zNI KGxENN15VhZSddhu0DK7Y/QDLR2ZJ54BCrIYzZk= X-Google-Smtp-Source: APBJJlFmGzYXgGwW2LQkQ8zflowHIynCLKdVI0QGFj2fbBlV54fx+DvZA6RCMFC99YzFwY5rRC+Utg== X-Received: by 2002:a05:6602:3815:b0:783:63e8:3bfc with SMTP id bb21-20020a056602381500b0078363e83bfcmr2601369iob.0.1689956216834; Fri, 21 Jul 2023 09:16:56 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id l7-20020a02a887000000b0042b599b224bsm1150450jam.121.2023.07.21.09.16.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Jul 2023 09:16:55 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org, linux-xfs@vger.kernel.org Cc: hch@lst.de, andres@anarazel.de, david@fromorbit.com, djwong@kernel.org, Jens Axboe Subject: [PATCH 1/9] iomap: cleanup up iomap_dio_bio_end_io() Date: Fri, 21 Jul 2023 10:16:42 -0600 Message-Id: <20230721161650.319414-2-axboe@kernel.dk> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230721161650.319414-1-axboe@kernel.dk> References: <20230721161650.319414-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Make the logic a bit easier to follow: 1) Add a release_bio out path, as everybody needs to touch that, and have our bio ref check jump there if it's non-zero. 2) Add a kiocb local variable. 3) Add comments for each of the three conditions (sync, inline, or async workqueue punt). No functional changes in this patch. Reviewed-by: Darrick J. Wong Reviewed-by: Christoph Hellwig Signed-off-by: Jens Axboe --- fs/iomap/direct-io.c | 46 +++++++++++++++++++++++++++++--------------- 1 file changed, 31 insertions(+), 15 deletions(-) diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c index ea3b868c8355..0ce60e80c901 100644 --- a/fs/iomap/direct-io.c +++ b/fs/iomap/direct-io.c @@ -152,27 +152,43 @@ void iomap_dio_bio_end_io(struct bio *bio) { struct iomap_dio *dio = bio->bi_private; bool should_dirty = (dio->flags & IOMAP_DIO_DIRTY); + struct kiocb *iocb = dio->iocb; if (bio->bi_status) iomap_dio_set_error(dio, blk_status_to_errno(bio->bi_status)); + if (!atomic_dec_and_test(&dio->ref)) + goto release_bio; - if (atomic_dec_and_test(&dio->ref)) { - if (dio->wait_for_completion) { - struct task_struct *waiter = dio->submit.waiter; - WRITE_ONCE(dio->submit.waiter, NULL); - blk_wake_io_task(waiter); - } else if (dio->flags & IOMAP_DIO_WRITE) { - struct inode *inode = file_inode(dio->iocb->ki_filp); - - WRITE_ONCE(dio->iocb->private, NULL); - INIT_WORK(&dio->aio.work, iomap_dio_complete_work); - queue_work(inode->i_sb->s_dio_done_wq, &dio->aio.work); - } else { - WRITE_ONCE(dio->iocb->private, NULL); - iomap_dio_complete_work(&dio->aio.work); - } + /* + * Synchronous dio, task itself will handle any completion work + * that needs after IO. All we need to do is wake the task. + */ + if (dio->wait_for_completion) { + struct task_struct *waiter = dio->submit.waiter; + + WRITE_ONCE(dio->submit.waiter, NULL); + blk_wake_io_task(waiter); + goto release_bio; + } + + /* Read completion can always complete inline. */ + if (!(dio->flags & IOMAP_DIO_WRITE)) { + WRITE_ONCE(iocb->private, NULL); + iomap_dio_complete_work(&dio->aio.work); + goto release_bio; } + /* + * Async DIO completion that requires filesystem level completion work + * gets punted to a work queue to complete as the operation may require + * more IO to be issued to finalise filesystem metadata changes or + * guarantee data integrity. + */ + WRITE_ONCE(iocb->private, NULL); + INIT_WORK(&dio->aio.work, iomap_dio_complete_work); + queue_work(file_inode(iocb->ki_filp)->i_sb->s_dio_done_wq, + &dio->aio.work); +release_bio: if (should_dirty) { bio_check_pages_dirty(bio); } else { From patchwork Fri Jul 21 16:16:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13322264 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7779AC001DC for ; Fri, 21 Jul 2023 16:18:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232529AbjGUQSh (ORCPT ); Fri, 21 Jul 2023 12:18:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46210 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232683AbjGUQSE (ORCPT ); Fri, 21 Jul 2023 12:18:04 -0400 Received: from mail-io1-xd2a.google.com (mail-io1-xd2a.google.com [IPv6:2607:f8b0:4864:20::d2a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C15CA4219 for ; Fri, 21 Jul 2023 09:17:01 -0700 (PDT) Received: by mail-io1-xd2a.google.com with SMTP id ca18e2360f4ac-785d3a53ed6so25862939f.1 for ; Fri, 21 Jul 2023 09:17:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20221208.gappssmtp.com; s=20221208; t=1689956218; x=1690561018; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=l7cLLw1JpMnOMXLNHlH7JagCdHX69bYdsN+9qujHNmA=; b=vAUGNAgV1Vbq0eLmgbSiSvCxRWW2+Cw9dLYbFJT3lDY7TdJ5AZFO7HBSWJdLUoNNIA XdMVnORN2Hq+gCapIl1iD8L515U+oRRCx5H8wKu4bfBM13MOfjjNqDlTcimtvilsRWFt h6RVxA+lgtJl9gnjCNmQqTi/HI3+iOmdW5lnItmg+NDFaLFRT4EUUW/FJ4OuKFHIPZxq BrWLIAmRfd+v7rLsQnNBbiVPJWO+xH3VoDmfomiD9AdisfnGxvlMrFhDUHSIPIT5arF/ m61tR+TRcbo23cOs5c9J4YdLTUd5kw+IKQxzRmndPACj0Rvd0hc1zHVtKOKQ3e4PXYIW VOmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689956218; x=1690561018; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=l7cLLw1JpMnOMXLNHlH7JagCdHX69bYdsN+9qujHNmA=; b=i/eYPF83+NLIgLA4IOlcVnnICUK4nzh9z6dQ33PEBVvyTRSJd90BYE8DTx2GAl9W5F luEzzI2+7nLyrlExuwZgDFJriq0MsE3raSVZJ15+Vb412iPZtMF3aiUcQEFsL+QfA6pR ruB6P41TsKgW4FTn2eae0t8FGk+b9VLuXpWCkJzA+ppWP1SS/seibHRxjnsYRB/wRpzS uzh2M+Vtjd21SAw/ApduTCfeHlVPJYwfp91ZI9FmyShPQaow7uwYL+kLqNFSUXU8B66G D+/evzMvco/UKKxBMjbqaQQqGQ9/AeYtJXeGwHDdEJ0Kf6dn/KDGtxfOrfrzFTA56ihm JPlQ== X-Gm-Message-State: ABy/qLZPrVjV1JY8NVMNMQiEUJVxzm/HfekjPB0oJL0UcXImCkrvzo63 wsgth8Ye7oZ3rQlOTDbj3Yn+on1aEciLvydYpT0= X-Google-Smtp-Source: APBJJlFG0eKMp+V3bd8Ngr/HkKy7BtsI6srIw0YeAKla/vxNKEZgZUFepNJgNUqes7/7Q1N6HppK3A== X-Received: by 2002:a05:6602:14d2:b0:783:6ec1:65f6 with SMTP id b18-20020a05660214d200b007836ec165f6mr3086501iow.1.1689956218099; Fri, 21 Jul 2023 09:16:58 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id l7-20020a02a887000000b0042b599b224bsm1150450jam.121.2023.07.21.09.16.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Jul 2023 09:16:57 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org, linux-xfs@vger.kernel.org Cc: hch@lst.de, andres@anarazel.de, david@fromorbit.com, djwong@kernel.org, Jens Axboe Subject: [PATCH 2/9] iomap: add IOMAP_DIO_INLINE_COMP Date: Fri, 21 Jul 2023 10:16:43 -0600 Message-Id: <20230721161650.319414-3-axboe@kernel.dk> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230721161650.319414-1-axboe@kernel.dk> References: <20230721161650.319414-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Rather than gate whether or not we need to punt a dio completion to a workqueue on whether the IO is a write or not, add an explicit flag for it. For now we treat them the same, reads always set the flags and async writes do not. No functional changes in this patch. Reviewed-by: Darrick J. Wong Reviewed-by: Christoph Hellwig Signed-off-by: Jens Axboe --- fs/iomap/direct-io.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c index 0ce60e80c901..c654612b24e5 100644 --- a/fs/iomap/direct-io.c +++ b/fs/iomap/direct-io.c @@ -20,6 +20,7 @@ * Private flags for iomap_dio, must not overlap with the public ones in * iomap.h: */ +#define IOMAP_DIO_INLINE_COMP (1 << 27) #define IOMAP_DIO_WRITE_FUA (1 << 28) #define IOMAP_DIO_NEED_SYNC (1 << 29) #define IOMAP_DIO_WRITE (1 << 30) @@ -171,8 +172,10 @@ void iomap_dio_bio_end_io(struct bio *bio) goto release_bio; } - /* Read completion can always complete inline. */ - if (!(dio->flags & IOMAP_DIO_WRITE)) { + /* + * Flagged with IOMAP_DIO_INLINE_COMP, we can complete it inline + */ + if (dio->flags & IOMAP_DIO_INLINE_COMP) { WRITE_ONCE(iocb->private, NULL); iomap_dio_complete_work(&dio->aio.work); goto release_bio; @@ -527,6 +530,9 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, iomi.flags |= IOMAP_NOWAIT; if (iov_iter_rw(iter) == READ) { + /* reads can always complete inline */ + dio->flags |= IOMAP_DIO_INLINE_COMP; + if (iomi.pos >= dio->i_size) goto out_free_dio; From patchwork Fri Jul 21 16:16:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13322265 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 454BCEB64DC for ; Fri, 21 Jul 2023 16:18:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232566AbjGUQSi (ORCPT ); Fri, 21 Jul 2023 12:18:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45766 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232694AbjGUQSF (ORCPT ); Fri, 21 Jul 2023 12:18:05 -0400 Received: from mail-io1-xd33.google.com (mail-io1-xd33.google.com [IPv6:2607:f8b0:4864:20::d33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D882D421C for ; Fri, 21 Jul 2023 09:17:02 -0700 (PDT) Received: by mail-io1-xd33.google.com with SMTP id ca18e2360f4ac-7835bbeb6a0so28197439f.0 for ; Fri, 21 Jul 2023 09:17:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20221208.gappssmtp.com; s=20221208; t=1689956219; x=1690561019; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=z7yoihIdu5UPPNOooKn2vi0vzZ8WciIEgPTPhkQQaos=; b=x8Lt3uasN44rcnJmIUZvfnrKhQLZC36ZOXD3rCJZ6chD4Xkj6qYJjWRuYnex7ZraWp s7gKjMrw0rzE7502lXlPwomvek+ykNBFkBzcWkfcrjAiK/EKMI1nx0LotnxFsmlbAegI 8l4NmDuosjGN5VnB+pgPWQ4PHX34au0MFG25qAgMkaSoM0Azdy8mxoEuaoue66L1mhai ma8PDpwJSawKMohbp2YQEW0hf9kXBlbv6pxvuao/xAAIxJ0OBn40pUmxevKU5485HXDg ESZyV6R5cclSYgFMyl16KYVT643Kh6s4HkfAQQMAJhJtuAe25ea9BZTlfD/YNhQafTYr MBWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689956219; x=1690561019; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=z7yoihIdu5UPPNOooKn2vi0vzZ8WciIEgPTPhkQQaos=; b=b0jFUFyZULACyp5zAN8r6d3M6Zz0fNpwEsUz9wwzAf5kJiet/zjnS3AiVbibeSGHfS fb45oFL/BwTIUw3reJzutKzKkYKcrtPut2OgRNRwC8RTJCYRloleIJXFe1E+20M7xEFN t1BJhzcyzikWksFt9fMr9D6yDlPH1/DNoifAD7mDALpGzVz0KMzf9TiU+peO0tEevOvG LmpifR7DIsne+QTYLFBhpGeg7iHT0cKCGivLWI3C/wVwaN0l/mwR+ziCHd+1Af7h0wm2 ULFmRG5F4wc0iLjEOLMVd6mE3N6eIrHqZacwdZdPQtjr7KsGDMNc7Hdc2e9Z+IriV3Ir 1dnw== X-Gm-Message-State: ABy/qLbE0FNgJDv4R4pr7gGrGDnAbAe55Y/tZAsXWLz/fksAhcFo07Bv P3z7DnJD5m19L3wfE3DlI1bHEADraXtP7/j0ji0= X-Google-Smtp-Source: APBJJlEcEMTariaTZWPEkwzOzKZajcPLUhmagDsKQgxPwC2xgu7zprXfCI6DVPyp3uqZ0W+mgc1V8Q== X-Received: by 2002:a05:6602:1a05:b0:780:c6bb:ad8d with SMTP id bo5-20020a0566021a0500b00780c6bbad8dmr2382978iob.0.1689956219453; Fri, 21 Jul 2023 09:16:59 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id l7-20020a02a887000000b0042b599b224bsm1150450jam.121.2023.07.21.09.16.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Jul 2023 09:16:58 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org, linux-xfs@vger.kernel.org Cc: hch@lst.de, andres@anarazel.de, david@fromorbit.com, djwong@kernel.org, Jens Axboe Subject: [PATCH 3/9] iomap: treat a write through cache the same as FUA Date: Fri, 21 Jul 2023 10:16:44 -0600 Message-Id: <20230721161650.319414-4-axboe@kernel.dk> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230721161650.319414-1-axboe@kernel.dk> References: <20230721161650.319414-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Whether we have a write back cache and are using FUA or don't have a write back cache at all is the same situation. Treat them the same. This makes the IOMAP_DIO_WRITE_FUA name a bit misleading, as we have two cases that provide stable writes: 1) Volatile write cache with FUA writes 2) Normal write without a volatile write cache Rename that flag to IOMAP_DIO_STABLE_WRITE to make that clearer, and update some of the FUA comments as well. Reviewed-by: Christoph Hellwig Signed-off-by: Jens Axboe Reviewed-by: Darrick J. Wong --- fs/iomap/direct-io.c | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-) diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c index c654612b24e5..17b695b0e9d6 100644 --- a/fs/iomap/direct-io.c +++ b/fs/iomap/direct-io.c @@ -21,7 +21,7 @@ * iomap.h: */ #define IOMAP_DIO_INLINE_COMP (1 << 27) -#define IOMAP_DIO_WRITE_FUA (1 << 28) +#define IOMAP_DIO_WRITE_THROUGH (1 << 28) #define IOMAP_DIO_NEED_SYNC (1 << 29) #define IOMAP_DIO_WRITE (1 << 30) #define IOMAP_DIO_DIRTY (1 << 31) @@ -222,7 +222,7 @@ static void iomap_dio_zero(const struct iomap_iter *iter, struct iomap_dio *dio, /* * Figure out the bio's operation flags from the dio request, the * mapping, and whether or not we want FUA. Note that we can end up - * clearing the WRITE_FUA flag in the dio request. + * clearing the WRITE_THROUGH flag in the dio request. */ static inline blk_opf_t iomap_dio_bio_opflags(struct iomap_dio *dio, const struct iomap *iomap, bool use_fua) @@ -236,7 +236,7 @@ static inline blk_opf_t iomap_dio_bio_opflags(struct iomap_dio *dio, if (use_fua) opflags |= REQ_FUA; else - dio->flags &= ~IOMAP_DIO_WRITE_FUA; + dio->flags &= ~IOMAP_DIO_WRITE_THROUGH; return opflags; } @@ -276,11 +276,13 @@ static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter, * Use a FUA write if we need datasync semantics, this is a pure * data IO that doesn't require any metadata updates (including * after IO completion such as unwritten extent conversion) and - * the underlying device supports FUA. This allows us to avoid - * cache flushes on IO completion. + * the underlying device either supports FUA or doesn't have + * a volatile write cache. This allows us to avoid cache flushes + * on IO completion. */ if (!(iomap->flags & (IOMAP_F_SHARED|IOMAP_F_DIRTY)) && - (dio->flags & IOMAP_DIO_WRITE_FUA) && bdev_fua(iomap->bdev)) + (dio->flags & IOMAP_DIO_WRITE_THROUGH) && + (bdev_fua(iomap->bdev) || !bdev_write_cache(iomap->bdev))) use_fua = true; } @@ -560,12 +562,15 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, /* * For datasync only writes, we optimistically try - * using FUA for this IO. Any non-FUA write that - * occurs will clear this flag, hence we know before - * completion whether a cache flush is necessary. + * using WRITE_THROUGH for this IO. Stable writes are + * either FUA with a write cache, or a normal write to + * a device without a volatile write cache. For the + * former, Any non-FUA write that occurs will clear this + * flag, hence we know before completion whether a cache + * flush is necessary. */ if (!(iocb->ki_flags & IOCB_SYNC)) - dio->flags |= IOMAP_DIO_WRITE_FUA; + dio->flags |= IOMAP_DIO_WRITE_THROUGH; } /* @@ -627,10 +632,10 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, iomap_dio_set_error(dio, ret); /* - * If all the writes we issued were FUA, we don't need to flush the + * If all the writes we issued were stable, we don't need to flush the * cache on IO completion. Clear the sync flag for this case. */ - if (dio->flags & IOMAP_DIO_WRITE_FUA) + if (dio->flags & IOMAP_DIO_WRITE_THROUGH) dio->flags &= ~IOMAP_DIO_NEED_SYNC; WRITE_ONCE(iocb->private, dio->submit.poll_bio); From patchwork Fri Jul 21 16:16:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13322266 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D4933C04E69 for ; Fri, 21 Jul 2023 16:18:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232604AbjGUQSl (ORCPT ); Fri, 21 Jul 2023 12:18:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45294 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232726AbjGUQSI (ORCPT ); Fri, 21 Jul 2023 12:18:08 -0400 Received: from mail-io1-xd31.google.com (mail-io1-xd31.google.com [IPv6:2607:f8b0:4864:20::d31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 03D004227 for ; Fri, 21 Jul 2023 09:17:06 -0700 (PDT) Received: by mail-io1-xd31.google.com with SMTP id ca18e2360f4ac-78706966220so20986139f.1 for ; Fri, 21 Jul 2023 09:17:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20221208.gappssmtp.com; s=20221208; t=1689956221; x=1690561021; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9Ml0myyraRmhTLtQmcxe+g3vSgtvxsjWF8x/ZfCtD8A=; b=4+ik4qcnknrZ2pBI823ys5bfjbDA0eKf1HwuFJb5qEwuLL/GknLELm7RnzBWEq6x0j EzxyjTatcLiofyhdoxiCFXazgpyy2K6+yXOwe2ii4HCDphMgaTT5WI5dMt+Q7PJcXztT B2NLamjAWVsHxQUSTLBQvNRbyBn49MsSm0X9VmRJIEzDTlsEiitNGbkLM2NpbUVkGlku ScJwDYH6cUeGRFObbVLo6WFGbSZTGS59Wve0uDyqZ3e+I14X8OrrvKO2frrmDBIxWdHm wqBntbhc610Em4847+Dj06OkihyM1YH95RtRgk2peZ7zdwpoFORWbng2NXSmj9HKJKzv 1pnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689956221; x=1690561021; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9Ml0myyraRmhTLtQmcxe+g3vSgtvxsjWF8x/ZfCtD8A=; b=RIoTSdwKvuHt9/Cd+KSsbwR8mXz/LkjEEr334A4j4/CkzQcKRcorWTnN4rBFVHFw0t fl2BS0aK+EzDohEWqXiwYbZCf3jTsSIA1Snxyz1YHO0FzMe9ZsLzB4eklsyGDiVlksB4 YXE0cDnBm5p2HVgntEcnNn31Nln0XAShcLu7/mqeKbKchLBUO4BNS0wbTpvwTDZsLPv3 NEoOJacwCA1110Bo6QVkwqriyjxoet5GB8bfzwkSqZWDTCEZVGRvtJg2iu5wClY56A4M b7dekglPNR0XsVowVBgFMEYqMrVyhC9Sr9SQitm7wqPTPYZHwECmXoJ+JLyfGZjXjfbV hQoA== X-Gm-Message-State: ABy/qLZ3iYoN8QLQRxYWzOBAMceaObr3r5OAO5HbUJMe+0q+YTMzW075 cyZn58v2Wu0uygM3mRWHOkvsGyRXmZ2Od+xlWko= X-Google-Smtp-Source: APBJJlGQn8MOj48TbeVTtv5LQAKVB2KD+GpZ/ZTqKNKeiMYFyZOz3Du19utC4HqplV51LvT1jhuErQ== X-Received: by 2002:a6b:b4d5:0:b0:788:2d78:813c with SMTP id d204-20020a6bb4d5000000b007882d78813cmr2386295iof.0.1689956220838; Fri, 21 Jul 2023 09:17:00 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id l7-20020a02a887000000b0042b599b224bsm1150450jam.121.2023.07.21.09.16.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Jul 2023 09:16:59 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org, linux-xfs@vger.kernel.org Cc: hch@lst.de, andres@anarazel.de, david@fromorbit.com, djwong@kernel.org, Jens Axboe Subject: [PATCH 4/9] iomap: completed polled IO inline Date: Fri, 21 Jul 2023 10:16:45 -0600 Message-Id: <20230721161650.319414-5-axboe@kernel.dk> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230721161650.319414-1-axboe@kernel.dk> References: <20230721161650.319414-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Polled IO is only allowed for conditions where task completion is safe anyway, so we can always complete it inline. This cannot easily be checked with a submission side flag, as the block layer may clear the polled flag and turn it into a regular IO instead. Hence we need to check this at completion time. If REQ_POLLED is still set, then we know that this IO was successfully polled, and is completing in task context. Reviewed-by: Darrick J. Wong Reviewed-by: Christoph Hellwig Signed-off-by: Jens Axboe --- fs/iomap/direct-io.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c index 17b695b0e9d6..2b453e12dc16 100644 --- a/fs/iomap/direct-io.c +++ b/fs/iomap/direct-io.c @@ -173,9 +173,19 @@ void iomap_dio_bio_end_io(struct bio *bio) } /* - * Flagged with IOMAP_DIO_INLINE_COMP, we can complete it inline + * Flagged with IOMAP_DIO_INLINE_COMP, we can complete it inline. + * Ditto for polled requests - if the flag is still at completion + * time, then we know the request was actually polled and completion + * is called from the task itself. This is why we need to check it + * here rather than flag it at issue time. */ - if (dio->flags & IOMAP_DIO_INLINE_COMP) { + if ((dio->flags & IOMAP_DIO_INLINE_COMP) || (bio->bi_opf & REQ_POLLED)) { + /* + * For polled IO, we need to clear ->private as it points to + * the bio being polled for. The completion side uses it to + * know if a given request has been found yet or not. For + * non-polled IO, ->private isn't applicable. + */ WRITE_ONCE(iocb->private, NULL); iomap_dio_complete_work(&dio->aio.work); goto release_bio; From patchwork Fri Jul 21 16:16:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13322267 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC707C04FDF for ; Fri, 21 Jul 2023 16:18:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232611AbjGUQSm (ORCPT ); Fri, 21 Jul 2023 12:18:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45908 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232739AbjGUQSJ (ORCPT ); Fri, 21 Jul 2023 12:18:09 -0400 Received: from mail-io1-xd33.google.com (mail-io1-xd33.google.com [IPv6:2607:f8b0:4864:20::d33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8CAAF4224 for ; Fri, 21 Jul 2023 09:17:06 -0700 (PDT) Received: by mail-io1-xd33.google.com with SMTP id ca18e2360f4ac-78706966220so20986539f.1 for ; Fri, 21 Jul 2023 09:17:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20221208.gappssmtp.com; s=20221208; t=1689956222; x=1690561022; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Oy9lAbgIR3CkabfYbdlhMEs1+yC10+0lr3pYvZ/QmXg=; b=yA33Zd9ldVZY9NhyqqsuksHoALn2lUyr0D0f/gixUZQWH8C4+c1CVBC4l8R43ZFLP2 N5gTPBqbYIOmh5fV+DUWjJo7f+HfJN8yZOboqOMTi4xZRiR0X1NqYFlRM3rDImIkb2b4 kRLj3GWIe/RSi9MZ1LCewwZsIg7T3DVHX+DbHVpMPrUb7LxxsiAGoH0S0d7OWiYn7Quf EyjnR+KLkNRR9cW7N8FaN3ifEglDPAept1S+foWG7lqtQgj1c5I8jhfq5aICdN/EQP9l LTyHQbZ2dmJC58eS4d2WoIbmXg8C/pqPZWC1PGOlylsmG9cZJrcM/PovfSmPdSH70JTl Lm9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689956222; x=1690561022; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Oy9lAbgIR3CkabfYbdlhMEs1+yC10+0lr3pYvZ/QmXg=; b=TrSNC75tdngMYyfJG68bhCPhWAVxwU2nbRkuck8JQwnUqR013KCpU2AXLzHEc9iXBm Ie6lTB1ADRO5OOeFxKyL9DAjnVXIG8tEFhajcUMXdgvCAp9LIQz6bfEteGpxG8J/txk5 JbWNMoza5qO4xt5RkEa4Tvrpa7m2NsWYBEHWy7u8R9v3y0ll0UPUMa6AeUftn/+MBOvA yuOme/re+6mNGE1bxv9uj90p6wkfdjD/DLj2L75QF5Yv1o4/Faw2nRRM0yTA2FWLQoIE Dvo7qck/JuYmlEVOl523VOmYX/sxbugg2ZsYT1zJbtpDGFVaxfVZ3cU3ub176xx/uxBj XVfQ== X-Gm-Message-State: ABy/qLbH9veTr+Swm+xU81Gg5AepJZReUOmV3zXTw8Ja7jukjFF87+4x h7+hASRJjWuwyi8Je75RIl+qge5QlLK7DSjVR/Q= X-Google-Smtp-Source: APBJJlFEmx3jVQR75X1RQwVd/oXQyZs3MLao8dRJtepXvYv9avdnDUd6eEC7lVQbGsW6gWOVuwrHfA== X-Received: by 2002:a05:6602:4809:b0:77a:ee79:652 with SMTP id ed9-20020a056602480900b0077aee790652mr2360499iob.1.1689956222242; Fri, 21 Jul 2023 09:17:02 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id l7-20020a02a887000000b0042b599b224bsm1150450jam.121.2023.07.21.09.17.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Jul 2023 09:17:01 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org, linux-xfs@vger.kernel.org Cc: hch@lst.de, andres@anarazel.de, david@fromorbit.com, djwong@kernel.org, Jens Axboe Subject: [PATCH 5/9] iomap: only set iocb->private for polled bio Date: Fri, 21 Jul 2023 10:16:46 -0600 Message-Id: <20230721161650.319414-6-axboe@kernel.dk> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230721161650.319414-1-axboe@kernel.dk> References: <20230721161650.319414-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org iocb->private is only used for polled IO, where the completer will find the bio to poll through that field. Assign it when we're submitting a polled bio, and get rid of the dio->poll_bio indirection. Reviewed-by: Darrick J. Wong Reviewed-by: Christoph Hellwig Signed-off-by: Jens Axboe --- fs/iomap/direct-io.c | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c index 2b453e12dc16..6ffa1b1ebe90 100644 --- a/fs/iomap/direct-io.c +++ b/fs/iomap/direct-io.c @@ -42,7 +42,6 @@ struct iomap_dio { struct { struct iov_iter *iter; struct task_struct *waiter; - struct bio *poll_bio; } submit; /* used for aio completion: */ @@ -64,12 +63,14 @@ static struct bio *iomap_dio_alloc_bio(const struct iomap_iter *iter, static void iomap_dio_submit_bio(const struct iomap_iter *iter, struct iomap_dio *dio, struct bio *bio, loff_t pos) { + struct kiocb *iocb = dio->iocb; + atomic_inc(&dio->ref); /* Sync dio can't be polled reliably */ - if ((dio->iocb->ki_flags & IOCB_HIPRI) && !is_sync_kiocb(dio->iocb)) { - bio_set_polled(bio, dio->iocb); - dio->submit.poll_bio = bio; + if ((iocb->ki_flags & IOCB_HIPRI) && !is_sync_kiocb(iocb)) { + bio_set_polled(bio, iocb); + WRITE_ONCE(iocb->private, bio); } if (dio->dops && dio->dops->submit_io) @@ -197,7 +198,6 @@ void iomap_dio_bio_end_io(struct bio *bio) * more IO to be issued to finalise filesystem metadata changes or * guarantee data integrity. */ - WRITE_ONCE(iocb->private, NULL); INIT_WORK(&dio->aio.work, iomap_dio_complete_work); queue_work(file_inode(iocb->ki_filp)->i_sb->s_dio_done_wq, &dio->aio.work); @@ -536,7 +536,6 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, dio->submit.iter = iter; dio->submit.waiter = current; - dio->submit.poll_bio = NULL; if (iocb->ki_flags & IOCB_NOWAIT) iomi.flags |= IOMAP_NOWAIT; @@ -648,8 +647,6 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, if (dio->flags & IOMAP_DIO_WRITE_THROUGH) dio->flags &= ~IOMAP_DIO_NEED_SYNC; - WRITE_ONCE(iocb->private, dio->submit.poll_bio); - /* * We are about to drop our additional submission reference, which * might be the last reference to the dio. There are three different From patchwork Fri Jul 21 16:16:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13322268 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B420C04FE0 for ; Fri, 21 Jul 2023 16:18:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232617AbjGUQSn (ORCPT ); Fri, 21 Jul 2023 12:18:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45390 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229736AbjGUQSM (ORCPT ); Fri, 21 Jul 2023 12:18:12 -0400 Received: from mail-io1-xd2b.google.com (mail-io1-xd2b.google.com [IPv6:2607:f8b0:4864:20::d2b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4F86F3C06 for ; Fri, 21 Jul 2023 09:17:07 -0700 (PDT) Received: by mail-io1-xd2b.google.com with SMTP id ca18e2360f4ac-7748ca56133so21204039f.0 for ; Fri, 21 Jul 2023 09:17:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20221208.gappssmtp.com; s=20221208; t=1689956223; x=1690561023; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yyNahqPc1pwMUX1EtsLZ1EOWwyO2cqWRsqmVBpYW9+Y=; b=0aWP+E4GFdNzNflre1kamqe9PsnlmI/2nmygaFNWCPd+YY0Hm0To8JVrGDpwOmY/Jy MswOWnIV9AXXasYUcaiYtWjMfW0mmViPwZu1nVQExuJg+jQMUXeguYDwnd4YMhImYw5Z KtSmd0IOrMhZ/oxbtJ6r7m4EY3Zyo7laIfbJkmDmKE4HhK7QNOHpz16ToyOc/ua5FI60 Q8PqqBUe1vFN3xwmGe/y0ZAOlhi8v8/RduhDD+CAibE231b7YNtFC83RzhHAXpa1dLwr XplSxoAjDSaoG8jpM7HN892Jvl2oSKfkBMw2WFNW3vLcaDGNMzf+oqwT1Dic1qUBZJyb wbWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689956223; x=1690561023; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yyNahqPc1pwMUX1EtsLZ1EOWwyO2cqWRsqmVBpYW9+Y=; b=hRhr0Xko8T1atvzssRRziNOGqeI0teeTI6NrwC7S6MwyDkaWT3RC8Ff4eHgQgXCp7w OipHutqki2cHRQUvKvISmseUTU5fUUunvmlTQQkkWkP+uA5x6wW2bfORtD3EZlxefh98 uzalalzklA6wJQ1yJTERQI1Wzj+Z/8g2gaFaaeBYZZrJojMOHJR2bheSGHBXaq/iMK2e xcKrY6ykzm6T0djtryjPIy4poKQKFODGE+QwJFu54gN4O3sNMWdQqJxX32lANB1Cm+Cp 48XOfgE1weuneM5D1ui2qGvuh81MWfBYdGnwi/VrUPdtNIDsR7J/dX+kChpDnSGEJF8H zUYg== X-Gm-Message-State: ABy/qLb3FhnwaxNjCgOahkg5Hfc57GwOm7BvPhK3DR41J5c8Nl/ajccp 42oUqQ8PEAAKVZKRiB9/IgkIGvtFhIBetAWozfs= X-Google-Smtp-Source: APBJJlFal8EmDiwtuR8AOwBCrps/dQIMAByn1uNcUwe3Vb9BP9vhPAAXlDgWaXYBHjITsO62APenbA== X-Received: by 2002:a05:6602:4809:b0:77a:ee79:652 with SMTP id ed9-20020a056602480900b0077aee790652mr2360554iob.1.1689956223422; Fri, 21 Jul 2023 09:17:03 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id l7-20020a02a887000000b0042b599b224bsm1150450jam.121.2023.07.21.09.17.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Jul 2023 09:17:02 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org, linux-xfs@vger.kernel.org Cc: hch@lst.de, andres@anarazel.de, david@fromorbit.com, djwong@kernel.org, Jens Axboe Subject: [PATCH 6/9] fs: add IOCB flags related to passing back dio completions Date: Fri, 21 Jul 2023 10:16:47 -0600 Message-Id: <20230721161650.319414-7-axboe@kernel.dk> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230721161650.319414-1-axboe@kernel.dk> References: <20230721161650.319414-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Async dio completions generally happen from hard/soft IRQ context, which means that users like iomap may need to defer some of the completion handling to a workqueue. This is less efficient than having the original issuer handle it, like we do for sync IO, and it adds latency to the completions. Add IOCB_DIO_CALLER_COMP, which the issuer can set if it is able to safely punt these completions to a safe context. If the dio handler is aware of this flag, assign a callback handler in kiocb->dio_complete and associated data io kiocb->private. The issuer will then call this handler with that data from task context. No functional changes in this patch. Reviewed-by: Christoph Hellwig Signed-off-by: Jens Axboe Reviewed-by: Darrick J. Wong --- include/linux/fs.h | 35 +++++++++++++++++++++++++++++++++-- 1 file changed, 33 insertions(+), 2 deletions(-) diff --git a/include/linux/fs.h b/include/linux/fs.h index 6867512907d6..60e2b4ecfc4d 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -338,6 +338,20 @@ enum rw_hint { #define IOCB_NOIO (1 << 20) /* can use bio alloc cache */ #define IOCB_ALLOC_CACHE (1 << 21) +/* + * IOCB_DIO_CALLER_COMP can be set by the iocb owner, to indicate that the + * iocb completion can be passed back to the owner for execution from a safe + * context rather than needing to be punted through a workqueue.If this If this + * flag is set, the bio completion handling may set iocb->dio_complete to a + * handler function and iocb->private to context information for that handler. + * The issuer should call the handler with that context information from task + * context to complete the processing of the iocb. Note that while this + * provides a task context for the dio_complete() callback, it should only be + * used on the completion side for non-IO generating completions. It's fine to + * call blocking functions from this callback, but they should not wait for + * unrelated IO (like cache flushing, new IO generation, etc). + */ +#define IOCB_DIO_CALLER_COMP (1 << 22) /* for use in trace events */ #define TRACE_IOCB_STRINGS \ @@ -351,7 +365,8 @@ enum rw_hint { { IOCB_WRITE, "WRITE" }, \ { IOCB_WAITQ, "WAITQ" }, \ { IOCB_NOIO, "NOIO" }, \ - { IOCB_ALLOC_CACHE, "ALLOC_CACHE" } + { IOCB_ALLOC_CACHE, "ALLOC_CACHE" }, \ + { IOCB_DIO_CALLER_COMP, "CALLER_COMP" } struct kiocb { struct file *ki_filp; @@ -360,7 +375,23 @@ struct kiocb { void *private; int ki_flags; u16 ki_ioprio; /* See linux/ioprio.h */ - struct wait_page_queue *ki_waitq; /* for async buffered IO */ + union { + /* + * Only used for async buffered reads, where it denotes the + * page waitqueue associated with completing the read. Valid + * IFF IOCB_WAITQ is set. + */ + struct wait_page_queue *ki_waitq; + /* + * Can be used for O_DIRECT IO, where the completion handling + * is punted back to the issuer of the IO. May only be set + * if IOCB_DIO_CALLER_COMP is set by the issuer, and the issuer + * must then check for presence of this handler when ki_complete + * is invoked. The data passed in to this handler must be + * assigned to ->private when dio_complete is assigned. + */ + ssize_t (*dio_complete)(void *data); + }; }; static inline bool is_sync_kiocb(struct kiocb *kiocb) From patchwork Fri Jul 21 16:16:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13322269 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6DD90C04A6A for ; Fri, 21 Jul 2023 16:18:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230413AbjGUQSp (ORCPT ); Fri, 21 Jul 2023 12:18:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46014 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230506AbjGUQSX (ORCPT ); Fri, 21 Jul 2023 12:18:23 -0400 Received: from mail-io1-xd36.google.com (mail-io1-xd36.google.com [IPv6:2607:f8b0:4864:20::d36]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 07BD0423A for ; Fri, 21 Jul 2023 09:17:10 -0700 (PDT) Received: by mail-io1-xd36.google.com with SMTP id ca18e2360f4ac-780c89d1998so28218139f.1 for ; Fri, 21 Jul 2023 09:17:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20221208.gappssmtp.com; s=20221208; t=1689956224; x=1690561024; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0K7Dvo7paWgjEVh49lf7ANu+PDRkYsLiCLFfzkR8qLs=; b=3xPTlZlQRZBxIsm15+JS0gtC8owGRTHiYuFA5FVf+3R59m8jSoVd+Aad9Gju5NSOME DFR4N+ny5AJPtBa4DIY+4OMRuFVywbPq07tOgAs7L2SrzWEkXL2MdHebpAEE9lXVp38I 8Oh3IHvSrtYXz/nLdFZ/ckMt+ASJ1b2JrMo1TvsQR4wKe/CxsXwgj0fTOTfHzaHWnvLG W5JoREMNpl6EF3alGMzryTEFVEMNWAs97JFUz/Ws0EsiEmSSOmxHB9JjiGKx3obwltwh U425f+oqLCmcnPPMP6oglw35A54wo7zoHa8BciLXc0BVvqbQQO2v3VvN77WNHuOfSDDo YHqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689956224; x=1690561024; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0K7Dvo7paWgjEVh49lf7ANu+PDRkYsLiCLFfzkR8qLs=; b=LSdtrmBoHEJaA7SXe/0F6Iq15M8P/8Gpa8zWS9STjuXB2vg0f6ASb2S7O6D4xwMUNA ElWzfO8NQijE+ARSO36pmbifo2+TUVr1+uW5B3MYw0J6BC2puUEcFf9MrCLlQkLUidOf t9fsiLBQ9BMyVs5uDuyX8VobtP/VkHgqSwZq26+BhBUx4cq8okjSnRgewhguCDyCEgxT 9W0p3dDgXcAtNSBCMg1yZQByqqbr+Cgc1LyCgASE1DmSID3xQ3R4pxVYiVN2r2yLT713 V2EaJWALE0VngVpker9XkT7+razvtGd7VRyb5Tzq1KOQDDKjzCAEBWDlOVno9sbeNPOW P02w== X-Gm-Message-State: ABy/qLbeW3IGKNMbAZP2T0cst1Dc1ArGBL6RCjDfpOBpO1baj7s5SWfY Otr5ZksaNjYERJ+c0Tz+IvB5/NRzjQho6TYOo44= X-Google-Smtp-Source: APBJJlGyujB7gg6Xoypqc8Ia7cP1l3+HuYf0PntZvVUWZ8Lc0yUkwdvkmlxtPWqSHyNJb9MxPcbysg== X-Received: by 2002:a05:6602:480b:b0:780:d6ef:160 with SMTP id ed11-20020a056602480b00b00780d6ef0160mr2532052iob.1.1689956224684; Fri, 21 Jul 2023 09:17:04 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id l7-20020a02a887000000b0042b599b224bsm1150450jam.121.2023.07.21.09.17.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Jul 2023 09:17:03 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org, linux-xfs@vger.kernel.org Cc: hch@lst.de, andres@anarazel.de, david@fromorbit.com, djwong@kernel.org, Jens Axboe Subject: [PATCH 7/9] io_uring/rw: add write support for IOCB_DIO_CALLER_COMP Date: Fri, 21 Jul 2023 10:16:48 -0600 Message-Id: <20230721161650.319414-8-axboe@kernel.dk> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230721161650.319414-1-axboe@kernel.dk> References: <20230721161650.319414-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org If the filesystem dio handler understands IOCB_DIO_CALLER_COMP, we'll get a kiocb->ki_complete() callback with kiocb->dio_complete set. In that case, rather than complete the IO directly through task_work, queue up an intermediate task_work handler that first processes this callback and then immediately completes the request. For XFS, this avoids a punt through a workqueue, which is a lot less efficient and adds latency to lower queue depth (or sync) O_DIRECT writes. Only do this for non-polled IO, as polled IO doesn't need this kind of deferral as it always completes within the task itself. This then avoids a check for deferral in the polled IO completion handler. Reviewed-by: Christoph Hellwig Signed-off-by: Jens Axboe Reviewed-by: Darrick J. Wong --- io_uring/rw.c | 26 +++++++++++++++++++++++--- 1 file changed, 23 insertions(+), 3 deletions(-) diff --git a/io_uring/rw.c b/io_uring/rw.c index 1bce2208b65c..f19f65b3f0ee 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -105,6 +105,7 @@ int io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe) } else { rw->kiocb.ki_ioprio = get_current_ioprio(); } + rw->kiocb.dio_complete = NULL; rw->addr = READ_ONCE(sqe->addr); rw->len = READ_ONCE(sqe->len); @@ -285,6 +286,14 @@ static inline int io_fixup_rw_res(struct io_kiocb *req, long res) void io_req_rw_complete(struct io_kiocb *req, struct io_tw_state *ts) { + struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw); + + if (rw->kiocb.dio_complete) { + long res = rw->kiocb.dio_complete(rw->kiocb.private); + + io_req_set_res(req, io_fixup_rw_res(req, res), 0); + } + io_req_io_end(req); if (req->flags & (REQ_F_BUFFER_SELECTED|REQ_F_BUFFER_RING)) { @@ -300,9 +309,11 @@ static void io_complete_rw(struct kiocb *kiocb, long res) struct io_rw *rw = container_of(kiocb, struct io_rw, kiocb); struct io_kiocb *req = cmd_to_io_kiocb(rw); - if (__io_complete_rw_common(req, res)) - return; - io_req_set_res(req, io_fixup_rw_res(req, res), 0); + if (!rw->kiocb.dio_complete) { + if (__io_complete_rw_common(req, res)) + return; + io_req_set_res(req, io_fixup_rw_res(req, res), 0); + } req->io_task_work.func = io_req_rw_complete; __io_req_task_work_add(req, IOU_F_TWQ_LAZY_WAKE); } @@ -916,6 +927,15 @@ int io_write(struct io_kiocb *req, unsigned int issue_flags) } kiocb->ki_flags |= IOCB_WRITE; + /* + * For non-polled IO, set IOCB_DIO_CALLER_COMP, stating that our handler + * groks deferring the completion to task context. This isn't + * necessary and useful for polled IO as that can always complete + * directly. + */ + if (!(kiocb->ki_flags & IOCB_HIPRI)) + kiocb->ki_flags |= IOCB_DIO_CALLER_COMP; + if (likely(req->file->f_op->write_iter)) ret2 = call_write_iter(req->file, kiocb, &s->iter); else if (req->file->f_op->write) From patchwork Fri Jul 21 16:16:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13322270 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 73125C001DF for ; Fri, 21 Jul 2023 16:18:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232625AbjGUQSr (ORCPT ); Fri, 21 Jul 2023 12:18:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45574 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231351AbjGUQS0 (ORCPT ); Fri, 21 Jul 2023 12:18:26 -0400 Received: from mail-io1-xd36.google.com (mail-io1-xd36.google.com [IPv6:2607:f8b0:4864:20::d36]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 53D193583 for ; Fri, 21 Jul 2023 09:17:11 -0700 (PDT) Received: by mail-io1-xd36.google.com with SMTP id ca18e2360f4ac-785ccd731a7so26078639f.0 for ; Fri, 21 Jul 2023 09:17:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20221208.gappssmtp.com; s=20221208; t=1689956226; x=1690561026; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wU0I1Crk0A9YQHr2dSuYw08fghtgkYJZMMemssUA97A=; b=Wt5B/jrczgGplo7izRUX1x6J0TXmv3X4LjWC0vhre8KoqZSeJ0R+n4O5/8E5Yn5RzU U8GcrQgZvg2nAnpoHBIhrA7J463hFjfY0zSft1/0m63AT+ULNz/2ipmMVeZYWRFWDIZT PR7sqWdxomduhToaloAmi41xmfvbS1b3OKK+CEkX5o+NqmzI82LS6FVfipirWK2GnN6V vkzuLZ4qkZQgtkCgsgbdmYbCUbgFi0xcZvVRoqADIjKeQlJHU1krPh18DhxVeOMXoSlE 2trh8lLHbuGH+lmA2ikTAiWdqguI0D9WGE4IHMbeZ47FQiuf/kjQfgKAN/sTifObJblc q4hA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689956226; x=1690561026; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wU0I1Crk0A9YQHr2dSuYw08fghtgkYJZMMemssUA97A=; b=dY8OZ8UoIUp20h4njspUzmT07F6bKw3e9TK+2DRvF0MbilIkdpNiLQhZad8E5k/YJ5 p9QgSIrSDr/E3PyeHNBhZoaVhwz1btLKqTFrY8i6Wauu4Lguh+afy5M/tm+Zyf/bFHdG X/9Vowb071rQoP2HRtrEIoBF7CWWcTbNNuAz+XLWn5qhTNqFHRVdIpGr4oHk8OUaUE/O TqCoaeD0nE0mtdOqM01hPo88JQHroj9vbVBQBmUi14QIKc6ph6yqHBsjUypVmyW+1/1S 2yO09IHS9EdOqDhKiKwE18FwRb20Nykqt9oKe3PSYDjrfomYtH2GHsscaiyLsd45gO8g jFpw== X-Gm-Message-State: ABy/qLaXWk3p2mA+iOGgyUxEj93u9i64b+0twocCUKZKZcLXk0oyYg2B H0uCbTp22TGNm+4V2bKkSi5snLGv6CHUK3WhKzs= X-Google-Smtp-Source: APBJJlFVkVxscNIXOSMsYXKUWkRhB6n4mYkuHnJWS77o7eUQJyD/zAvZuu23HuKW4mNHvm0bDEzxVg== X-Received: by 2002:a05:6602:14d2:b0:783:6ec1:65f6 with SMTP id b18-20020a05660214d200b007836ec165f6mr3087186iow.1.1689956225757; Fri, 21 Jul 2023 09:17:05 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id l7-20020a02a887000000b0042b599b224bsm1150450jam.121.2023.07.21.09.17.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Jul 2023 09:17:05 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org, linux-xfs@vger.kernel.org Cc: hch@lst.de, andres@anarazel.de, david@fromorbit.com, djwong@kernel.org, Jens Axboe Subject: [PATCH 8/9] iomap: support IOCB_DIO_CALLER_COMP Date: Fri, 21 Jul 2023 10:16:49 -0600 Message-Id: <20230721161650.319414-9-axboe@kernel.dk> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230721161650.319414-1-axboe@kernel.dk> References: <20230721161650.319414-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org If IOCB_DIO_CALLER_COMP is set, utilize that to set kiocb->dio_complete handler and data for that callback. Rather than punt the completion to a workqueue, we pass back the handler and data to the issuer and will get a callback from a safe task context. Using the following fio job to randomly dio write 4k blocks at queue depths of 1..16: fio --name=dio-write --filename=/data1/file --time_based=1 \ --runtime=10 --bs=4096 --rw=randwrite --norandommap --buffered=0 \ --cpus_allowed=4 --ioengine=io_uring --iodepth=$depth shows the following results before and after this patch: Stock Patched Diff ======================================= QD1 155K 162K + 4.5% QD2 290K 313K + 7.9% QD4 533K 597K +12.0% QD8 604K 827K +36.9% QD16 615K 845K +37.4% which shows nice wins all around. If we factored in per-IOP efficiency, the wins look even nicer. This becomes apparent as queue depth rises, as the offloaded workqueue completions runs out of steam. Reviewed-by: Darrick J. Wong Reviewed-by: Christoph Hellwig Signed-off-by: Jens Axboe --- fs/iomap/direct-io.c | 55 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 54 insertions(+), 1 deletion(-) diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c index 6ffa1b1ebe90..ae9046d16d71 100644 --- a/fs/iomap/direct-io.c +++ b/fs/iomap/direct-io.c @@ -20,6 +20,7 @@ * Private flags for iomap_dio, must not overlap with the public ones in * iomap.h: */ +#define IOMAP_DIO_CALLER_COMP (1 << 26) #define IOMAP_DIO_INLINE_COMP (1 << 27) #define IOMAP_DIO_WRITE_THROUGH (1 << 28) #define IOMAP_DIO_NEED_SYNC (1 << 29) @@ -132,6 +133,11 @@ ssize_t iomap_dio_complete(struct iomap_dio *dio) } EXPORT_SYMBOL_GPL(iomap_dio_complete); +static ssize_t iomap_dio_deferred_complete(void *data) +{ + return iomap_dio_complete(data); +} + static void iomap_dio_complete_work(struct work_struct *work) { struct iomap_dio *dio = container_of(work, struct iomap_dio, aio.work); @@ -192,6 +198,31 @@ void iomap_dio_bio_end_io(struct bio *bio) goto release_bio; } + /* + * If this dio is flagged with IOMAP_DIO_CALLER_COMP, then schedule + * our completion that way to avoid an async punt to a workqueue. + */ + if (dio->flags & IOMAP_DIO_CALLER_COMP) { + /* only polled IO cares about private cleared */ + iocb->private = dio; + iocb->dio_complete = iomap_dio_deferred_complete; + + /* + * Invoke ->ki_complete() directly. We've assigned our + * dio_complete callback handler, and since the issuer set + * IOCB_DIO_CALLER_COMP, we know their ki_complete handler will + * notice ->dio_complete being set and will defer calling that + * handler until it can be done from a safe task context. + * + * Note that the 'res' being passed in here is not important + * for this case. The actual completion value of the request + * will be gotten from dio_complete when that is run by the + * issuer. + */ + iocb->ki_complete(iocb, 0); + goto release_bio; + } + /* * Async DIO completion that requires filesystem level completion work * gets punted to a work queue to complete as the operation may require @@ -288,12 +319,17 @@ static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter, * after IO completion such as unwritten extent conversion) and * the underlying device either supports FUA or doesn't have * a volatile write cache. This allows us to avoid cache flushes - * on IO completion. + * on IO completion. If we can't use writethrough and need to + * sync, disable in-task completions as dio completion will + * need to call generic_write_sync() which will do a blocking + * fsync / cache flush call. */ if (!(iomap->flags & (IOMAP_F_SHARED|IOMAP_F_DIRTY)) && (dio->flags & IOMAP_DIO_WRITE_THROUGH) && (bdev_fua(iomap->bdev) || !bdev_write_cache(iomap->bdev))) use_fua = true; + else if (dio->flags & IOMAP_DIO_NEED_SYNC) + dio->flags &= ~IOMAP_DIO_CALLER_COMP; } /* @@ -319,6 +355,14 @@ static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter, pad = pos & (fs_block_size - 1); if (pad) iomap_dio_zero(iter, dio, pos - pad, pad); + + /* + * If need_zeroout is set, then this is a new or unwritten + * extent, or dirty file metadata have not been persisted to + * disk. These need extra handling at completion time, so + * disable in-task deferred completion for those. + */ + dio->flags &= ~IOMAP_DIO_CALLER_COMP; } /* @@ -557,6 +601,15 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, iomi.flags |= IOMAP_WRITE; dio->flags |= IOMAP_DIO_WRITE; + /* + * Flag as supporting deferred completions, if the issuer + * groks it. This can avoid a workqueue punt for writes. + * We may later clear this flag if we need to do other IO + * as part of this IO completion. + */ + if (iocb->ki_flags & IOCB_DIO_CALLER_COMP) + dio->flags |= IOMAP_DIO_CALLER_COMP; + if (dio_flags & IOMAP_DIO_OVERWRITE_ONLY) { ret = -EAGAIN; if (iomi.pos >= dio->i_size || From patchwork Fri Jul 21 16:16:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13322271 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 20E4BC04A6A for ; Fri, 21 Jul 2023 16:18:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231552AbjGUQSt (ORCPT ); Fri, 21 Jul 2023 12:18:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46112 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231478AbjGUQS1 (ORCPT ); Fri, 21 Jul 2023 12:18:27 -0400 Received: from mail-io1-xd2b.google.com (mail-io1-xd2b.google.com [IPv6:2607:f8b0:4864:20::d2b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 92A434490 for ; Fri, 21 Jul 2023 09:17:12 -0700 (PDT) Received: by mail-io1-xd2b.google.com with SMTP id ca18e2360f4ac-780c89d1998so28218739f.1 for ; Fri, 21 Jul 2023 09:17:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20221208.gappssmtp.com; s=20221208; t=1689956227; x=1690561027; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/0ou5k87I8sTPpRqra9CBhOtTJt5c6QaC2AAQjrnZvU=; b=BRMTxmI+PY4omQ235bdiMvhNB9Ce0C4YAQO7D88bN1+VRHjEB099g87e/bvfP/1YhR 5hpT1PMNCuZ3F/8gb/DiAwM8QPNGFzTCjcjYwZwF/tpdZAEDSfg+e3YQ6yvlwwHtADKC NmfjXaRAmejWyQAiMtaKd23p/fO3w/y1DeeBzQUJY1xSEKaGu8dQdse9qxvi2jKG8FeL BTxkGLhrs9jz9T/OlYUpOIBN4UWYc7azcEoyEKe+JAwQIqChR0GcB7gRne2A6ggKOKwO /16n9VzKV1pfwLrpyfswYblpDYe6DS8h95HQwZWrgFxzMDKjl5R1nHa5xIVEpjuwY3g4 Y7tw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689956227; x=1690561027; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/0ou5k87I8sTPpRqra9CBhOtTJt5c6QaC2AAQjrnZvU=; b=a+86x6E0/B4VhT5t67hrZBXMCbdApcAVWfotMZA9x9vk5/+zeEG3nSIt3ACsFvzEIw qmAlTK9QTLjTZhZZwEsDvoTTidq+zISJ9hNYiCdFfDIpYr5w1L2XLHT2e4+iy7HrqMDJ mtQ/C+e5paiVG+oIc+21CAFcMOXXKU4ZB2ynLEPiM6msDtrOZheDNA0MEgRMcUBCz/Ir YLqcOVQvb4qgyjcdI162JJya2uk3fet0PTZGCOYyJGiqZ4k4WK+3A9X4nKIPPusmRlGb p9hOfQ473BrYMCeSch1j+GQYyNXfOHWrqAd1XfwI/ngzlKMi1HjYDZdyT8H1JzBVYERu rorw== X-Gm-Message-State: ABy/qLbM/TMKohV67odSM9kQueayrNokDkyINgLYsRMAJDLp2DEYVEjQ IHL1+hQqZqofB6U2sbNp45nrdw9BWQSEImOz4zg= X-Google-Smtp-Source: APBJJlGSsWXWU4FHIUZtiYJSZA11IzqG/adw0m0igijzMWLTfMvgHAO4+Tzm08vKqUEF0hI3XCz9YA== X-Received: by 2002:a05:6602:1a05:b0:780:c6bb:ad8d with SMTP id bo5-20020a0566021a0500b00780c6bbad8dmr2383411iob.0.1689956227115; Fri, 21 Jul 2023 09:17:07 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id l7-20020a02a887000000b0042b599b224bsm1150450jam.121.2023.07.21.09.17.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Jul 2023 09:17:06 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org, linux-xfs@vger.kernel.org Cc: hch@lst.de, andres@anarazel.de, david@fromorbit.com, djwong@kernel.org, Jens Axboe Subject: [PATCH 9/9] iomap: use an unsigned type for IOMAP_DIO_* defines Date: Fri, 21 Jul 2023 10:16:50 -0600 Message-Id: <20230721161650.319414-10-axboe@kernel.dk> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230721161650.319414-1-axboe@kernel.dk> References: <20230721161650.319414-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org IOMAP_DIO_DIRTY shifts by 31 bits, which makes UBSAN unhappy. Clean up all the defines by making the shifted value an unsigned value. Reported-by: Darrick J. Wong Signed-off-by: Jens Axboe Reviewed-by: Darrick J. Wong Reviewed-by: Christoph Hellwig --- fs/iomap/direct-io.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c index ae9046d16d71..dc9fe2ac9136 100644 --- a/fs/iomap/direct-io.c +++ b/fs/iomap/direct-io.c @@ -20,12 +20,12 @@ * Private flags for iomap_dio, must not overlap with the public ones in * iomap.h: */ -#define IOMAP_DIO_CALLER_COMP (1 << 26) -#define IOMAP_DIO_INLINE_COMP (1 << 27) -#define IOMAP_DIO_WRITE_THROUGH (1 << 28) -#define IOMAP_DIO_NEED_SYNC (1 << 29) -#define IOMAP_DIO_WRITE (1 << 30) -#define IOMAP_DIO_DIRTY (1 << 31) +#define IOMAP_DIO_CALLER_COMP (1U << 26) +#define IOMAP_DIO_INLINE_COMP (1U << 27) +#define IOMAP_DIO_WRITE_THROUGH (1U << 28) +#define IOMAP_DIO_NEED_SYNC (1U << 29) +#define IOMAP_DIO_WRITE (1U << 30) +#define IOMAP_DIO_DIRTY (1U << 31) struct iomap_dio { struct kiocb *iocb;