From patchwork Wed Apr 3 13:52:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13616208 Received: from mail-il1-f172.google.com (mail-il1-f172.google.com [209.85.166.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E3D23148844 for ; Wed, 3 Apr 2024 13:56:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712152572; cv=none; b=bOt3rEs62Rf3gwwf4hxyYbn/D8MFPdS7u1sLKNNu71tSwEj/8DfO5IPWjlLt4myfpQ+Y48L6PgLNF0aBxB+7FxnEH1jwqp4NIoy3kccvhgGsbSwoY7R0D0vA1J0VQm97+lX+dKAxFtsL15hCjkpxFIo43cr+9d4930ljvpX7y7U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712152572; c=relaxed/simple; bh=2zyZqrh0NeVxqGOPo5tUGzqQ6zCy6N2KuRJRBu9FFKs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=niFzg6jkzx+sYVaBdjWs5XC3a5LmRecO6ndAjVLs8fg3KocPVEVmtSwD+/JJU8pXqHqqEXiFDbOovbcVxeIGL8//k9+qWrkzpoCBHaayxAQtBtuboJ/oJc1wvIgv1jL3/En5EUMgd+q+/7LboFblHSkT5qjFKjPjpxQMjEOsrMY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=YO6hvnhb; arc=none smtp.client-ip=209.85.166.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="YO6hvnhb" Received: by mail-il1-f172.google.com with SMTP id e9e14a558f8ab-369f8526c85so306545ab.1 for ; Wed, 03 Apr 2024 06:56:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1712152568; x=1712757368; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YpyMnkGLPfuCdpP3GgziuHRh4o2UqLyww0R8LKUm/K0=; b=YO6hvnhbZcyHlgMzrC0B7Y6MGGkjl+29cXx5eqei3F3YbQtwC+aPDtCGS/TNxGkIAX lzZiPgBMa5f9/sKKkf03mqn0ta1e0xIVY+G7MiZDIhrwAZNZDX9gVsDl+gZjEH8GzFmR XUjw/K6xoAhzibpxfarxyHPkBIOm7LOzxvFVD48opM5GkCyaaj6H2ktOcaU3DfBlN7G1 I7gF/8KYfWQUeVCVlLoDcCSrv0k4x0DovzxXdvqYyl3iWBr1MO47qH6eZu5UuLL12T4R 76ngEptSEyzoY5ZSweZxn2DHNGgDbtP0qwcOsLv2RX5+N9hRlMyR3alqdDJzeUwHJ7dV ndWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712152568; x=1712757368; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YpyMnkGLPfuCdpP3GgziuHRh4o2UqLyww0R8LKUm/K0=; b=hD8MzeqAxPN1/53ADr1Yqu2oj0LCuU94Mc6tlBH5ODiphNEq8l5OOsSyveDUDydSp4 iJIcXoy88pe2D8rAdQqfWbh2LNrEXHA/UtJpBlt3yBmviS5X38CJJlU0qnNdDLytJVrF HdwnW7tZtTp4hsTGTQqLHrl7+F97oihojY+aSGjCm3YVo7pql92nKgFX647i6dE/RcjR rxn2+Tuj9B6wIj7SrMjurZfD39uN+pmoKYqyie2XtsfsQhGT24n6rSCFXYBkz8dm0h0T QuTs9Qvrn8Ax0fZfILjUydhD32PxX0ZyYmprmmqS/+BZUJ2sbdt7zhuHuuT4lTYLu553 Vr4w== X-Gm-Message-State: AOJu0YyznECFwW1nphYNIdytGExSUhyfuGnbZsEqaEh5Du/JOBJsnlrX DIkV4yVliMrtOc9FOvN3nAhYra2KDK5YmJOe6x6ekS48m6ahIUzJ3uBXydn6JhT0n/QxaIr02fE q X-Google-Smtp-Source: AGHT+IHyA0Uc96KgIHa3U6IJQjDMOIVhB0P5ca7DEqSCC1nFojhhMnImqj34EY72Yehdc5elZ+cgfQ== X-Received: by 2002:a6b:5907:0:b0:7c8:d514:9555 with SMTP id n7-20020a6b5907000000b007c8d5149555mr14852479iob.1.1712152568532; Wed, 03 Apr 2024 06:56:08 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id l14-20020a02ccee000000b0047ec296d3c1sm3839460jaq.19.2024.04.03.06.56.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Apr 2024 06:56:06 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe , stable@vger.kernel.org Subject: [PATCH 1/4] io_uring/kbuf: get rid of lower BGID lists Date: Wed, 3 Apr 2024 07:52:34 -0600 Message-ID: <20240403135602.1623312-2-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240403135602.1623312-1-axboe@kernel.dk> References: <20240403135602.1623312-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Just rely on the xarray for any kind of bgid. This simplifies things, and it really doesn't bring us much, if anything. Cc: stable@vger.kernel.org # v6.4+ Signed-off-by: Jens Axboe --- include/linux/io_uring_types.h | 1 - io_uring/io_uring.c | 2 - io_uring/kbuf.c | 70 ++++------------------------------ 3 files changed, 8 insertions(+), 65 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index e24893625085..05df0e399d7c 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -294,7 +294,6 @@ struct io_ring_ctx { struct io_submit_state submit_state; - struct io_buffer_list *io_bl; struct xarray io_bl_xa; struct io_hash_table cancel_table_locked; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index d1defb99b89e..bc730f59265f 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -351,7 +351,6 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) err: kfree(ctx->cancel_table.hbs); kfree(ctx->cancel_table_locked.hbs); - kfree(ctx->io_bl); xa_destroy(&ctx->io_bl_xa); kfree(ctx); return NULL; @@ -2932,7 +2931,6 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) io_napi_free(ctx); kfree(ctx->cancel_table.hbs); kfree(ctx->cancel_table_locked.hbs); - kfree(ctx->io_bl); xa_destroy(&ctx->io_bl_xa); kfree(ctx); } diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index 693c26da4ee1..8bf0121f00af 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -17,8 +17,6 @@ #define IO_BUFFER_LIST_BUF_PER_PAGE (PAGE_SIZE / sizeof(struct io_uring_buf)) -#define BGID_ARRAY 64 - /* BIDs are addressed by a 16-bit field in a CQE */ #define MAX_BIDS_PER_BGID (1 << 16) @@ -40,13 +38,9 @@ struct io_buf_free { int inuse; }; -static struct io_buffer_list *__io_buffer_get_list(struct io_ring_ctx *ctx, - struct io_buffer_list *bl, - unsigned int bgid) +static inline struct io_buffer_list *__io_buffer_get_list(struct io_ring_ctx *ctx, + unsigned int bgid) { - if (bl && bgid < BGID_ARRAY) - return &bl[bgid]; - return xa_load(&ctx->io_bl_xa, bgid); } @@ -55,7 +49,7 @@ static inline struct io_buffer_list *io_buffer_get_list(struct io_ring_ctx *ctx, { lockdep_assert_held(&ctx->uring_lock); - return __io_buffer_get_list(ctx, ctx->io_bl, bgid); + return __io_buffer_get_list(ctx, bgid); } static int io_buffer_add_list(struct io_ring_ctx *ctx, @@ -68,10 +62,6 @@ static int io_buffer_add_list(struct io_ring_ctx *ctx, */ bl->bgid = bgid; smp_store_release(&bl->is_ready, 1); - - if (bgid < BGID_ARRAY) - return 0; - return xa_err(xa_store(&ctx->io_bl_xa, bgid, bl, GFP_KERNEL)); } @@ -208,24 +198,6 @@ void __user *io_buffer_select(struct io_kiocb *req, size_t *len, return ret; } -static __cold int io_init_bl_list(struct io_ring_ctx *ctx) -{ - struct io_buffer_list *bl; - int i; - - bl = kcalloc(BGID_ARRAY, sizeof(struct io_buffer_list), GFP_KERNEL); - if (!bl) - return -ENOMEM; - - for (i = 0; i < BGID_ARRAY; i++) { - INIT_LIST_HEAD(&bl[i].buf_list); - bl[i].bgid = i; - } - - smp_store_release(&ctx->io_bl, bl); - return 0; -} - /* * Mark the given mapped range as free for reuse */ @@ -300,13 +272,6 @@ void io_destroy_buffers(struct io_ring_ctx *ctx) struct list_head *item, *tmp; struct io_buffer *buf; unsigned long index; - int i; - - for (i = 0; i < BGID_ARRAY; i++) { - if (!ctx->io_bl) - break; - __io_remove_buffers(ctx, &ctx->io_bl[i], -1U); - } xa_for_each(&ctx->io_bl_xa, index, bl) { xa_erase(&ctx->io_bl_xa, bl->bgid); @@ -489,12 +454,6 @@ int io_provide_buffers(struct io_kiocb *req, unsigned int issue_flags) io_ring_submit_lock(ctx, issue_flags); - if (unlikely(p->bgid < BGID_ARRAY && !ctx->io_bl)) { - ret = io_init_bl_list(ctx); - if (ret) - goto err; - } - bl = io_buffer_get_list(ctx, p->bgid); if (unlikely(!bl)) { bl = kzalloc(sizeof(*bl), GFP_KERNEL_ACCOUNT); @@ -507,14 +466,9 @@ int io_provide_buffers(struct io_kiocb *req, unsigned int issue_flags) if (ret) { /* * Doesn't need rcu free as it was never visible, but - * let's keep it consistent throughout. Also can't - * be a lower indexed array group, as adding one - * where lookup failed cannot happen. + * let's keep it consistent throughout. */ - if (p->bgid >= BGID_ARRAY) - kfree_rcu(bl, rcu); - else - WARN_ON_ONCE(1); + kfree_rcu(bl, rcu); goto err; } } @@ -679,12 +633,6 @@ int io_register_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg) if (reg.ring_entries >= 65536) return -EINVAL; - if (unlikely(reg.bgid < BGID_ARRAY && !ctx->io_bl)) { - int ret = io_init_bl_list(ctx); - if (ret) - return ret; - } - bl = io_buffer_get_list(ctx, reg.bgid); if (bl) { /* if mapped buffer ring OR classic exists, don't allow */ @@ -734,10 +682,8 @@ int io_unregister_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg) return -EINVAL; __io_remove_buffers(ctx, bl, -1U); - if (bl->bgid >= BGID_ARRAY) { - xa_erase(&ctx->io_bl_xa, bl->bgid); - kfree_rcu(bl, rcu); - } + xa_erase(&ctx->io_bl_xa, bl->bgid); + kfree_rcu(bl, rcu); return 0; } @@ -771,7 +717,7 @@ void *io_pbuf_get_address(struct io_ring_ctx *ctx, unsigned long bgid) { struct io_buffer_list *bl; - bl = __io_buffer_get_list(ctx, smp_load_acquire(&ctx->io_bl), bgid); + bl = __io_buffer_get_list(ctx, bgid); if (!bl || !bl->is_mmap) return NULL; From patchwork Wed Apr 3 13:52:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13616209 Received: from mail-il1-f179.google.com (mail-il1-f179.google.com [209.85.166.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0537B14831E for ; Wed, 3 Apr 2024 13:56:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712152574; cv=none; b=bBz4HBEVs1dlDNskt+Jt0C7vvnPWidohOG1jZPktDbyf09m6yAnuu+m5E7ezSot/Cw/Nw6Sn+cBsWfVM6M9baf8pcR/Kw+A9rcb4aCqfZDJw+6ayfHDOJhHUZQXaqMT9OFTeL6cvWUEnMTrBEkAQOPfdJM0xLCg/6ZEnSjXBWa0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712152574; c=relaxed/simple; bh=r6XgXgs9mKWgscIg4VAtSi4STibkfVlgbPBlVpthNeo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=T5TwwZlAduIyh+DHgAltPgRH0mhGjB9gllCpD3K93xZ7ysuWLFap+8xjhLu9FN+jv8AFAaKS1q7ZCnu7JYdHyMa18c4654OlS73BeFKRgTAiqk1RGXZYidk/t7ugZs7k6+mdsDA344USIqHwPpPgkkdYtTPKbJcH5bpiBKh0Q+s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=ojex1MN0; arc=none smtp.client-ip=209.85.166.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="ojex1MN0" Received: by mail-il1-f179.google.com with SMTP id e9e14a558f8ab-3688f1b7848so3848335ab.0 for ; Wed, 03 Apr 2024 06:56:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1712152571; x=1712757371; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dFSNOKrcq5ApWo4Nga/6FYpd0SvuTRy5qoeoKrxISH4=; b=ojex1MN0hquv2mV3urXahKOBXDBbMpQ3XmrzcCaH1hvQpDodFMRaExgnh9bFTan3dL wxBynVuuV9nGIUPiggPLs0v6AcMBB81Ni2TC4VjrBO3UiZP1aACoXKH5zUWwHesWIz2d rMiUhCsiCPr62XWmNrcEzIWAqwi6DsZdgEHKbi3/nz+t14Q/WcB3XS5XC2hUoR7sHS5B eoiIcdP7qTBZ3dv7DGWSh5zeSZa5LKHcD+ys1OGW8RCZEc87NdeAEH52OKQ2DJalRgD8 8HXVWU+KRiklik0HdQweY7khFRuLpk8j4Ui8wUNvhPejx/8r6abbP1HMTR3wPMMP77os HWsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712152571; x=1712757371; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dFSNOKrcq5ApWo4Nga/6FYpd0SvuTRy5qoeoKrxISH4=; b=E3CfZIzGhj/TMwFCx4/fd/BY0jrD944gNKMu+VkupwBmLoI53UXU4mWsbZpdOcbqN4 hSR+MeU+ejkWGQc64664wGxckDMrd9bTv9FZ0fzWBsFYF6+r4lw2iY4UYv3LnEvglvem 1WTPemj+nTKlWMRC6t+rwSO6IdBvDYmMxlBkGz2GML8CKVhaRVliKxBMPcC6cGIwgaUw M5maTGVQIw6sccxN1uL8QN8iC+wapxQ4xCjzobFQOSWdsHq6bOB9O3uuxH6qdbMl9fLY J77Diqb2JA0nOYrnBI8jaZiHLEr45sUdfCa9wfFKbmTSJwia6M6rz7XS8B8llqX3ByT/ iB3w== X-Gm-Message-State: AOJu0Ywzw83Un7fzuUMHiXPANJ4Vxr649a1g0682ci9RJ8F+aWnsTGJR PqFKMaOr5ZVa/5xl5OJUoRzqOiI/PitLx/sQ3yvtHpMSfIAouXNmWApJ5FivmpdnuNWFAJnLi4u p X-Google-Smtp-Source: AGHT+IF9Zi4uvNNy7T1FuA+tqiqprZAvAFxTCAusdlcoyvAnEk+vYjAs4AoV51xGoFTdzwF3oo52vQ== X-Received: by 2002:a6b:5907:0:b0:7c8:d514:9555 with SMTP id n7-20020a6b5907000000b007c8d5149555mr14852580iob.1.1712152570679; Wed, 03 Apr 2024 06:56:10 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id l14-20020a02ccee000000b0047ec296d3c1sm3839460jaq.19.2024.04.03.06.56.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Apr 2024 06:56:08 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe , stable@vger.kernel.org Subject: [PATCH 2/4] io_uring/kbuf: get rid of bl->is_ready Date: Wed, 3 Apr 2024 07:52:35 -0600 Message-ID: <20240403135602.1623312-3-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240403135602.1623312-1-axboe@kernel.dk> References: <20240403135602.1623312-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Now that xarray is being exclusively used for the buffer_list lookup, this check is no longer needed. Get rid of it and the is_ready member. Cc: stable@vger.kernel.org # v6.4+ Signed-off-by: Jens Axboe --- io_uring/kbuf.c | 8 -------- io_uring/kbuf.h | 2 -- 2 files changed, 10 deletions(-) diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index 8bf0121f00af..011280d873e7 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -61,7 +61,6 @@ static int io_buffer_add_list(struct io_ring_ctx *ctx, * always under the ->uring_lock, but the RCU lookup from mmap does. */ bl->bgid = bgid; - smp_store_release(&bl->is_ready, 1); return xa_err(xa_store(&ctx->io_bl_xa, bgid, bl, GFP_KERNEL)); } @@ -721,13 +720,6 @@ void *io_pbuf_get_address(struct io_ring_ctx *ctx, unsigned long bgid) if (!bl || !bl->is_mmap) return NULL; - /* - * Ensure the list is fully setup. Only strictly needed for RCU lookup - * via mmap, and in that case only for the array indexed groups. For - * the xarray lookups, it's either visible and ready, or not at all. - */ - if (!smp_load_acquire(&bl->is_ready)) - return NULL; return bl->buf_ring; } diff --git a/io_uring/kbuf.h b/io_uring/kbuf.h index 1c7b654ee726..fdbb10449513 100644 --- a/io_uring/kbuf.h +++ b/io_uring/kbuf.h @@ -29,8 +29,6 @@ struct io_buffer_list { __u8 is_buf_ring; /* ring mapped provided buffers, but mmap'ed by application */ __u8 is_mmap; - /* bl is visible from an RCU point of view for lookup */ - __u8 is_ready; }; struct io_buffer { From patchwork Wed Apr 3 13:52:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13616210 Received: from mail-io1-f50.google.com (mail-io1-f50.google.com [209.85.166.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E475B148FEA for ; Wed, 3 Apr 2024 13:56:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712152575; cv=none; b=Le35xFcnVotTZPAsWMFhPSU0hQCHut8bUyGle0kmfh5MwECOQCz+Hh4Oc4vuRV3IX3LdL0KbOB1ycwE+jdi7SEuFL6c8WJC3YNWAJWkeEKd8FRpztJJhOawFoNe85KJhMyLE+bVmUHKqdTWLP8LHYU5rHB3OQixwXVehHVA4tzU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712152575; c=relaxed/simple; bh=Q3ertwRbJ2RqliQR4i/fwo7C3RrhtepZnt+fHIKsOs8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=egx48O8YgxD8O4PEGqCR8IRhCDIAk6YrghxuznVE/kFgMXzhMgJuoDnlneMUCzD/DLJBDNRotJOazMfzv4qsAA7wSq3ScNpDptY+RKzlZHTOtMkZ2+rFtLV8uWUNMDMGkwYiy0V/9fUjDhPlDNHbr7Zkn/pOhUXNq3vcF3hRDxI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=nu0J10Zb; arc=none smtp.client-ip=209.85.166.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="nu0J10Zb" Received: by mail-io1-f50.google.com with SMTP id ca18e2360f4ac-7cc0e831e11so60237239f.1 for ; Wed, 03 Apr 2024 06:56:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1712152572; x=1712757372; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7ZqqJA8ALtTyDvrrt7I55rk6zP/dlq3hEacG6s3xgPg=; b=nu0J10Zb1SzD4K4YZCTugZgc1J+/l4T7BQfaNwBR4JiPHL+NNG4IXoyUSs4WsVM5Qt Ty9pxAjncaAX5dnqco6DDvq9B2rcPEP+QONiE823USUsG2EyRY7iCmBfUb/Yr+UmBQ+R yq6dRJMT4mJTY8BGHLYifGmNjUS/Y383WoAz71rEQLHS2vmNT65lh3gQmJC5EfE/0fuT sb+sb8Ar0DIEPatL67pdFJC2yucbs2GO3Zkq13HkdbU4HsVYlaU8v7BCxqDJaMApT5Vz xAqieyMIdyb1wlt/oEKStZXikxmFpPHCtmMRvunzUmAb7a7JFCAUhwu8LAoMbfPdg51V f9+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712152572; x=1712757372; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7ZqqJA8ALtTyDvrrt7I55rk6zP/dlq3hEacG6s3xgPg=; b=vGHmrhsY6loBRlQreOg1G12l3oFc1h/wHRqwTeYmhdKL168SCA2CxQibpRb4pJefXR aig0PLvsrMRxvql+4ltoJTTCnAtc0UheSS4odD50G69oBbaoGeRAyj28P3hzo9Ug7lnF If6fo7ldO/3lT7P8JMf5ibXKRcl2b6R8J+rHFh9mAoww434sQWmNWZ/CEIuSIdbMONUV Wj0vLTt7k/nF+2pnhua3H58UL6jav03OD5a5NiLV1YD1zXGSq3qiEbih+pOemiNk0weS Q2uBJsVU/lw7HcETRIO0a//PjvB5a7Z6a/rEvbMoyzMMZzxYFoazVnXYprwJzY9BJRIN akhw== X-Gm-Message-State: AOJu0YyjWEGxXIPBk4WVZxR6r6x0uuK8c9MsPVaDF974ceh+gb5DHfAP KehJuhLb8OAyQhaSm9CDwhsN/Tp5ltzvvmVweUAyV8/gLuX50XaYFX/NRx19KiKbdJiwITHPZln 6 X-Google-Smtp-Source: AGHT+IEQZcSivTFZIHLlrt0jRWclzP+hLkh7BSrRwFr6CyInaPLSBuHjFf9QQChSbWn6Jabz3IdU8A== X-Received: by 2002:a6b:c949:0:b0:7d0:bd2b:43ba with SMTP id z70-20020a6bc949000000b007d0bd2b43bamr11499467iof.0.1712152572647; Wed, 03 Apr 2024 06:56:12 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id l14-20020a02ccee000000b0047ec296d3c1sm3839460jaq.19.2024.04.03.06.56.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Apr 2024 06:56:11 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe , stable@vger.kernel.org Subject: [PATCH 3/4] io_uring/kbuf: protect io_buffer_list teardown with a reference Date: Wed, 3 Apr 2024 07:52:36 -0600 Message-ID: <20240403135602.1623312-4-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240403135602.1623312-1-axboe@kernel.dk> References: <20240403135602.1623312-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 No functional changes in this patch, just in preparation for being able to keep the buffer list alive outside of the ctx->uring_lock. Cc: stable@vger.kernel.org # v6.4+ Signed-off-by: Jens Axboe --- io_uring/kbuf.c | 15 +++++++++++---- io_uring/kbuf.h | 2 ++ 2 files changed, 13 insertions(+), 4 deletions(-) diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index 011280d873e7..2edc6854f6f3 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -61,6 +61,7 @@ static int io_buffer_add_list(struct io_ring_ctx *ctx, * always under the ->uring_lock, but the RCU lookup from mmap does. */ bl->bgid = bgid; + atomic_set(&bl->refs, 1); return xa_err(xa_store(&ctx->io_bl_xa, bgid, bl, GFP_KERNEL)); } @@ -265,6 +266,14 @@ static int __io_remove_buffers(struct io_ring_ctx *ctx, return i; } +static void io_put_bl(struct io_ring_ctx *ctx, struct io_buffer_list *bl) +{ + if (atomic_dec_and_test(&bl->refs)) { + __io_remove_buffers(ctx, bl, -1U); + kfree_rcu(bl, rcu); + } +} + void io_destroy_buffers(struct io_ring_ctx *ctx) { struct io_buffer_list *bl; @@ -274,8 +283,7 @@ void io_destroy_buffers(struct io_ring_ctx *ctx) xa_for_each(&ctx->io_bl_xa, index, bl) { xa_erase(&ctx->io_bl_xa, bl->bgid); - __io_remove_buffers(ctx, bl, -1U); - kfree_rcu(bl, rcu); + io_put_bl(ctx, bl); } /* @@ -680,9 +688,8 @@ int io_unregister_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg) if (!bl->is_buf_ring) return -EINVAL; - __io_remove_buffers(ctx, bl, -1U); xa_erase(&ctx->io_bl_xa, bl->bgid); - kfree_rcu(bl, rcu); + io_put_bl(ctx, bl); return 0; } diff --git a/io_uring/kbuf.h b/io_uring/kbuf.h index fdbb10449513..8b868a1744e2 100644 --- a/io_uring/kbuf.h +++ b/io_uring/kbuf.h @@ -25,6 +25,8 @@ struct io_buffer_list { __u16 head; __u16 mask; + atomic_t refs; + /* ring mapped provided buffers */ __u8 is_buf_ring; /* ring mapped provided buffers, but mmap'ed by application */ From patchwork Wed Apr 3 13:52:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13616211 Received: from mail-il1-f170.google.com (mail-il1-f170.google.com [209.85.166.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B9AC1148FE1 for ; Wed, 3 Apr 2024 13:56:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712152577; cv=none; b=CoC9W619ezRKLhbExxYa5B5aEUawL3pezmTaLX7E2BMVpfsrpxbhTuKGDxy8KYvpPDqHFSnAb/CTQ2bqJaug/XgeZr3UujgZ+5O5QNEQrJdazmBovFJjZQY17kR/uaz2Rf3dogLQoAg8rMNfhVkFfDpMO6Kqyj7v1oTaPM8JVrU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712152577; c=relaxed/simple; bh=xBW+RidIDSaDi5t9YoWxK8yP48UNn3X5QEgKFKpA+hU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=VvaiAPO6rsQXdzfSRMCxm6TxZm/rjejlQZn4JVawiLK8qx50kgcKZjvCXZI/r2p3GeWpThvXmqUdFlZGxkrTw3e3prpg+aVkyEvQ+hf0grtmk6yKLIuDUoPg/agSaf3OY4vWtitQxV7ce5K3jT9rdk/BvNkfUJ9JZr75MdL7a6M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=ckKe8nfK; arc=none smtp.client-ip=209.85.166.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="ckKe8nfK" Received: by mail-il1-f170.google.com with SMTP id e9e14a558f8ab-369f8526c85so306715ab.1 for ; Wed, 03 Apr 2024 06:56:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1712152574; x=1712757374; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qyjiacDLhmmFlItH/f0t7RHBaXaSqUN8QHctrPBrOeE=; b=ckKe8nfKxnLIImX85CR/HuwNn2tKRo1UU+XcZmsur3R0MJ5Bqe6+r1kujDPogTODVi 2CDLJ5kMDHuwH06X1jzZxcbLwDvYBHBuv0NJQ2XODfy5vbCiyv/vN80wKglQJ4GiKJct 8qlXs9jYT9GxFTBpAHxIra2b7Zscf23IjHF/kq1jzEUOa915nVhy0yfhsA4OotPmtJfS 8sQGPYWM1CN/jZnrAG0yDS+kZXHHDGxjaZvXexh29yK3hE0MxPQUo/EEcz6ICw9K5qsu zH9u5r5alU+Di9gJmUvDNbKax4+9Fx+Rl5EMXSJHtwRvwT544YkK9TYZ2YnOqsa4z/+T rmng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712152574; x=1712757374; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qyjiacDLhmmFlItH/f0t7RHBaXaSqUN8QHctrPBrOeE=; b=sS3vfJFNPzU/NIMsMomRulW3K0dXrRPUsCb3kJ8EngbPKf1G2dTi8ycwPryLtAQ4ud MEA3eg9wbHX0caCRkTwNtIM/WPrfLdWz8wfneeJ6bCSyJ+91DCqlkyH8FSRKXGRjR94/ gFX8wOpOjRZM5tmsC0qayo4jfVkev5rJPQ+IJPmzMxmkXl4oAi3WlB62rk8OQZVE6dkQ 6hKQ+PHY0sePeZvQng2F/O/hZE6WSXVEiKDOpWO0zCVT9q3TRZ+xAijbNLS5X5M0kXlP AZ6a26gpuArXVokgv+RD1FzV2/PVt2RalBj6Dj6/gLfpNjYPZRowijIR5Ae68k8tZhDA zJNg== X-Gm-Message-State: AOJu0YwcGhv7EHakl6ugAEHR5BeT/Ip81jpMJQCHZe2YBXG4cxChZ367 sG67pM1uIWm8I9HyjxroBIqOEC7fQqtBPA91rzHDsgNEc9Ooy4/jbB8YfbaJBHsOQylsuhGj0hW 8 X-Google-Smtp-Source: AGHT+IGYgQYm7nodhPxgrPIWI3m469yFPtVR9OSIwyCmRhjwkKXK+ZGyKgjdmmaaK+nKZ4WkMSFK2g== X-Received: by 2002:a5e:9914:0:b0:7d3:433a:d33d with SMTP id t20-20020a5e9914000000b007d3433ad33dmr2951968ioj.2.1712152574556; Wed, 03 Apr 2024 06:56:14 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id l14-20020a02ccee000000b0047ec296d3c1sm3839460jaq.19.2024.04.03.06.56.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Apr 2024 06:56:13 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe , stable@vger.kernel.org Subject: [PATCH 4/4] io_uring/kbuf: hold io_buffer_list reference over mmap Date: Wed, 3 Apr 2024 07:52:37 -0600 Message-ID: <20240403135602.1623312-5-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240403135602.1623312-1-axboe@kernel.dk> References: <20240403135602.1623312-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 If we look up the kbuf, ensure that it doesn't get unregistered until after we're done with it. Since we're inside mmap, we cannot safely use the io_uring lock. Rely on the fact that we can lookup the buffer list under RCU now and grab a reference to it, preventing it from being unregistered until we're done with it. The lookup returns the io_buffer_list directly with it referenced. Cc: stable@vger.kernel.org # v6.4+ Fixes: 5cf4f52e6d8a ("io_uring: free io_buffer_list entries via RCU") Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 11 ++++++----- io_uring/kbuf.c | 35 +++++++++++++++++++++++++++-------- io_uring/kbuf.h | 4 +++- 3 files changed, 36 insertions(+), 14 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index bc730f59265f..4521c2b66b98 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -3447,14 +3447,15 @@ static void *io_uring_validate_mmap_request(struct file *file, ptr = ctx->sq_sqes; break; case IORING_OFF_PBUF_RING: { + struct io_buffer_list *bl; unsigned int bgid; bgid = (offset & ~IORING_OFF_MMAP_MASK) >> IORING_OFF_PBUF_SHIFT; - rcu_read_lock(); - ptr = io_pbuf_get_address(ctx, bgid); - rcu_read_unlock(); - if (!ptr) - return ERR_PTR(-EINVAL); + bl = io_pbuf_get_bl(ctx, bgid); + if (IS_ERR(bl)) + return bl; + ptr = bl->buf_ring; + io_put_bl(ctx, bl); break; } default: diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index 2edc6854f6f3..3aa16e27f509 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -266,7 +266,7 @@ static int __io_remove_buffers(struct io_ring_ctx *ctx, return i; } -static void io_put_bl(struct io_ring_ctx *ctx, struct io_buffer_list *bl) +void io_put_bl(struct io_ring_ctx *ctx, struct io_buffer_list *bl) { if (atomic_dec_and_test(&bl->refs)) { __io_remove_buffers(ctx, bl, -1U); @@ -719,16 +719,35 @@ int io_register_pbuf_status(struct io_ring_ctx *ctx, void __user *arg) return 0; } -void *io_pbuf_get_address(struct io_ring_ctx *ctx, unsigned long bgid) +struct io_buffer_list *io_pbuf_get_bl(struct io_ring_ctx *ctx, + unsigned long bgid) { struct io_buffer_list *bl; + bool ret; - bl = __io_buffer_get_list(ctx, bgid); - - if (!bl || !bl->is_mmap) - return NULL; - - return bl->buf_ring; + /* + * We have to be a bit careful here - we're inside mmap and cannot grab + * the uring_lock. This means the buffer_list could be simultaneously + * going away, if someone is trying to be sneaky. Look it up under rcu + * so we know it's not going away, and attempt to grab a reference to + * it. If the ref is already zero, then fail the mapping. If successful, + * the caller will call io_put_bl() to drop the the reference at at the + * end. This may then safely free the buffer_list (and drop the pages) + * at that point, vm_insert_pages() would've already grabbed the + * necessary vma references. + */ + rcu_read_lock(); + bl = xa_load(&ctx->io_bl_xa, bgid); + /* must be a mmap'able buffer ring and have pages */ + ret = false; + if (bl && bl->is_mmap) + ret = atomic_inc_not_zero(&bl->refs); + rcu_read_unlock(); + + if (ret) + return bl; + + return ERR_PTR(-EINVAL); } /* diff --git a/io_uring/kbuf.h b/io_uring/kbuf.h index 8b868a1744e2..df365b8860cf 100644 --- a/io_uring/kbuf.h +++ b/io_uring/kbuf.h @@ -61,7 +61,9 @@ void __io_put_kbuf(struct io_kiocb *req, unsigned issue_flags); bool io_kbuf_recycle_legacy(struct io_kiocb *req, unsigned issue_flags); -void *io_pbuf_get_address(struct io_ring_ctx *ctx, unsigned long bgid); +void io_put_bl(struct io_ring_ctx *ctx, struct io_buffer_list *bl); +struct io_buffer_list *io_pbuf_get_bl(struct io_ring_ctx *ctx, + unsigned long bgid); static inline bool io_kbuf_recycle_ring(struct io_kiocb *req) {