From patchwork Sun Nov 24 21:12:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13884189 Received: from mail-wm1-f44.google.com (mail-wm1-f44.google.com [209.85.128.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9071613FD72 for ; Sun, 24 Nov 2024 21:12:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482725; cv=none; b=S21xsfv29r8ir5Km6tfKUJhC0w9+t04lKf4u1CQrImd/u+/rEj9gQN9ZTHMMxpz84t1obtJ9oHkjnXMzsg/zAe9TjDmwUAq/Eb9jyCbpIU5wW/nRjGsLY4Znn4RSZCQ92uNdzwDOBKNNcCkLBaANQtO/GzvhgkJRJOwzHWhc5+A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482725; c=relaxed/simple; bh=QA+01plKRIkmGEc891jaogWe5XZER36lyT3oLddZ68E=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hrQa4fydy7qOSV3eJ3eQFg2sjH9wV+rvLMiWVXC4qzEU79H+pZExRaD8yCsuhefjgW900399Aom+/86UuSttfHuZBcAsSBiZuf0cVZJ0ER8uAieP87zbIdsKjvy+HPiVWTKnwUVbXDql2xxjV4odd44cNNUXfivUXA9eKv6EjRw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ErhkeFsh; arc=none smtp.client-ip=209.85.128.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ErhkeFsh" Received: by mail-wm1-f44.google.com with SMTP id 5b1f17b1804b1-4349cc45219so6848425e9.3 for ; Sun, 24 Nov 2024 13:12:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732482722; x=1733087522; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=J8cmqKslAko5zF5qRZC7c7DtkzAvHjFueUMLswLyZOw=; b=ErhkeFshrCdNhjRCbHSe1CeJfo1rkLz/hLkDWSwE9hy70oYXJy8FcRZCN0dAJLGoDC K38EMRKEuLmFWcYKHnzJLZdzVHQ+GWJ40evCZ3IXRTJHeWDikDKP1EbVnVhDq++us1o9 LvIBF3poYpHTVXWmlslAACEb53xB/W7hRgh4M39G5omBkCr3rODtEgKTI30VpFBWiDtH BInrQNat5dJ3Ojmr1ub3HzZ+1gFZQ4XE+eRwM+69rQmvlhrBJHarHtzPdqtpK0hSCcmi I+EA9a4VDX/a0j/ilSZhs+ib8XUaEUVqMQEJUjXKBESRn8MYouTbm4ukipuRQpyGhYSj KYfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732482722; x=1733087522; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=J8cmqKslAko5zF5qRZC7c7DtkzAvHjFueUMLswLyZOw=; b=hEpIfLTlUl8k5MR03s1d+pC23Grrmx/p0z3eFmuEdNUY4ZDPbMjhdUrRWhScFyrCkG LOXAxh3htJcIbklPZp2EyCPj1J0H9sF8yhi8yznpsHElkimYjd/Uo/m16QyL23Ix/0kQ Z2xasKLVWupy3zTIKNTnCqTntpE5Kbdx5xZl0VZlUVpCbuzUqUH8bN+fjsdxj55ANugi N2MDBL7M+sVocEeFBpYw+Wpuw9VgRu312iP8anmdW+NudgcraRA2wCwdDDoTIF3Lc1Jc c0Qj5mEY/83gkIXIIrlIHNFgz9gpRUs5rCQSuHZx8pYMUUZ3bk+r9jpmgPiGF/M1EY4W 7u5A== X-Gm-Message-State: AOJu0Yxl0WsBWfZrd9b17pLsnQU2FYek1e+SMtakU0kA7H0KEEAyT0ws EOJl8fkD9t9KEcdgQUiSpp01KhuV3HIKegbxca2n+XN+su6gvFfa9nr9rA== X-Gm-Gg: ASbGnct93t/c1FMh8VE//Ivo114CXyFGdbIC/T7iOecuacD/lrymnEKbNdI6Y5fCe0u FUb4tGGgJ6rZSlo3kRQn+hQHkosnBfVEKGKBlkVlA7qVdWQrNqBdhX6bwydt6FuHd5PrPon3Lz5 ipOO8N4rttNs/FjoflrzvU7K32LWWPPryVaKQu/nVvyWwtNIAFQJaOD+4mEUgK4sG580O8SdGsF NMEB41qD565rylet3wx/5MPGdnNe1FEig8p9PTvzOkuQVEQtkvCarsJCTnaO40= X-Google-Smtp-Source: AGHT+IFM/h858U0X4lLXNs4Y0gGcK3hy36gRenWrADttddu7sssVWGQRt/zQzttG/DhVbQLx/j7zsQ== X-Received: by 2002:a05:600c:a06:b0:42c:bb10:7292 with SMTP id 5b1f17b1804b1-433ce4100c1mr79800945e9.1.1732482721499; Sun, 24 Nov 2024 13:12:01 -0800 (PST) Received: from 127.0.0.1localhost ([85.255.235.224]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-432f643b299sm132733745e9.0.2024.11.24.13.12.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 24 Nov 2024 13:12:01 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v2 01/18] io_uring: rename ->resize_lock Date: Sun, 24 Nov 2024 21:12:18 +0000 Message-ID: X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 ->resize_lock is used for resizing rings, but it's a good idea to reuse it in other cases as well. Rename it into mmap_lock as it's protects from races with mmap. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 2 +- io_uring/io_uring.c | 2 +- io_uring/memmap.c | 6 +++--- io_uring/register.c | 8 ++++---- 4 files changed, 9 insertions(+), 9 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 3e934feb3187..adb36e0da40e 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -423,7 +423,7 @@ struct io_ring_ctx { * side will need to grab this lock, to prevent either side from * being run concurrently with the other. */ - struct mutex resize_lock; + struct mutex mmap_lock; /* * If IORING_SETUP_NO_MMAP is used, then the below holds diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index bfa93888f862..fee3b7f50176 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -351,7 +351,7 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) INIT_WQ_LIST(&ctx->submit_state.compl_reqs); INIT_HLIST_HEAD(&ctx->cancelable_uring_cmd); io_napi_init(ctx); - mutex_init(&ctx->resize_lock); + mutex_init(&ctx->mmap_lock); return ctx; diff --git a/io_uring/memmap.c b/io_uring/memmap.c index 3d71756bc598..771a57a4a16b 100644 --- a/io_uring/memmap.c +++ b/io_uring/memmap.c @@ -322,7 +322,7 @@ __cold int io_uring_mmap(struct file *file, struct vm_area_struct *vma) unsigned int npages; void *ptr; - guard(mutex)(&ctx->resize_lock); + guard(mutex)(&ctx->mmap_lock); ptr = io_uring_validate_mmap_request(file, vma->vm_pgoff, sz); if (IS_ERR(ptr)) @@ -358,7 +358,7 @@ unsigned long io_uring_get_unmapped_area(struct file *filp, unsigned long addr, if (addr) return -EINVAL; - guard(mutex)(&ctx->resize_lock); + guard(mutex)(&ctx->mmap_lock); ptr = io_uring_validate_mmap_request(filp, pgoff, len); if (IS_ERR(ptr)) @@ -408,7 +408,7 @@ unsigned long io_uring_get_unmapped_area(struct file *file, unsigned long addr, struct io_ring_ctx *ctx = file->private_data; void *ptr; - guard(mutex)(&ctx->resize_lock); + guard(mutex)(&ctx->mmap_lock); ptr = io_uring_validate_mmap_request(file, pgoff, len); if (IS_ERR(ptr)) diff --git a/io_uring/register.c b/io_uring/register.c index 1e99c783abdf..ba61697d7a53 100644 --- a/io_uring/register.c +++ b/io_uring/register.c @@ -486,15 +486,15 @@ static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg) } /* - * We'll do the swap. Grab the ctx->resize_lock, which will exclude + * We'll do the swap. Grab the ctx->mmap_lock, which will exclude * any new mmap's on the ring fd. Clear out existing mappings to prevent * mmap from seeing them, as we'll unmap them. Any attempt to mmap * existing rings beyond this point will fail. Not that it could proceed * at this point anyway, as the io_uring mmap side needs go grab the - * ctx->resize_lock as well. Likewise, hold the completion lock over the + * ctx->mmap_lock as well. Likewise, hold the completion lock over the * duration of the actual swap. */ - mutex_lock(&ctx->resize_lock); + mutex_lock(&ctx->mmap_lock); spin_lock(&ctx->completion_lock); o.rings = ctx->rings; ctx->rings = NULL; @@ -561,7 +561,7 @@ static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg) ret = 0; out: spin_unlock(&ctx->completion_lock); - mutex_unlock(&ctx->resize_lock); + mutex_unlock(&ctx->mmap_lock); io_register_free_rings(&p, to_free); if (ctx->sq_data) From patchwork Sun Nov 24 21:12:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13884190 Received: from mail-wm1-f42.google.com (mail-wm1-f42.google.com [209.85.128.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1CF5B163 for ; Sun, 24 Nov 2024 21:12:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482726; cv=none; b=h4/kUCsz/PY93yyPTPNvJ4bz6RqzdYeFsqejLxEnQfqURYfjlwbpNmBqti1T9BfufiGpQl3bfXjbgV3EIoV7PuekF2QC4gPRWO06cCzFwldVLfrmqNDJxG9SXdGJ2RVK44Wi9twXLKydPP0SSmI4RHgXcZl436rzP3PngtPEgDg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482726; c=relaxed/simple; bh=1jmb/YGUScs5yrYsukVscAuiJXOxz0cRSODGRVeVGzI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SKOYXeU/KnVUWTqGIxPlmi9N6sPSulSybkCom3S8uqBO7hNhV3kfZMS9NCpjWgTdVxz/MeqzlNIO/WogE8FydNiVZsQ+phxkCSAyT8jr7abzYyYS/KNxNUEJLTmmFz4rAyWOWo3HBFQdLEEYFq+HLpFACDJmu5ua+LqLEtvtY9M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=btr6UIMz; arc=none smtp.client-ip=209.85.128.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="btr6UIMz" Received: by mail-wm1-f42.google.com with SMTP id 5b1f17b1804b1-434a099ba95so129555e9.0 for ; Sun, 24 Nov 2024 13:12:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732482723; x=1733087523; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gaPvSNc+wa5krf/6t9pRuh8xeS0CHz0dlFRi5Cy4Ht8=; b=btr6UIMzI88dtQ0kKhuLu1jrhnNZ9jqjCfhixpxFAmtXPfkHF2nxdF0hZdT9h+7CAt meQW7CKMZdOT8THsrsB5GfbhysL2j1ldO1UjttS8O7GNZkyR7QtnKNxqH7iBxmPqFjWz wX+wNFv6m5XBpwbyWWze+1eXozP7tP3NUOUFfeYr6nQAcFYx6sH4McxxBEZFNhNIE9Sa lkbSVhyO9C5r3etmw3+Aju+6Km1fdzB2Fn+zYLSV4Jm5rJlCFdv6qwBCLlnyiElifGoV KZ9f8J/4uVYH3KC9L06KxT2zJA611xSzzzRcr1h6SewPSRfxnUOefuNNkFTtarIthFCD UzKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732482723; x=1733087523; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gaPvSNc+wa5krf/6t9pRuh8xeS0CHz0dlFRi5Cy4Ht8=; b=jH9F5Atjk4v/Qw2N77GxK5/b/2gL3Vg98moG6N2veyFPfCBtmjRk0Jz7nrwhnavcfN cTblDy0VSC5JKVMtrex8I43A7ld5zdhInAWjUMbUifIljDLpQ3ClTXjo5MEqbsU+/SIQ EKurx4s/wZlaRSJ6M64bI7dt+vpQwROaOIei0fA9R2c7qoGpos0kjgEKLxP/1BdDp7kp c+YT8DKs7fe/S7eFcyeY0rMgpFOT5EWpJJ9cfiXscGGM37B74e8zjbvxDE070m9guJ1p 42t0Q6PhGh9N21iUGI5A1phUAJdvgCEwpWzN499783WR3eIojIg+gnTpmmcwqynXm3dM DnPg== X-Gm-Message-State: AOJu0Ywyplx0cCH+SK+Zlbf9t2RUzPmWkN292CJWuxLboeNrjLRekvM+ ZNTQBgCfSBMp7BLaZ5rzkhvCe3IZ//jtpmmIh2amRet4TaigqGIynoWiBg== X-Gm-Gg: ASbGncvPyR/0rkA1weSEBzWo+fFSgUWk9gipr+OMJiYaR2sk0x/jBv00QjejYliO5rN 8jGHy7tL+b1SExKRBjpBGy7McCdrBNYlPMkU1SJfapu+jxOb2Uw3DJ//xFgAfMBofcRqejjMc6X xk8uK6GDLkrp7CerZSLcbr8KYK61oonJVwQNSAsKaAs+UB9the0DFqagBw7WbOtTrEbD5/vi02F 5gJZ6daexg6/i3M1NJ8GrJ/9A55nNXSX2cNXbLyTpGe5If8NH0VkO/PWxVuxRQ= X-Google-Smtp-Source: AGHT+IGZO+gfmdQKj8NYX5z7sQDyeONWOeooKM6+STiKR/au2AZVcwRCAKoSsARdhKCyRCHSTIBFug== X-Received: by 2002:a05:6000:401f:b0:37c:d57d:71cd with SMTP id ffacd0b85a97d-38260be22f9mr9864420f8f.52.1732482722480; Sun, 24 Nov 2024 13:12:02 -0800 (PST) Received: from 127.0.0.1localhost ([85.255.235.224]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-432f643b299sm132733745e9.0.2024.11.24.13.12.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 24 Nov 2024 13:12:01 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v2 02/18] io_uring/rsrc: export io_check_coalesce_buffer Date: Sun, 24 Nov 2024 21:12:19 +0000 Message-ID: <4a82c18ed2615f3381f1849f108a421c2022c846.1732481694.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 io_try_coalesce_buffer() is a useful helper collecting useful info about a set of pages, I want to reuse it for analysing ring/etc. mappings. I don't need the entire thing and only interested if it can be coalesced into a single page, but that's better than duplicating the parsing. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 22 ++++++++++++---------- io_uring/rsrc.h | 4 ++++ 2 files changed, 16 insertions(+), 10 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index adaae8630932..e51e5ddae728 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -626,11 +626,12 @@ static int io_buffer_account_pin(struct io_ring_ctx *ctx, struct page **pages, return ret; } -static bool io_do_coalesce_buffer(struct page ***pages, int *nr_pages, - struct io_imu_folio_data *data, int nr_folios) +static bool io_coalesce_buffer(struct page ***pages, int *nr_pages, + struct io_imu_folio_data *data) { struct page **page_array = *pages, **new_array = NULL; int nr_pages_left = *nr_pages, i, j; + int nr_folios = data->nr_folios; /* Store head pages only*/ new_array = kvmalloc_array(nr_folios, sizeof(struct page *), @@ -667,15 +668,14 @@ static bool io_do_coalesce_buffer(struct page ***pages, int *nr_pages, return true; } -static bool io_try_coalesce_buffer(struct page ***pages, int *nr_pages, - struct io_imu_folio_data *data) +bool io_check_coalesce_buffer(struct page **page_array, int nr_pages, + struct io_imu_folio_data *data) { - struct page **page_array = *pages; struct folio *folio = page_folio(page_array[0]); unsigned int count = 1, nr_folios = 1; int i; - if (*nr_pages <= 1) + if (nr_pages <= 1) return false; data->nr_pages_mid = folio_nr_pages(folio); @@ -687,7 +687,7 @@ static bool io_try_coalesce_buffer(struct page ***pages, int *nr_pages, * Check if pages are contiguous inside a folio, and all folios have * the same page count except for the head and tail. */ - for (i = 1; i < *nr_pages; i++) { + for (i = 1; i < nr_pages; i++) { if (page_folio(page_array[i]) == folio && page_array[i] == page_array[i-1] + 1) { count++; @@ -715,7 +715,8 @@ static bool io_try_coalesce_buffer(struct page ***pages, int *nr_pages, if (nr_folios == 1) data->nr_pages_head = count; - return io_do_coalesce_buffer(pages, nr_pages, data, nr_folios); + data->nr_folios = nr_folios; + return true; } static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, @@ -729,7 +730,7 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, size_t size; int ret, nr_pages, i; struct io_imu_folio_data data; - bool coalesced; + bool coalesced = false; if (!iov->iov_base) return NULL; @@ -749,7 +750,8 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, } /* If it's huge page(s), try to coalesce them into fewer bvec entries */ - coalesced = io_try_coalesce_buffer(&pages, &nr_pages, &data); + if (io_check_coalesce_buffer(pages, nr_pages, &data)) + coalesced = io_coalesce_buffer(&pages, &nr_pages, &data); imu = kvmalloc(struct_size(imu, bvec, nr_pages), GFP_KERNEL); if (!imu) diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 7a4668deaa1a..c8b093584461 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -40,6 +40,7 @@ struct io_imu_folio_data { /* For non-head/tail folios, has to be fully included */ unsigned int nr_pages_mid; unsigned int folio_shift; + unsigned int nr_folios; }; struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx, int type); @@ -66,6 +67,9 @@ int io_register_rsrc_update(struct io_ring_ctx *ctx, void __user *arg, int io_register_rsrc(struct io_ring_ctx *ctx, void __user *arg, unsigned int size, unsigned int type); +bool io_check_coalesce_buffer(struct page **page_array, int nr_pages, + struct io_imu_folio_data *data); + static inline struct io_rsrc_node *io_rsrc_node_lookup(struct io_rsrc_data *data, int index) { From patchwork Sun Nov 24 21:12:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13884192 Received: from mail-wr1-f49.google.com (mail-wr1-f49.google.com [209.85.221.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 56E17163 for ; Sun, 24 Nov 2024 21:12:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482729; cv=none; b=FNfxh7lbg/28mIobqp0AjsT1WB8uaWq7uECLm2ZXP2jPNxwnWFOetzzR7hSJK4y2mqoKQi7E1Dod4MppjsbToiILZJhOMMO1HDosu277dS4iAHd3OxAfv7pc0F4lWCPvIgjo7bZXHCrUbPK0GVB2rxKrgDK4+W2U6iHlEurEsnY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482729; c=relaxed/simple; bh=5erk2yEKdY6mg7w6j7Qd+n7h2gEJkjZ3W2/qXdoyLx4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QXs9i54W9Q4XFghKd2dhHqKViB3za3BOa7IzAf0r+FF/bF/gFh9yGc0f82oseG+cEyP020OrNkQc+jCNFF2DOf0uGmDd0JhvxJ4CYv8Ia3xmyCTDM7FvaT0ya99c6GY7qwvEyv+nCbVzGkHaaUPwNXpZ/YQnAc4AL0xSGrZvLXI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=KNE0Lbp7; arc=none smtp.client-ip=209.85.221.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="KNE0Lbp7" Received: by mail-wr1-f49.google.com with SMTP id ffacd0b85a97d-3823194a879so2732275f8f.0 for ; Sun, 24 Nov 2024 13:12:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732482725; x=1733087525; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GUoVGY3EqP0iERkK/1XCe67cv9MaIdhCuF65Sh1CrCU=; b=KNE0Lbp7JH7G395omFNkfWch0Tm4iq8RJqvGA3WZAnDd5wsJZtCjLpFEXgOABYBvX5 qWy2n5mzb5/Nf8EX/FQ9f9nKjcM6qy40JksORAvab9oBc/yb1BnNOuCWS0NLI9xNHCUk 9JN/z4vJa+K7nU2I0fFmePgagK98vtpAKhRYLusjqeIuw6FewRNlQEI2pKEI4QufYajs L295KwSxIC8DqkuNyqB1CN56WW8Z4RklV9M0s1GII/w27FDt17LJLlRmobwTJtsIkHom SaNE/EUcJ8y2pvIRi370jF/lcPP60+GSVRiRCazuAsZjhFDvZ0eP+2U37tI/0KPf7BGC OAug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732482725; x=1733087525; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GUoVGY3EqP0iERkK/1XCe67cv9MaIdhCuF65Sh1CrCU=; b=bCXNvKs+hFYtVyEwZDDJSVLUjlypjmKG+I3EE9TOKX9SZVITiMhghFGtd882Apoksi bOMWslJTj9VBrGsnbralbRj0t/n7ykALarvsxmvYBR0RjNlLuXpHaSrLd6YGgYtxMKS4 sD351c9fhndrYH0cwXYVVsFvXl0fwpfBZvynI2LuLWAf9fBdNDLcAUrjplJ/CgSG4THG mgRHs/GFVMvTiJ8uFJojW28SKhOUUhJKpuTvuiAAIxrJifGYsqMepsJafA3HcVfrTx02 vFMcpCICwGYCQg+NEpD4PxFiV6k28ps//QJQBQCEeVF4uBq2dvtx66aCjzBf8LJ3Ebum B7SA== X-Gm-Message-State: AOJu0YznqzOMDnhNpTUP5C7fj6nUbtvX9UQ37eUZ2ImUy6FulHA/qriD 9kiAwcw3qCBu3HIM+e6Dq3meZlX49905yp0RiDnJ5luB33qwFyUcH1nmVQ== X-Gm-Gg: ASbGncuchQDcwUFHReF39o64+4m3ACUTlSFlyJKOn9teZAvq/PvP9pOo5wHGZ/1civx b08NhFuTWzct0ADyWzuzYak7SoYjlChWKqBjWYB/YMF+v39Mt1PjoFTHhY3ha2qme0cP5uYbvO6 V5940qrYWZf3guB/9NkOnk82u0zK/YUsNuj5fQ0PB3kcCi8q0Xebo1QhqfdB9VmU65fXvNAXdw3 M/ox4zUATKPg4feoIXodYQWuVWECDCpVkZ/cwOZN+ln5YdIpMRxNTSWKiMsLjc= X-Google-Smtp-Source: AGHT+IEuwLa5Gv7urFIak8BJ8372UL+hiNxZR36FeGAiq0+UVeS6Y9JT1/tn/pFH8RuJL3E39cJBtg== X-Received: by 2002:a5d:64ad:0:b0:382:4721:2391 with SMTP id ffacd0b85a97d-38260bf7395mr8628238f8f.57.1732482723975; Sun, 24 Nov 2024 13:12:03 -0800 (PST) Received: from 127.0.0.1localhost ([85.255.235.224]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-432f643b299sm132733745e9.0.2024.11.24.13.12.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 24 Nov 2024 13:12:03 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v2 03/18] io_uring/memmap: add internal region flags Date: Sun, 24 Nov 2024 21:12:20 +0000 Message-ID: <42ed6e9ec19f0391188f405716aeb86bb6396fba.1732481694.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add internal flags for struct io_mapped_region, it will help to add more functionality while not bloating struct io_mapped_region. Use it to mark if the pointer needs to be vunmap'ed. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 5 +++-- io_uring/memmap.c | 13 +++++++++---- io_uring/memmap.h | 2 +- 3 files changed, 13 insertions(+), 7 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index adb36e0da40e..4cee414080fd 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -77,8 +77,9 @@ struct io_hash_table { struct io_mapped_region { struct page **pages; - void *vmap_ptr; - size_t nr_pages; + void *ptr; + unsigned nr_pages; + unsigned flags; }; /* diff --git a/io_uring/memmap.c b/io_uring/memmap.c index 771a57a4a16b..21353ea09b39 100644 --- a/io_uring/memmap.c +++ b/io_uring/memmap.c @@ -195,14 +195,18 @@ void *__io_uaddr_map(struct page ***pages, unsigned short *npages, return ERR_PTR(-ENOMEM); } +enum { + IO_REGION_F_VMAP = 1, +}; + void io_free_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr) { if (mr->pages) { unpin_user_pages(mr->pages, mr->nr_pages); kvfree(mr->pages); } - if (mr->vmap_ptr) - vunmap(mr->vmap_ptr); + if ((mr->flags & IO_REGION_F_VMAP) && mr->ptr) + vunmap(mr->ptr); if (mr->nr_pages && ctx->user) __io_unaccount_mem(ctx->user, mr->nr_pages); @@ -218,7 +222,7 @@ int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, void *vptr; u64 end; - if (WARN_ON_ONCE(mr->pages || mr->vmap_ptr || mr->nr_pages)) + if (WARN_ON_ONCE(mr->pages || mr->ptr || mr->nr_pages)) return -EFAULT; if (memchr_inv(®->__resv, 0, sizeof(reg->__resv))) return -EINVAL; @@ -253,8 +257,9 @@ int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, } mr->pages = pages; - mr->vmap_ptr = vptr; + mr->ptr = vptr; mr->nr_pages = nr_pages; + mr->flags |= IO_REGION_F_VMAP; return 0; out_free: if (pages_accounted) diff --git a/io_uring/memmap.h b/io_uring/memmap.h index f361a635b6c7..2096a8427277 100644 --- a/io_uring/memmap.h +++ b/io_uring/memmap.h @@ -28,7 +28,7 @@ int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, static inline void *io_region_get_ptr(struct io_mapped_region *mr) { - return mr->vmap_ptr; + return mr->ptr; } static inline bool io_region_is_set(struct io_mapped_region *mr) From patchwork Sun Nov 24 21:12:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13884191 Received: from mail-wm1-f50.google.com (mail-wm1-f50.google.com [209.85.128.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 184FF165EFC for ; Sun, 24 Nov 2024 21:12:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482728; cv=none; b=ndNQZJMlzlf0Q0/TOZf81U4t0gUj0fIb4+t7HPU5Co9SOgyoq3B3oTwtlKI5mTdgdznpz0Uahwok3/AxTr1L+dBPKL0YrqnzJVSpeHsm1IoeKYjvDF0dMaTjByXtFys1iDSwm2MqoA/MGq03xBBsblxTR6FdCSEmg+o+TIBG2eY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482728; c=relaxed/simple; bh=1gdiLeCNSwVhK3nzWHMR4xFD8ClHJFNdvlyHODABro0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=A0hefYjvqqSN2nPjRGdH79tuGB7qEVOj91AQW/8yvQ+z9N4Opq9IxoyTZKOINgr55yDCgu7mqKYi5BhOPd5zYqxFqYobn1fZoVLj9MuGHLQBilARlGNlmOUoz8x+rJx95OUWdHr0ErF7laKCzsyPTUinBIB6uyh7oWn4/XH/Efk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=kowIrtKs; arc=none smtp.client-ip=209.85.128.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="kowIrtKs" Received: by mail-wm1-f50.google.com with SMTP id 5b1f17b1804b1-43168d9c6c9so34993105e9.3 for ; Sun, 24 Nov 2024 13:12:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732482725; x=1733087525; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=usEgNxoUAlzm7LvnyJowdrWjygvIn9+DGkjQ7Uu9NSo=; b=kowIrtKsk1htJr6Rtl5vLTfbIf+RNfEUmzAMmirUcZot1JtH5heR9jbDTHlrOQODfj hiGs0m9Z22fNXf1AN9SNM4zL/dlnZ5yisjUeDcsvRE/Kxm/niGL9i6lAglSwamr/udlg uV9+GIbVxwEySop6Mbqa/B8rSjn4IYgxUZl4gsKqoYm3UqdIVgaJhm9oIaMMq+Ky5MHt dDbnDztI89zI2NETcPprAJk4NMjWxdFSPIoBUX13r/+G5yHVvD/C0OCqWqm/49coFkrU LFuoatO4mN6WOLxXzgjNvs30SePAmVw1pw9xcnaZTP0UozS9mifDDbgFGw6yL9ngIW2R uBFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732482725; x=1733087525; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=usEgNxoUAlzm7LvnyJowdrWjygvIn9+DGkjQ7Uu9NSo=; b=RyVjsSBEx8uYHu9M93Oj8oGELqyxPA6LX4ZSPwdV8graGIVMGK6eekXKXSIrR0n9D2 sBLVQk+KF0eHWVjj2cHSIEGyaH/YY8f2a9D44tEpzxCH/jN0/ZKuYkfaNVjqmnmrm3+v VS4mp46/AgwGQoALd8QKWBgAjSpUAKkU/x2Y5dGiTgFP2x50aaAy7RZYujTc/THu/rnK LbvK8igmnrnKOd5SoNRekzpV5awemxx30rIlu0b5Dfqx81XX279ZldWzBd2OxFsVMbQm RC95tKM9Bk6G3fQaj+DissA7OsyfQQy6SmYqoiTi6VYl5B5hTFKK/5treTR8OeNTkGOv IRyg== X-Gm-Message-State: AOJu0YytWWOFA6pdnRoR3TJzKUT42/ZLdwozHVSv8Lr4Aq+opeplzG7B VOtwk3A1ONpYUtcNJjesxm2naLQuusuDK7lUv6cDS2niwluzWhE3VB1PiA== X-Gm-Gg: ASbGncs6lf6rTPpiLpF4J5HfC3wRJJZ5AG7VjYG+AO651aMmT1HlEcAX0Z0z6kWsyYr w3sNT1uyIRceAfcTTeGeK5VHjuu5Uoy/okzAM7c9+jlpz7WA92AnNEJUNVeORcK0WCRfWJkSTpi yzEhnxhisvrfmKZs7daPzcK1vxNp1UbXtQsetW29+9ye1tTqAYMwPeC+T+LlE0MG9cPzAKXYX7m wJDWk5EC4n9CC6zFNDsRX7qXmZR8pnlLgYbtmfWkCCD+xYjfOzEf3E/W0Xm0Yc= X-Google-Smtp-Source: AGHT+IF6S6c25dScAr6hUeKDZYVyQyU446CpUJC+pjEmxe1svaNtuFSzcIIeLiF5i3Ao4SmfseEwIA== X-Received: by 2002:a05:600c:3c92:b0:42c:b905:2bf9 with SMTP id 5b1f17b1804b1-433ce42883fmr102884415e9.16.1732482725039; Sun, 24 Nov 2024 13:12:05 -0800 (PST) Received: from 127.0.0.1localhost ([85.255.235.224]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-432f643b299sm132733745e9.0.2024.11.24.13.12.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 24 Nov 2024 13:12:04 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v2 04/18] io_uring/memmap: flag regions with user pages Date: Sun, 24 Nov 2024 21:12:21 +0000 Message-ID: <29de51f8d8b328125f647226e76850b21f554423.1732481694.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 In preparation to kernel allocated regions add a flag telling if the region contains user pinned pages or not. Signed-off-by: Pavel Begunkov --- io_uring/memmap.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/io_uring/memmap.c b/io_uring/memmap.c index 21353ea09b39..f76bee5a861a 100644 --- a/io_uring/memmap.c +++ b/io_uring/memmap.c @@ -197,12 +197,16 @@ void *__io_uaddr_map(struct page ***pages, unsigned short *npages, enum { IO_REGION_F_VMAP = 1, + IO_REGION_F_USER_PINNED = 2, }; void io_free_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr) { if (mr->pages) { - unpin_user_pages(mr->pages, mr->nr_pages); + if (mr->flags & IO_REGION_F_USER_PINNED) + unpin_user_pages(mr->pages, mr->nr_pages); + else + release_pages(mr->pages, mr->nr_pages); kvfree(mr->pages); } if ((mr->flags & IO_REGION_F_VMAP) && mr->ptr) @@ -259,7 +263,7 @@ int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, mr->pages = pages; mr->ptr = vptr; mr->nr_pages = nr_pages; - mr->flags |= IO_REGION_F_VMAP; + mr->flags |= IO_REGION_F_VMAP | IO_REGION_F_USER_PINNED; return 0; out_free: if (pages_accounted) From patchwork Sun Nov 24 21:12:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13884193 Received: from mail-wm1-f51.google.com (mail-wm1-f51.google.com [209.85.128.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 98754156960 for ; Sun, 24 Nov 2024 21:12:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482730; cv=none; b=BS7mO2gGkYjYUWh+7AuvmgCCsShllQ4Q7iN5qJ8rSMkl2gw3c1AriKjYrSwMhSmoIeOG15s6drXHXj1StKn8p6dCttRK+CmKOOxkdYTOXcKrx2CjAAuCnA8fFNYkNVR+WfylVBY5YTeyEN7Aunp7YITmi7g6+KiWeZLg/L3fGaA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482730; c=relaxed/simple; bh=A/BG30k4dPSYeGyQ9WWlDhGgvjjruOFJ2EsBN5g0230=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=eHv6Fy2Cn5WJLBwFQQ7HZagiHRCjxa71M+zB6/ElxeQvd++5OKar7++6TiFdLUH8mXCn6xRR31Eyie8vEt+SDWWKzb0PyLpt/4hSIXSnMxrWwW87skiSaHNazU8FRaYtI1oFKrz8fGD8D84Mx7iYOMbjJiUzDkeP6RnTCK5FwZ4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=FgkExJsN; arc=none smtp.client-ip=209.85.128.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="FgkExJsN" Received: by mail-wm1-f51.google.com with SMTP id 5b1f17b1804b1-43152b79d25so34026435e9.1 for ; Sun, 24 Nov 2024 13:12:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732482726; x=1733087526; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5cQnp+lrUnHITWCsKfUZMU+VFdBNN5rUz4oDhu1P1PU=; b=FgkExJsNOiPu6+InZ1AyoMXm4nBYqwFVGiN5/tcppV388zig4UypsP6z9Yl8FnAZAF MZiJcRUXVWHrtwHzasblZgPNuRGSN7w2pOV+xKe2w6W0k/72e0DmX+GwC7TK2XgV+HtH BD6SDRWmce7KE7DEYCee0C+JIdLUci+izY4XVsjc0qDRRclRKnMfSQM20i8jSAsNPfDf 0xDrrg48C+yAVr9VZhGZ7HFkFrppwyq6aJ0M5eZSkUnrtTN9eKAS9a8m2mgZdVxvhPn+ OyOoIRSqHb1XLAhXbZSHg37lH6w/dQEFfzPZSbYtq6nUwDmpk1/xnJzuXMLogw8RR9jo xUrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732482726; x=1733087526; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5cQnp+lrUnHITWCsKfUZMU+VFdBNN5rUz4oDhu1P1PU=; b=j9m0UBuxmQvdFDDjGVU4rkJcgc4gFFyIsAdo3GoiT1lVNvSyNoDdvDBTYqcyadcDNW DO6njE/BVXz3BgtG6q6LHnYgWDQmNv8Unkn62xn/7UKTyCm0fZeGSGxLWldOiJrBsVtQ RiEYvr5Nx7sbaxJIrrOS1fEE7FFMx6JKwUAoEFeOVbn4QtYsh35nX4O2QtqTGZ/5HMQv j8vj+25WarGfHkcatFDi2di6SDJnpLomCGuHpSXyYPbToUAEn0cVJQDy97uFJ2fW/Q2F Iq+aE0WgxTgsdD8QG9QoVqmVAfbnRMD1RSH9WQctE8AaIL8D2DoUvyXku/8drOqRuNF+ 0IwA== X-Gm-Message-State: AOJu0Yxe+niQgfp+dyLB3kPeE3I8cwGetwiLW7uWKvKBy4pUf3aIqNjE iqr+09svBrbX5zWV54Rvo0QUSrv5IOdaY3HJJJXTEJlHyoIMLU/Uyl9Afw== X-Gm-Gg: ASbGncvXq0T7N1OzsfRqQ0m9Sk+NCx0yyLOA7Fvo+vtkGRqCAmJwhcVcMwxzl7COQ4d MAqnTK4PDPWHcc4FIznHoUUclflmd9H04GVS6ScgHjKks/bB2revAtZhPno7fmJyU3Uvrrw+cBf 75je9caXwcobVXQxooGi2fLpdwi2E24VhW0WSjxkpe5o57JEEq/eq4DIFoNo3n4pzAfWU/AwZow LJdKsNdJYTCmWEHHBI8s7Cn7dAYw7wajLC6VcQnT0y0klxSNADkXH1VUHq+KFw= X-Google-Smtp-Source: AGHT+IHiFkzJjuDs4LcWCvAlzsfaVRFGPRDADzkmwlHomkgvLWXqzHuA7E1fgdkW7vZy49e35L+VgQ== X-Received: by 2002:a05:600c:3b19:b0:42c:bae0:f05b with SMTP id 5b1f17b1804b1-433ce413c8emr78182685e9.1.1732482726372; Sun, 24 Nov 2024 13:12:06 -0800 (PST) Received: from 127.0.0.1localhost ([85.255.235.224]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-432f643b299sm132733745e9.0.2024.11.24.13.12.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 24 Nov 2024 13:12:05 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v2 05/18] io_uring/memmap: account memory before pinning Date: Sun, 24 Nov 2024 21:12:22 +0000 Message-ID: <686422cc34b24ffd682abd6443904d18c0d0e06b.1732481694.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Move memory accounting before page pinning. It shouldn't even try to pin pages if it's not allowed, and accounting is also relatively inexpensive. It also give a better code structure as we do generic accounting and then can branch for different mapping types. Signed-off-by: Pavel Begunkov --- io_uring/memmap.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/io_uring/memmap.c b/io_uring/memmap.c index f76bee5a861a..cc5f6f69ee6c 100644 --- a/io_uring/memmap.c +++ b/io_uring/memmap.c @@ -243,17 +243,21 @@ int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, if (check_add_overflow(reg->user_addr, reg->size, &end)) return -EOVERFLOW; - pages = io_pin_pages(reg->user_addr, reg->size, &nr_pages); - if (IS_ERR(pages)) - return PTR_ERR(pages); - + nr_pages = reg->size >> PAGE_SHIFT; if (ctx->user) { ret = __io_account_mem(ctx->user, nr_pages); if (ret) - goto out_free; + return ret; pages_accounted = nr_pages; } + pages = io_pin_pages(reg->user_addr, reg->size, &nr_pages); + if (IS_ERR(pages)) { + ret = PTR_ERR(pages); + pages = NULL; + goto out_free; + } + vptr = vmap(pages, nr_pages, VM_MAP, PAGE_KERNEL); if (!vptr) { ret = -ENOMEM; @@ -268,7 +272,8 @@ int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, out_free: if (pages_accounted) __io_unaccount_mem(ctx->user, pages_accounted); - io_pages_free(&pages, nr_pages); + if (pages) + io_pages_free(&pages, nr_pages); return ret; } From patchwork Sun Nov 24 21:12:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13884194 Received: from mail-wm1-f51.google.com (mail-wm1-f51.google.com [209.85.128.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 85159163 for ; Sun, 24 Nov 2024 21:12:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482731; cv=none; b=eiztTIqIxQS67wHxz3kqFs/pR7DIFdJz9/hbGl2VnCvwziyjShjhcM4nTG/2J+OpNYzEfYKpCwlw6T0WwcjR8GwV6yi0HKntckCr3WTSr2loi82Z0ZY7C1rAC/V78pIPP/yhQuGTguf+2AAKC0mvmkcfYAxGBlvbC0wFVQmTlKo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482731; c=relaxed/simple; bh=dujS0aFvz1kY8S8u3h9P118Wqvzr24FydnRRbGhqzO0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=h5xExTBVLvKxRZmV10EuLZ1rbEs0bLl5Mjju4phzLRLGLmWysxAnrXacvoAX/Qgs349ijBjkxCJ6f4DK3mb+pcF+J1onHnYDmHvKsIZ8R7EdKxfdL5bAC2sYdF/KarYcZ7CT0H8csn4ebbYvquCaHD7Os1JrmNwG9ZDBvp/pCCg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=gaILUWc9; arc=none smtp.client-ip=209.85.128.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="gaILUWc9" Received: by mail-wm1-f51.google.com with SMTP id 5b1f17b1804b1-4349fb56260so1841045e9.3 for ; Sun, 24 Nov 2024 13:12:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732482728; x=1733087528; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sPuIWteBXaKyE6IW5ybcaZa+VOFWSvl/ffKsXytsJvI=; b=gaILUWc9TNCTJiplsHAdyAnKbSt4sV4eF2VuCk/ysvACpuP7NLz0j0e5IGDnS3CuY2 h7XxcGdeQVSGG0DZF/ZBujesFDvTh0w/38ItosgGQSWWDW04gSDBTejnqhmm+f8GBA4j mCT0Xgrm9PwHfEXzyP3/peuUuJFgCcODkH+sjjnA+7UfLRQDhToIeK2T9ryVzwy2ijiG jcGyM6drIqTy4yBrcbwrFNJuUiYywbW7DPb17n/cJU7UKZQ2RgohJa2djE5LxUfyfaiV BEYJIS+etsNNhvXOvucnUEnJzv+d03NX5AMu22MTSto+h8ygXI1pgFg5R1QjvEsKSVfb vayg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732482728; x=1733087528; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sPuIWteBXaKyE6IW5ybcaZa+VOFWSvl/ffKsXytsJvI=; b=CsskT//MgRnbYCOXMEO7H7y7EcHFcZj97AL+r0ZGA/r24CXhGqOihrxLaVdj/H4qkT dk0fXZrZsuwuo0JkhMo8UUvDZrAmUe+bz81/LrIZX8SL3Bmiq9ojJXRBFrp5JT4mpHlH Kd/undUnvmCLyytFZo6xXJTCLZbFfmWcwE2MUZf9DesrC2MrMMXOK/8xreJvOvYvwPNS wI1IDCGC/xvT+Gus3v/8r1ldb0vAPO7TGQDO1pBu1sz6VLqcfygywi8j3fOk7EEgqZ0t kobS/+ruedFBQJ9mshq3iqMC0yxmfdc8S+h+QW0p8j9f5feCQjYQV+Lx8HsdjZ0qMM4+ Z8nQ== X-Gm-Message-State: AOJu0YyN4oWhNIfFv5uHLkenjhUf9yYHgFIi6lCN79qunDlCt5v1LOE0 /ZS3OF5FaiRSgDtEgF2+82KVwPlsZ/0uRYWcTGayQZ9DU+FKpBGzI+51uA== X-Gm-Gg: ASbGncvZUuffkrLNZOuBGadSXtvGNDSohfat+8cTMZYjRpA6wqMI4lWpsJ3FXwD4F4l f5Hnl33XCvtyEN5M7l7HPzQh11haSIxr3drkCsGvcqOoxX5+dVAQWgc2SK9VmoWF2rxDkAcM3mH RTd/KUhPSP1QxEcdG6pvM+EmWDnLLUm75ZUQVk0Y+qS/2cErLri/9IX0Q3gSz3aSuNJPCv97NqR 3TmX1tvTfcKlKojLZsE07MDhg9WUJoDi7781ASnqK8jMgcifPbcALRJDDjnAlg= X-Google-Smtp-Source: AGHT+IHd0iE3BCSRGeO8VjoEfrfhIYbJu4koAcnNoEc5/3Nw0nrDDDydq9jEILXWJooOs+yALvNBfQ== X-Received: by 2002:a05:600d:4:b0:431:5847:f63f with SMTP id 5b1f17b1804b1-433ce426802mr96062305e9.13.1732482727531; Sun, 24 Nov 2024 13:12:07 -0800 (PST) Received: from 127.0.0.1localhost ([85.255.235.224]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-432f643b299sm132733745e9.0.2024.11.24.13.12.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 24 Nov 2024 13:12:07 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v2 06/18] io_uring/memmap: reuse io_free_region for failure path Date: Sun, 24 Nov 2024 21:12:23 +0000 Message-ID: <79927f47b1506508b877607e82af0d6a2c53c293.1732481694.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Regions are going to become more complex with allocation options and optimisations, I want to split initialisation into steps and for that it needs a sane fail path. Reuse io_free_region(), it's smart enough to undo only what's needed and leaves the structure in a consistent state. Signed-off-by: Pavel Begunkov --- io_uring/memmap.c | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/io_uring/memmap.c b/io_uring/memmap.c index cc5f6f69ee6c..2b3cb3fd3fdf 100644 --- a/io_uring/memmap.c +++ b/io_uring/memmap.c @@ -220,7 +220,6 @@ void io_free_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr) int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, struct io_uring_region_desc *reg) { - int pages_accounted = 0; struct page **pages; int nr_pages, ret; void *vptr; @@ -248,32 +247,27 @@ int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, ret = __io_account_mem(ctx->user, nr_pages); if (ret) return ret; - pages_accounted = nr_pages; } + mr->nr_pages = nr_pages; pages = io_pin_pages(reg->user_addr, reg->size, &nr_pages); if (IS_ERR(pages)) { ret = PTR_ERR(pages); - pages = NULL; goto out_free; } + mr->pages = pages; + mr->flags |= IO_REGION_F_USER_PINNED; vptr = vmap(pages, nr_pages, VM_MAP, PAGE_KERNEL); if (!vptr) { ret = -ENOMEM; goto out_free; } - - mr->pages = pages; mr->ptr = vptr; - mr->nr_pages = nr_pages; - mr->flags |= IO_REGION_F_VMAP | IO_REGION_F_USER_PINNED; + mr->flags |= IO_REGION_F_VMAP; return 0; out_free: - if (pages_accounted) - __io_unaccount_mem(ctx->user, pages_accounted); - if (pages) - io_pages_free(&pages, nr_pages); + io_free_region(ctx, mr); return ret; } From patchwork Sun Nov 24 21:12:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13884195 Received: from mail-wm1-f41.google.com (mail-wm1-f41.google.com [209.85.128.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D68E9156960 for ; Sun, 24 Nov 2024 21:12:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482732; cv=none; b=jR2b1S7/Q+oFTQLOW7pylDtR09RKty/w7BI4HVb0Ng8sZXwgyrmeqqBp243iMMTeG0pa4dI7W0VVbLo4r3WCoOg3iyJP8bGIXGv8QZ1tHd9CGk36muAogq6LxGH+4ANcMqilM5tt5vNEYfFIdbbArE5e786AL5+pxoyWhhpQE3A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482732; c=relaxed/simple; bh=55xt3/pYTVPzdu81FVZJ0x8k5BIhUfm79mNIEsz93uw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LA6aazYOUJ1eiQMknHTZRG29KyIacQRyuZUHFGSZPbLgTdRFYNmO0Iw/9GBnCVvO9Zhk5GP4Uwse9n1o52dffs1tKC6Gt+AH986WtFze8kDSkgwxSHWGsogyMGzn4aazrah4pal+WS072OZcSSqNFM+zJ+pdBBjyO+qCYzoU6pY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=TeocTknT; arc=none smtp.client-ip=209.85.128.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="TeocTknT" Received: by mail-wm1-f41.google.com with SMTP id 5b1f17b1804b1-4315f24a6bbso32131865e9.1 for ; Sun, 24 Nov 2024 13:12:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732482729; x=1733087529; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=K+skNG/NMNLar+6yG7jRI9A/owfpoygJmWoUnCIcEc0=; b=TeocTknTO7tlEbMDvKYf6x+Mfr0n+4MQEpisG0Y4J/qxl3qOgSxf0zXofKKEbwW/Mz 7Wf472Xg0Mq2fG5lRC1gIWqx2azCPnlpQu6zQx6XiPKRtyi6eRHzWsyK2kygNNE04xSC eFMwTrmMiRDC83eSWUJ6geD4d80zBjbmhcCR1k7D8cFz4HyoYHsD/eTxfWGEse/oYEhS 6saocPvMa63XRY37Dy4VvxHDxo8VKUxnACgIA2G7P557j2RtaKnMte+hM6vGozQNPZX6 rIq7n6ObZsHE2PuwbAdso+nRLImkyopB/2hkkpoqljUAbK9niEuxoTxddVGa3IeJfhTQ PFJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732482729; x=1733087529; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=K+skNG/NMNLar+6yG7jRI9A/owfpoygJmWoUnCIcEc0=; b=ccroWT8zZR6NBZYcy4TusaoIJRV2zLa75n85KABARp+FUMxOdbQQKfiKs3P3ZnyAEJ t6BkDjRjk1eTHccE1sAnOyLFRkE26wjHWiZhq+o7e9Kolr2neSuBEhunO21Q7kuKnJwi Fb0Qzjcly41r7upV2bQefYq/WW7GNhDcQYAybrWRmjBAcghZjxDFl2bOPUBK9yxDCzkI +HbuFxgmuCMTfA0c8x96qYCeIxPEVNXsAK+9QZXVXJ/Qpl0q/sk/XXxjCCD13OfTfCL1 1RTdpeAoaxmOIL1POoiLFmBYbBcAxmh6Z5C3JDNEOkoLW+HHo2pBerT5zrYqfK5AMyoA bKuQ== X-Gm-Message-State: AOJu0YxfqDZvTHzh+4PVC+wfKgkvkaMUVGjRv+GfFKXQ/yPMN+d5zRnr 4ArXRDHmhYEL68oBUkE6+Dvsw7l0+UlkaXXH8bvK4YncsT0wUDN5IYJrGQ== X-Gm-Gg: ASbGnctesA/HsCfYK7ES1rH1tEF2j4g1Wwk2g/MdyFefvNkTwBLJXJuKeT5dFMRdScc wWTeyFM6Jg4smKtgNJVY/Cz4tHD7fiRxJvSQOfD8W/Cso18/pkTTVPW6ODWZXXcp0PqelbKa+l7 5ElNNuBxeQLI9DiioP2wpt/cerAnY9OzT9/W3zoBcHlmWsJTfT5r3mRJ15ZmERttC7CKSxAoPnI 5cYe94mgqb3Wzx853lZtKSutVjdWH1kTZuThUfb1PrbUEBzW3x9nwrEaGsEhi8= X-Google-Smtp-Source: AGHT+IEqbXpV1lmNv5WfvqmcWc7iyx+9ha6sC30QFtb3kghFEjd9q0w0j1CPQMLKMoXO8dsJwfOHQg== X-Received: by 2002:a05:600c:1c09:b0:42f:4f6:f8f3 with SMTP id 5b1f17b1804b1-433ce420b21mr87039705e9.7.1732482728954; Sun, 24 Nov 2024 13:12:08 -0800 (PST) Received: from 127.0.0.1localhost ([85.255.235.224]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-432f643b299sm132733745e9.0.2024.11.24.13.12.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 24 Nov 2024 13:12:08 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v2 07/18] io_uring/memmap: optimise single folio regions Date: Sun, 24 Nov 2024 21:12:24 +0000 Message-ID: <398023f58961f57ae2c4974ef6327f5572bf8e73.1732481694.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 We don't need to vmap if memory is already physically contiguous. There are two important cases it covers: PAGE_SIZE regions and huge pages. Use io_check_coalesce_buffer() to get the number of contiguous folios. Signed-off-by: Pavel Begunkov --- io_uring/memmap.c | 29 ++++++++++++++++++++++------- 1 file changed, 22 insertions(+), 7 deletions(-) diff --git a/io_uring/memmap.c b/io_uring/memmap.c index 2b3cb3fd3fdf..32d2a39aff02 100644 --- a/io_uring/memmap.c +++ b/io_uring/memmap.c @@ -217,12 +217,31 @@ void io_free_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr) memset(mr, 0, sizeof(*mr)); } +static int io_region_init_ptr(struct io_mapped_region *mr) +{ + struct io_imu_folio_data ifd; + void *ptr; + + if (io_check_coalesce_buffer(mr->pages, mr->nr_pages, &ifd)) { + if (ifd.nr_folios == 1) { + mr->ptr = page_address(mr->pages[0]); + return 0; + } + } + ptr = vmap(mr->pages, mr->nr_pages, VM_MAP, PAGE_KERNEL); + if (!ptr) + return -ENOMEM; + + mr->ptr = ptr; + mr->flags |= IO_REGION_F_VMAP; + return 0; +} + int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, struct io_uring_region_desc *reg) { struct page **pages; int nr_pages, ret; - void *vptr; u64 end; if (WARN_ON_ONCE(mr->pages || mr->ptr || mr->nr_pages)) @@ -258,13 +277,9 @@ int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, mr->pages = pages; mr->flags |= IO_REGION_F_USER_PINNED; - vptr = vmap(pages, nr_pages, VM_MAP, PAGE_KERNEL); - if (!vptr) { - ret = -ENOMEM; + ret = io_region_init_ptr(mr); + if (ret) goto out_free; - } - mr->ptr = vptr; - mr->flags |= IO_REGION_F_VMAP; return 0; out_free: io_free_region(ctx, mr); From patchwork Sun Nov 24 21:12:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13884196 Received: from mail-wm1-f48.google.com (mail-wm1-f48.google.com [209.85.128.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5D0F115B97D for ; Sun, 24 Nov 2024 21:12:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482733; cv=none; b=U3yrD0aIj/NcDKXcL+seMX0Db60mwLbCT1RijMRTRvrMghojpE/wceG1ZvMYhwUrvm4ElvqjjXEhxfrWtYQIdseUAk6FwtVhYkw2o2mKhO8ivgF1gajLFSHVpK9SMNBTMmWS2M68JVcUrV6UZ9DNB9g41K8SQxrg6tr/0kaGON0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482733; c=relaxed/simple; bh=AOfi6x19gzyu/5rrZe/6KOA7e2hddpGJfDp/n7HaVeA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=W8KxXKqqYZIBjk1HINF5QwE3D/ITCBULVcA3wp+TNdyO0/MpIntHITNKaiAIlBoLCDowkdPmp3VlGqfP7wZ12aVX2f+3QyaDmyfB64QrKIFuqoN1xNnfyq6j/DaS9DZqziUFTJfx889B3V9YVRYZlDk+le5kLSar1qCi3ly9d7E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=GSKAMIOi; arc=none smtp.client-ip=209.85.128.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="GSKAMIOi" Received: by mail-wm1-f48.google.com with SMTP id 5b1f17b1804b1-43162cf1eaaso48596845e9.0 for ; Sun, 24 Nov 2024 13:12:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732482730; x=1733087530; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cKExDGox31CnOmhkyqcAp7yYwe/A8AEg0kEs/PZerJw=; b=GSKAMIOiExYOQUqVwt5muf1F7Me3oR6Jxm5kMo6c2rWZr5kcVNTaSthCFzG9OFgHP+ 9dcTUzxCtSyKE39NGODG1L38fnZi8rjFV5CmDpUzjrlQHpn8F2IgGq1ksWvx5cJAksPz D8YS/ozbgD6eVz3mXxnCyew7mN9s0dPNOrptwhpICCvoS7aNu0Asmx5dwAoC+kbt3WdQ XL+PvGnv+DimvW8E/XtmGkRAdkrCaZBK048XZpuH5SwQNLxiR0clZkE+jhSJXQ9b4GbL sISGh+4nFRbcC6yy8avk+mb4M8e24hUV+6pgqQhdAj0V7YoTRc76lLDlSg1WGYN0xIJy QBag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732482730; x=1733087530; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cKExDGox31CnOmhkyqcAp7yYwe/A8AEg0kEs/PZerJw=; b=kPzCXtAkWKvQVakB1BNsaLLWnqQqnkxISKJagn7NXShqxizXgLZ+yFytH1yBoHr4EV 6h/+E7Tw8uM3XOXMr+M74/IW3J+Mmy366k+7jAUmh9aNCkW2MqG7U7qwpFryigqEYCdy wVa7T7espWgmNJfLdzupX5qv43zLCt/KDZiyhqtvzEUZtk/D5H+sLv2v6umltBLTmwgz PK17LYYpZpck92uaf7FNTXhEJqeJG6E6RVZVvNZpYE42ymQ47vJp03ffPlfmpAXb12im Ua3nkX9nZM30UiF61D8TcbJ5EtC0aR7yS/ixpx+lWt2Zi5Nas5YYgIzPHRsE1gv3Y6mj tbAA== X-Gm-Message-State: AOJu0Yxc3N0+4Fl57icvhZBVtADBqPViLpayQ0lDgO3dIWtapl00RgW9 JzJejS6lHCms5v/V6fS7i7WUON+7rzcjfB46jM++F3slrIHUV9AO5sFZPA== X-Gm-Gg: ASbGncvtLEgxjOiH5A94rOO2X4y87HTbww4r2Njkn+iwZWTT5dl9yoFlYwKWxvIHcZD QY13OFZ7V+jMFvs2iBA0HiZqCthjR2n+mXB/d2pmDSwkOF1cxZRMULyRM09GslcSEwiikObLlfx Y3MXtBJLTqv1+DXY8jXnY4nVx3lrZbTvJMa3tnBgnun117NHXsn6uvUBOj4cflFwPSGbbG+U5HG N8cbA1nrailEVJImUHs+IPPPnl+wM6iM24Q/OOyY59U4jPZjqtKrdJXGH6yesI= X-Google-Smtp-Source: AGHT+IGtPe5Wu0SEuXNZJQ3EQyZ6browCpeZ5VDV+UdyUpZJD/TTNW0HegwjKG/dWYajR847Oexn2g== X-Received: by 2002:a05:600c:3ac3:b0:431:4b88:d407 with SMTP id 5b1f17b1804b1-433ce410255mr105545035e9.5.1732482730126; Sun, 24 Nov 2024 13:12:10 -0800 (PST) Received: from 127.0.0.1localhost ([85.255.235.224]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-432f643b299sm132733745e9.0.2024.11.24.13.12.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 24 Nov 2024 13:12:09 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v2 08/18] io_uring/memmap: helper for pinning region pages Date: Sun, 24 Nov 2024 21:12:25 +0000 Message-ID: <694614a8654a16d846439cebe666f1e04d4dd15e.1732481694.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 In preparation to adding kernel allocated regions extract a new helper that pins user pages. Signed-off-by: Pavel Begunkov --- io_uring/memmap.c | 29 +++++++++++++++++++++-------- 1 file changed, 21 insertions(+), 8 deletions(-) diff --git a/io_uring/memmap.c b/io_uring/memmap.c index 32d2a39aff02..15fefbed77ec 100644 --- a/io_uring/memmap.c +++ b/io_uring/memmap.c @@ -237,10 +237,28 @@ static int io_region_init_ptr(struct io_mapped_region *mr) return 0; } +static int io_region_pin_pages(struct io_ring_ctx *ctx, + struct io_mapped_region *mr, + struct io_uring_region_desc *reg) +{ + unsigned long size = mr->nr_pages << PAGE_SHIFT; + struct page **pages; + int nr_pages; + + pages = io_pin_pages(reg->user_addr, size, &nr_pages); + if (IS_ERR(pages)) + return PTR_ERR(pages); + if (WARN_ON_ONCE(nr_pages != mr->nr_pages)) + return -EFAULT; + + mr->pages = pages; + mr->flags |= IO_REGION_F_USER_PINNED; + return 0; +} + int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, struct io_uring_region_desc *reg) { - struct page **pages; int nr_pages, ret; u64 end; @@ -269,14 +287,9 @@ int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, } mr->nr_pages = nr_pages; - pages = io_pin_pages(reg->user_addr, reg->size, &nr_pages); - if (IS_ERR(pages)) { - ret = PTR_ERR(pages); + ret = io_region_pin_pages(ctx, mr, reg); + if (ret) goto out_free; - } - mr->pages = pages; - mr->flags |= IO_REGION_F_USER_PINNED; - ret = io_region_init_ptr(mr); if (ret) goto out_free; From patchwork Sun Nov 24 21:12:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13884197 Received: from mail-wm1-f45.google.com (mail-wm1-f45.google.com [209.85.128.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3572913FD72 for ; Sun, 24 Nov 2024 21:12:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482734; cv=none; b=ktfOw3Fk0sA+3LUKtYwz6psTqRtGsfTjExQhhbY4V4MaDT3Qsq9trix+KjAsbb7Ac7Tz+fQ5M/G4RZVs8FlZF376vUaOxc3ypUZJ4Yyzzs7XKw209RqxWiV5kHHM0FiNVkfHpuR/II3coH0bd9VNmom2uiCrxrbIs9Wy74A6AaU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482734; c=relaxed/simple; bh=tZdEZvN/5hCLNJpMtjgBHfxWGrgUbt1UcPId78L2bOo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DHoQFdN+pHWfxSeYJoAGB828NGttTOCaeGvjTDSwwfDf0gY04fQOrAA9gEI9GWBrTrEOMGeeo6I2BEHmRUWD8mT+T13kiYlivFYs2d+J+6KmVIVXOf0t2IVWe1D1oF1qdWyCyhzThRquf+tg6KdyAdNKr9/BeCcO7af61T4iTjY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=PICifKIE; arc=none smtp.client-ip=209.85.128.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="PICifKIE" Received: by mail-wm1-f45.google.com with SMTP id 5b1f17b1804b1-4314fa33a35so33370585e9.1 for ; Sun, 24 Nov 2024 13:12:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732482731; x=1733087531; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TOubji84NJuUbvOMUGscG3na9pQtpTrNDF4xMRHuW3Y=; b=PICifKIEfHBYoWpHOtjNx/39/7CmsS8pv4IWucjt/zDJ/DSBolgQU3f58jYbt5xEnF gCliLy6SCt3tAs0/FEBEpUHKnKrpajxsFA4uhMRVsxn6hXSePHpmG0ln6hOFHtUI2oss mKF25Vc0y7CTaC4+Q8IHN5Kef0Ys5r8h1OJL8XKlVYDJZDSIlvGXhSs6iwMv4stRcicJ rxebK/WSHQZ/x0/oiY4zojkv9nbfS8SfkTLrxyIe8zPwhtSvVKEAdkglrK4LktuBSRFo 3i1TJhYRI2SBAafBcMk2eO50qRzLBHG+TOJOGvEZzs41jwkWb19422sShwEpDAdDZ39n yGiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732482731; x=1733087531; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TOubji84NJuUbvOMUGscG3na9pQtpTrNDF4xMRHuW3Y=; b=IL1825RWUQNeqbiJ8+cgOSxlh0HsmcWSBFHmQKRP7ikzOzH62O9sEG0mTFvjQbbzS0 XGnIQQp7ksyXnoX0VcPcdi4xgHZ8yP5YlhCYamUT2cY8K+ilvC5J6obWy8RUUgpEzImr 83K4oA1JLIBNwiXh+Tp6Mh9vSewpqeGniZ+k5bgR7z0r/oqoCWZ0pw/az/9i+F8/AWLG w0Ik/WiZof7hBJ5NGRRYsxKRcQMVO3cC7vxfS+y9XgRvmlzxng6izEf7eeT5Z2C3bCan 5jtz0bj3VHZauXSk6bL02V3sHY3TzR1ds64hUkJLban8Vj5vYwuXiuroHZUNsxoZGsl3 tWEA== X-Gm-Message-State: AOJu0Yydj6mPGpdkJqJHnClD87dwFBc2WtTErGb/Lb9Ml9tCWN60/aGE YPlbvrKWItbB4dC5ZE0XweLh6aX05U1r8e1OuCi1UlV88ejqjGHi4XlgyA== X-Gm-Gg: ASbGncsGQ5VP8EFzcYuGAYAw3xMPOJIDeJ0MAMGgPb0EyZJqzRiWQaWdr75e7hYnnOG HqXUv02uL5FcTfHe0JYQvZIkcDDtihg0z/+iR4f/xlvIh5LLopoemHnQPnvJQ6UIcGDhzxfJ0qe cSsiL51kVf3e+QTx7BC5zX2VCnEpDbVAtPiAA9RVCF4coVPMRfP56/FMrdAfQmmwUDfQZoRXpRU A2dhIt8Dv92CjBx5xHdnVzRkkHH9+T46atj9UKzb8YFr7TqQohMHij6XWeb30c= X-Google-Smtp-Source: AGHT+IFWp2F53oDb1JcXByDkaHiptaVWMLdUzz/9JX1ENXFMbofQJTaIEoNRRLrj7q7ApQEJHIVxcA== X-Received: by 2002:a7b:c7da:0:b0:434:9499:9e87 with SMTP id 5b1f17b1804b1-4349499a073mr35372655e9.25.1732482731400; Sun, 24 Nov 2024 13:12:11 -0800 (PST) Received: from 127.0.0.1localhost ([85.255.235.224]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-432f643b299sm132733745e9.0.2024.11.24.13.12.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 24 Nov 2024 13:12:10 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v2 09/18] io_uring/memmap: add IO_REGION_F_SINGLE_REF Date: Sun, 24 Nov 2024 21:12:26 +0000 Message-ID: X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Kernel allocated compound pages will have just one reference for the entire page array, add a flag telling io_free_region about that. Signed-off-by: Pavel Begunkov --- io_uring/memmap.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/io_uring/memmap.c b/io_uring/memmap.c index 15fefbed77ec..cdd620bdd3ee 100644 --- a/io_uring/memmap.c +++ b/io_uring/memmap.c @@ -198,15 +198,22 @@ void *__io_uaddr_map(struct page ***pages, unsigned short *npages, enum { IO_REGION_F_VMAP = 1, IO_REGION_F_USER_PINNED = 2, + IO_REGION_F_SINGLE_REF = 4, }; void io_free_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr) { if (mr->pages) { + long nr_pages = mr->nr_pages; + + if (mr->flags & IO_REGION_F_SINGLE_REF) + nr_pages = 1; + if (mr->flags & IO_REGION_F_USER_PINNED) - unpin_user_pages(mr->pages, mr->nr_pages); + unpin_user_pages(mr->pages, nr_pages); else - release_pages(mr->pages, mr->nr_pages); + release_pages(mr->pages, nr_pages); + kvfree(mr->pages); } if ((mr->flags & IO_REGION_F_VMAP) && mr->ptr) From patchwork Sun Nov 24 21:12:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13884198 Received: from mail-wm1-f53.google.com (mail-wm1-f53.google.com [209.85.128.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 985E215A848 for ; Sun, 24 Nov 2024 21:12:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482736; cv=none; b=G0+GzKVgVRgS74R5+TRd1myq9M5KCxyuFvVrgXgGUkFqwuFzxElCyuSU07/E2y0q6gUVPo3KHhSrvxAwdh0bFU1jBBYP/x0xGc1u/9AvUptyEr28OYfyddgDbg5bbccVk/IvqNq5R/Y864hrYLJK3c4ULF8FDp5qz+71Uc3Rc9M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482736; c=relaxed/simple; bh=Ojbg/pyG/fWFfozOjF7t6/LoeMXwPm8VkT4/RWIuwuo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YVieGsuMQWhBWB/fjb/e9pDSsxgc/VdFFexu70Fj4pJrycNMWvTGd5eAA4goWRtuzoylAZhkx5w4uT99djXijFLTNkhN8U8N2At0B3paLbMYr7iVItT9oOu8B/0WnGLul2gJeG7vVkjd0kCMbRSZINelzW2yWZNhJ67XDqjkQ7I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=eKhu1FoK; arc=none smtp.client-ip=209.85.128.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="eKhu1FoK" Received: by mail-wm1-f53.google.com with SMTP id 5b1f17b1804b1-4315f24a6bbso32132115e9.1 for ; Sun, 24 Nov 2024 13:12:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732482733; x=1733087533; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HuAPtcJgL3o5hKcE460Eafo6KlyiNWFiJdD3Q2BBUe8=; b=eKhu1FoKQVQdeWiIU3sLyvDjX6fA8ZW1KySaPy9mvyGr1cQikg3mhM2ZdrKvtJd6mh GXsMumbh+npJM633ZOt0Sc1rdjzejQsb4908r2oh59fDOVdCrfNHLHa/fGlTRHJ+g+XN rSUj+Qlx8HhUP9BFjnV0xqkKzUsx2zVp26HQ+bNIDFUozQVEcOYGBlA/xjckrK+VJWJT YfwQRAiYfY9BEdZOCADS4/FaUiIYcHEPxvcmx6St+qYlBl35JcZY21FgSxTKUdwnEUza 8Rb0HM5aGlD1djm7TEECrsu42eBLxd0Jma3w8eHKIVgNzlgc7cdv3rHdEp6ick0h4cAl UvjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732482733; x=1733087533; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HuAPtcJgL3o5hKcE460Eafo6KlyiNWFiJdD3Q2BBUe8=; b=q1Dbh+z34X+t94CHlqXMEw2PuTOOi0Y4bcVjboc0nwVIJG1dCe9t2fLVrQ2BND7Eve 9SJdOPg/xwsWU5sSVGyZKDkPPPdItxMRGrTMFR0qIOqbodUopY4ehddlFLMDFuQFOZhO plnqSrPMvxUdMtdRcT6cXXVt2zkyUjEcffZtynp55hc9Gdk+OMYW0IvwLcsBBZ15u/4C GzGUimMXGJnGxlndysj5mED9LKb4zNrvmMidnSdnTbW8Ywqek8dZn0eBo2XqWCQKlMt8 uWD49Kx90tUm6WEdhcUV16PrHRoc8M0c+xQiPG280PrHmgWEpv1w/Xm2APeO15FIAtvu rVJQ== X-Gm-Message-State: AOJu0YzQQmPZk6jPmYjIB78V8d1sJ77rxd567maZbWS5BaL0Xr3qYPgx POOUl3i4x7Zk+bnoI4Lf9mUPcmf2ehpO/hTFkOmErPUjewVPueRYg/D1Cg== X-Gm-Gg: ASbGncs7AE4lJm4iGEwy5k3gN9rFX0KlxZX4XLax4T8Xvv3vtb4UyOlqfwk+TuJ6Akp uavGLt5mBZiDnDK1KERI8Kb1XGBYI0XRdD8+y9t5dz64CXeZ/yqy0nFHHO5eedLQGpGGEzbXpGy BsOJSbl9LWI/FlakyWkUCVrzSK3iLeSNdL1Cu3nEYhQw/mVdn2bQ0LV8tOHGrH9EFoddXneTwjx mLiEIwmzHIst7dZd9bU6p44GkMa96a7AzXIEysbPPJoZsKYfzKW1MvSt8tWxDg= X-Google-Smtp-Source: AGHT+IGAj+cll/siFaAuCJRudSw1QW7TbwmjdYDBUMr4tX9aschVIPIEmUX8r8/ee0KyWnazjGFGSQ== X-Received: by 2002:a05:600c:1d0f:b0:430:54a4:5b03 with SMTP id 5b1f17b1804b1-433ce420a72mr83342015e9.6.1732482732703; Sun, 24 Nov 2024 13:12:12 -0800 (PST) Received: from 127.0.0.1localhost ([85.255.235.224]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-432f643b299sm132733745e9.0.2024.11.24.13.12.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 24 Nov 2024 13:12:12 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v2 10/18] io_uring/memmap: implement kernel allocated regions Date: Sun, 24 Nov 2024 21:12:27 +0000 Message-ID: X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Allow the kernel to allocate memory for a region. That's the classical way SQ/CQ are allocated. It's not yet useful to user space as there is no way to mmap it, which is why it's explicitly disabled in io_register_mem_region(). Signed-off-by: Pavel Begunkov --- io_uring/memmap.c | 44 +++++++++++++++++++++++++++++++++++++++++--- io_uring/register.c | 2 ++ 2 files changed, 43 insertions(+), 3 deletions(-) diff --git a/io_uring/memmap.c b/io_uring/memmap.c index cdd620bdd3ee..8598770bc385 100644 --- a/io_uring/memmap.c +++ b/io_uring/memmap.c @@ -263,6 +263,39 @@ static int io_region_pin_pages(struct io_ring_ctx *ctx, return 0; } +static int io_region_allocate_pages(struct io_ring_ctx *ctx, + struct io_mapped_region *mr, + struct io_uring_region_desc *reg) +{ + gfp_t gfp = GFP_KERNEL_ACCOUNT | __GFP_ZERO | __GFP_NOWARN; + unsigned long size = mr->nr_pages << PAGE_SHIFT; + unsigned long nr_allocated; + struct page **pages; + void *p; + + pages = kvmalloc_array(mr->nr_pages, sizeof(*pages), gfp); + if (!pages) + return -ENOMEM; + + p = io_mem_alloc_compound(pages, mr->nr_pages, size, gfp); + if (!IS_ERR(p)) { + mr->flags |= IO_REGION_F_SINGLE_REF; + mr->pages = pages; + return 0; + } + + nr_allocated = alloc_pages_bulk_noprof(gfp, numa_node_id(), NULL, + mr->nr_pages, NULL, pages); + if (nr_allocated != mr->nr_pages) { + if (nr_allocated) + release_pages(pages, nr_allocated); + kvfree(pages); + return -ENOMEM; + } + mr->pages = pages; + return 0; +} + int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, struct io_uring_region_desc *reg) { @@ -273,9 +306,10 @@ int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, return -EFAULT; if (memchr_inv(®->__resv, 0, sizeof(reg->__resv))) return -EINVAL; - if (reg->flags != IORING_MEM_REGION_TYPE_USER) + if (reg->flags & ~IORING_MEM_REGION_TYPE_USER) return -EINVAL; - if (!reg->user_addr) + /* user_addr should be set IFF it's a user memory backed region */ + if ((reg->flags & IORING_MEM_REGION_TYPE_USER) != !!reg->user_addr) return -EFAULT; if (!reg->size || reg->mmap_offset || reg->id) return -EINVAL; @@ -294,9 +328,13 @@ int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, } mr->nr_pages = nr_pages; - ret = io_region_pin_pages(ctx, mr, reg); + if (reg->flags & IORING_MEM_REGION_TYPE_USER) + ret = io_region_pin_pages(ctx, mr, reg); + else + ret = io_region_allocate_pages(ctx, mr, reg); if (ret) goto out_free; + ret = io_region_init_ptr(mr); if (ret) goto out_free; diff --git a/io_uring/register.c b/io_uring/register.c index ba61697d7a53..f043d3f6b026 100644 --- a/io_uring/register.c +++ b/io_uring/register.c @@ -586,6 +586,8 @@ static int io_register_mem_region(struct io_ring_ctx *ctx, void __user *uarg) if (copy_from_user(&rd, rd_uptr, sizeof(rd))) return -EFAULT; + if (!(rd.flags & IORING_MEM_REGION_TYPE_USER)) + return -EINVAL; if (memchr_inv(®.__resv, 0, sizeof(reg.__resv))) return -EINVAL; if (reg.flags & ~IORING_MEM_REGION_REG_WAIT_ARG) From patchwork Sun Nov 24 21:12:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13884199 Received: from mail-wm1-f42.google.com (mail-wm1-f42.google.com [209.85.128.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8B865156960 for ; Sun, 24 Nov 2024 21:12:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482738; cv=none; b=k8xV/kmO8smcN9aSoPNGacg3t6dCGo/+hfi5T49zvjrUTKLuGYqIh7r+kFhybj9TDnxR7TzM6G9bOiSkK6cmaxjNrlnwbcsJnlL4V76KzUF5x0e1xEIIuc6cnQeRXE68rlfJ6+YdR2zt9itzaw7AmJGFUuPqUQwZy7CSEgl69Z0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482738; c=relaxed/simple; bh=n3OStU6FcBGysp3Pci4+yNu5CjV7dxL2Qx1q/VYalHo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hgXv0xkViKw/mUUdi2i/ILe/kLBHJvLxJaSXNX/R0nw8CcrhH+lnogaUWXJK5LQ4GzBICoQPa0xoqYv8iHub/DZSou9tusi6LZd0pIp79ad3aDLsSqvk86G1UYIKeL5LrpF+/OfEW0JBs+TMWCdOptp04LOMarB8FjlcvuLBCf0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=cMA/6BBV; arc=none smtp.client-ip=209.85.128.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="cMA/6BBV" Received: by mail-wm1-f42.google.com with SMTP id 5b1f17b1804b1-43152b79d25so34027035e9.1 for ; Sun, 24 Nov 2024 13:12:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732482735; x=1733087535; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=304V2YUK9klc+k7M6AeMClL+WGzyroOEloK7wWLTT48=; b=cMA/6BBVpyWJ3koLN/6MYjvnROqi0dZPgXf/Lo7Y6FNi1FwjdVpaR/PsMhqdkJTMyr 7kCG04cz0kVPeNrnt4JndErsnren90jzwRCYhtgL5poBlTRWppxHH1jHXR70gHinVXmu lN8e6Gpox+tLCjTWczynXZBPETaKzD6yg+sUVbcj7Fvmq376Jw9yhjU3geNv6IO1Ths2 Q1DAzZvqRda/kVRmqb3vp/C/oXByjWoRpVuoUI67R2pKpXB3jYBBBOY3dhWeX+YBGGtL wrhQWezpUPeDM7sqad/hBjLT9OnXn4RIybNDagBu8kaZPkg9Q4KIlE1MwPvmvl/dltBa cd+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732482735; x=1733087535; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=304V2YUK9klc+k7M6AeMClL+WGzyroOEloK7wWLTT48=; b=qcH0ZTTT8QvTHnaHATaNP/ZUFu1WbjHAyWF+4kjefY5I1LOSIhXtM9c2tavc2oyaQ8 9cCMFYzCPHZNPdVb5NeXIIjuOClIzcRHb8L1E6CIAyui6RA2VQ+QNQ5aqzTrLqLP4gvh 3TmQYhgn1AdTarq2Q4B+np6bvbgvqmdTNcLYMKncEHjpeTfwFz89silKkBYDdDaA0DSI eObgkTF9S/nffKZyhkkvbjOkz/rlyi5OZrb9XiJdiq7FeRgRDF6Cj6GBE3875+YSWFl0 cWrUpwYMoi1o23HNeuzO1/W8IWv/ddq/4698nglTeSu+RwC1PpHbYgJ+ajLTKT20T+k1 3kvw== X-Gm-Message-State: AOJu0YyCoWm7O1E3P6yJcfdwd+Yq47A3WVjLnPaCE5jT7D61IKyJr+pG 0AiynZMn6K7KhITQY6KrwsiaNBJcMNe1mVwOn9ls7654wEdi8T73Tw+9rQ== X-Gm-Gg: ASbGncsFwZKGLVgxcgqXLHzI4gwMXPNOKYc3DxuYciD1bwAPE2FCrdFxAyBb0XtLHQ1 /lsauIXecRBHm6zc8guD6TEh+FzhiXakssjGMfJKYyTa2zp798GQ7VOkPy5U9NJAq7/LRNe9MrL 7T0QFmSFXEmtx06WdTDajAFZQLqvib88C3qOs23nIsPZBIMF/pXdat+LXe1BJgowbtGr6v0FDG+ 17LytO/otff439lC+f/Is1yi6pg/7S+wWYVVjiiNDV1b37KRM2l0iPXcjHNOXk= X-Google-Smtp-Source: AGHT+IFGr0NARd5uhvzSz3qATcINtrjS2yg3pgA6tYvaNhfIG6LUj5HZiVDams2fqUAccjDTQBrCOQ== X-Received: by 2002:a05:600c:5254:b0:431:58b3:affa with SMTP id 5b1f17b1804b1-433ce42488fmr96086725e9.9.1732482734447; Sun, 24 Nov 2024 13:12:14 -0800 (PST) Received: from 127.0.0.1localhost ([85.255.235.224]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-432f643b299sm132733745e9.0.2024.11.24.13.12.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 24 Nov 2024 13:12:14 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v2 11/18] io_uring/memmap: implement mmap for regions Date: Sun, 24 Nov 2024 21:12:28 +0000 Message-ID: <367482d02a9a78861c9da43be8373a26b585eac6.1732481694.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 The patch implements mmap for the param region and enables the kernel allocation mode. Internally it uses a fixed mmap offset, however the user has to use the offset returned in struct io_uring_region_desc::mmap_offset. Note, mmap doesn't and can't take ->uring_lock and the region / ring lookup is protected by ->mmap_lock, and it's directly peeking at ctx->param_region. We can't protect io_create_region() with the mmap_lock as it'd deadlock, which is why io_create_region_mmap_safe() initialises it for us in a temporary variable and then publishes it with the lock taken. It's intentionally decoupled from main region helpers, and in the future we might want to have a list of active regions, which then could be protected by the ->mmap_lock. Signed-off-by: Pavel Begunkov --- io_uring/memmap.c | 61 +++++++++++++++++++++++++++++++++++++++++---- io_uring/memmap.h | 10 +++++++- io_uring/register.c | 6 ++--- 3 files changed, 67 insertions(+), 10 deletions(-) diff --git a/io_uring/memmap.c b/io_uring/memmap.c index 8598770bc385..5d971ba33d5a 100644 --- a/io_uring/memmap.c +++ b/io_uring/memmap.c @@ -265,7 +265,8 @@ static int io_region_pin_pages(struct io_ring_ctx *ctx, static int io_region_allocate_pages(struct io_ring_ctx *ctx, struct io_mapped_region *mr, - struct io_uring_region_desc *reg) + struct io_uring_region_desc *reg, + unsigned long mmap_offset) { gfp_t gfp = GFP_KERNEL_ACCOUNT | __GFP_ZERO | __GFP_NOWARN; unsigned long size = mr->nr_pages << PAGE_SHIFT; @@ -280,8 +281,7 @@ static int io_region_allocate_pages(struct io_ring_ctx *ctx, p = io_mem_alloc_compound(pages, mr->nr_pages, size, gfp); if (!IS_ERR(p)) { mr->flags |= IO_REGION_F_SINGLE_REF; - mr->pages = pages; - return 0; + goto done; } nr_allocated = alloc_pages_bulk_noprof(gfp, numa_node_id(), NULL, @@ -292,12 +292,15 @@ static int io_region_allocate_pages(struct io_ring_ctx *ctx, kvfree(pages); return -ENOMEM; } +done: + reg->mmap_offset = mmap_offset; mr->pages = pages; return 0; } int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, - struct io_uring_region_desc *reg) + struct io_uring_region_desc *reg, + unsigned long mmap_offset) { int nr_pages, ret; u64 end; @@ -331,7 +334,7 @@ int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, if (reg->flags & IORING_MEM_REGION_TYPE_USER) ret = io_region_pin_pages(ctx, mr, reg); else - ret = io_region_allocate_pages(ctx, mr, reg); + ret = io_region_allocate_pages(ctx, mr, reg, mmap_offset); if (ret) goto out_free; @@ -344,6 +347,50 @@ int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, return ret; } +int io_create_region_mmap_safe(struct io_ring_ctx *ctx, struct io_mapped_region *mr, + struct io_uring_region_desc *reg, + unsigned long mmap_offset) +{ + struct io_mapped_region tmp_mr; + int ret; + + memcpy(&tmp_mr, mr, sizeof(tmp_mr)); + ret = io_create_region(ctx, &tmp_mr, reg, mmap_offset); + if (ret) + return ret; + + /* + * Once published mmap can find it without holding only the ->mmap_lock + * and not ->uring_lock. + */ + guard(mutex)(&ctx->mmap_lock); + memcpy(mr, &tmp_mr, sizeof(tmp_mr)); + return 0; +} + +static void *io_region_validate_mmap(struct io_ring_ctx *ctx, + struct io_mapped_region *mr) +{ + lockdep_assert_held(&ctx->mmap_lock); + + if (!io_region_is_set(mr)) + return ERR_PTR(-EINVAL); + if (mr->flags & IO_REGION_F_USER_PINNED) + return ERR_PTR(-EINVAL); + + return io_region_get_ptr(mr); +} + +static int io_region_mmap(struct io_ring_ctx *ctx, + struct io_mapped_region *mr, + struct vm_area_struct *vma) +{ + unsigned long nr_pages = mr->nr_pages; + + vm_flags_set(vma, VM_DONTEXPAND); + return vm_insert_pages(vma, vma->vm_start, mr->pages, &nr_pages); +} + static void *io_uring_validate_mmap_request(struct file *file, loff_t pgoff, size_t sz) { @@ -379,6 +426,8 @@ static void *io_uring_validate_mmap_request(struct file *file, loff_t pgoff, io_put_bl(ctx, bl); return ptr; } + case IORING_MAP_OFF_PARAM_REGION: + return io_region_validate_mmap(ctx, &ctx->param_region); } return ERR_PTR(-EINVAL); @@ -419,6 +468,8 @@ __cold int io_uring_mmap(struct file *file, struct vm_area_struct *vma) ctx->n_sqe_pages); case IORING_OFF_PBUF_RING: return io_pbuf_mmap(file, vma); + case IORING_MAP_OFF_PARAM_REGION: + return io_region_mmap(ctx, &ctx->param_region, vma); } return -EINVAL; diff --git a/io_uring/memmap.h b/io_uring/memmap.h index 2096a8427277..2402bca3d700 100644 --- a/io_uring/memmap.h +++ b/io_uring/memmap.h @@ -1,6 +1,8 @@ #ifndef IO_URING_MEMMAP_H #define IO_URING_MEMMAP_H +#define IORING_MAP_OFF_PARAM_REGION 0x20000000ULL + struct page **io_pin_pages(unsigned long ubuf, unsigned long len, int *npages); void io_pages_free(struct page ***pages, int npages); int io_uring_mmap_pages(struct io_ring_ctx *ctx, struct vm_area_struct *vma, @@ -24,7 +26,13 @@ int io_uring_mmap(struct file *file, struct vm_area_struct *vma); void io_free_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr); int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, - struct io_uring_region_desc *reg); + struct io_uring_region_desc *reg, + unsigned long mmap_offset); + +int io_create_region_mmap_safe(struct io_ring_ctx *ctx, + struct io_mapped_region *mr, + struct io_uring_region_desc *reg, + unsigned long mmap_offset); static inline void *io_region_get_ptr(struct io_mapped_region *mr) { diff --git a/io_uring/register.c b/io_uring/register.c index f043d3f6b026..5b099ec36d00 100644 --- a/io_uring/register.c +++ b/io_uring/register.c @@ -585,9 +585,6 @@ static int io_register_mem_region(struct io_ring_ctx *ctx, void __user *uarg) rd_uptr = u64_to_user_ptr(reg.region_uptr); if (copy_from_user(&rd, rd_uptr, sizeof(rd))) return -EFAULT; - - if (!(rd.flags & IORING_MEM_REGION_TYPE_USER)) - return -EINVAL; if (memchr_inv(®.__resv, 0, sizeof(reg.__resv))) return -EINVAL; if (reg.flags & ~IORING_MEM_REGION_REG_WAIT_ARG) @@ -602,7 +599,8 @@ static int io_register_mem_region(struct io_ring_ctx *ctx, void __user *uarg) !(ctx->flags & IORING_SETUP_R_DISABLED)) return -EINVAL; - ret = io_create_region(ctx, &ctx->param_region, &rd); + ret = io_create_region_mmap_safe(ctx, &ctx->param_region, &rd, + IORING_MAP_OFF_PARAM_REGION); if (ret) return ret; if (copy_to_user(rd_uptr, &rd, sizeof(rd))) { From patchwork Sun Nov 24 21:12:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13884200 Received: from mail-wm1-f53.google.com (mail-wm1-f53.google.com [209.85.128.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3CF9F15A848 for ; Sun, 24 Nov 2024 21:12:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482739; cv=none; b=W7Yp2kbL7jFaIbs1XT62+371pL/iZdKtcBW2LGhyq7cB1GrYzHnyvvXYb4T/k7G6xnXNvn2P5mUAnREKB64qHgLqHbkaIDMg2iT+Yr9tQ8Gk23JtglS8YMrwOrYapQD4u2B1GphRDS9J8K4G+sHfJOtvsc0ebw0a6iwOonFJRFs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482739; c=relaxed/simple; bh=eEGFpZJs/btPVtQBKq8PqhPNCWaZDhL6G6EIywe/sxk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hFi+oM5pkuE56ir0nx/HFEAK+F4GwVwljF1GF6JRST47t4iLxRdQmFXIv3tIHopAVsxlaBn9XriEDmGUzWBbzZc8XEp64EZMSsPkMk+Y5oxuum+P3pceSsO6sm32+KJWAqF/9JCF94aTIsAZ2kiG4UYP0smq3sENRg0Uda4991w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=WeKJmL/d; arc=none smtp.client-ip=209.85.128.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="WeKJmL/d" Received: by mail-wm1-f53.google.com with SMTP id 5b1f17b1804b1-431ac30d379so35075595e9.1 for ; Sun, 24 Nov 2024 13:12:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732482736; x=1733087536; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eroRJWDn4SnJNKrnqVa/UqcgSVxmuwFUrHyAThGO38E=; b=WeKJmL/dVyWKn0C5q+63AvX/l9LQSRvF4+Fso2AbinUY/7493OlzsslCACs6xtijmC SXqRSendRKcxAWAG86sToW1QRiRrkZJFtQewVEq+fwfIr2m62AChv+GoZrHUwqA7PPVU 8TJUEHjFb0PwvlBh86nx47vx+2m0my32wtV36RfHxBAZDNa8s6Ovsq7VrwTBs0cKjJYM W5mhs9EICzN3skCrfpuEmtDm6ozE3/veii8y8LeFQgWBJLfypn1NIHK3snyxwzewUjZB I3XzF+2SSDOeLcMow/eFGcCBMy8L+s6bbIm7hLlbcoTKzmqyoi2LKsk4BO2H05kmSCUi grCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732482736; x=1733087536; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eroRJWDn4SnJNKrnqVa/UqcgSVxmuwFUrHyAThGO38E=; b=J9w+aUY/E9rjIJFmReRFL2ltpvJ+PwRf5OLsv05IeUybbq19KxhOWzuvdoI/HZSh03 5RJ3jPHENzdBJGoQZZLdGiDINOwiL1ULqJi3ESmtEHgqpqpyl+1ATyl3T0TmTeHSGNtg aJRhGqTBjxnMwqyf9MfBPuCg432tK2bgtwKXSpsdQ9+LkuGV09KnBc4cTEqdGZslu0yO b6m85Ygu3pew/HVp8p5kUPQy8+Q+3JWSiNIZtk/hO88LrqaNIGUwD3pgOwxi/LZTdpiF QxjOTHuXRSlmSK5WeEQ5hlQBASWYpnJd+iJMGhu1vyBEpPCWnUXhWLhEiQNP2RHHSv7l vLTg== X-Gm-Message-State: AOJu0YzuWGsMpdsIhBKSFyr+N5qjhPXaAVCL2TxtMM7h0WV2qvnMcxq4 PNgFRDUHES9BzESn2lfUiattvgqVEpJxbgY8y6zvHs6GjvWZKM+35Jx8dQ== X-Gm-Gg: ASbGnctjdLPKvFIzX4KFCkc/asGNoZN/AT/wKLSGNeqjmYoN+Qs6SJx+X0pZ5NWC5Tq tZQ+lfZYfwQOZtPkrDobEKsv5PI/JebC88coqz+x3Xxuf+BuOtKTb/jS3RTQXK79f52VAnG3dP3 0EShMaPAiocIBTyb6GQBLHNTUgI9g3T7g7LDsHJkOoBsT1rQrmdn9joVQOUF2govowKVUp7d/0z +unO0iWePb99yaV/DenO1WnLzvS7/i83a+cmxX7yRCEauXyE3Aj63m9DfKi+ZI= X-Google-Smtp-Source: AGHT+IHYOiVhCzq0vDGsuMbIG1TuX+3i6zQKrTTycIJ7pp+0ktrZb7TXc+v6WdAIfvcWf93LpiV2bw== X-Received: by 2002:a05:600c:1f88:b0:42c:b4f2:7c30 with SMTP id 5b1f17b1804b1-433ce49112fmr78676765e9.23.1732482735994; Sun, 24 Nov 2024 13:12:15 -0800 (PST) Received: from 127.0.0.1localhost ([85.255.235.224]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-432f643b299sm132733745e9.0.2024.11.24.13.12.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 24 Nov 2024 13:12:15 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v2 12/18] io_uring: pass ctx to io_register_free_rings Date: Sun, 24 Nov 2024 21:12:29 +0000 Message-ID: <9656c906a0ab1adb4441250e3c78c017122dad26.1732481694.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 A preparation patch, pass the context to io_register_free_rings. Signed-off-by: Pavel Begunkov --- io_uring/register.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/io_uring/register.c b/io_uring/register.c index 5b099ec36d00..5e07205fb071 100644 --- a/io_uring/register.c +++ b/io_uring/register.c @@ -375,7 +375,8 @@ struct io_ring_ctx_rings { struct io_rings *rings; }; -static void io_register_free_rings(struct io_uring_params *p, +static void io_register_free_rings(struct io_ring_ctx *ctx, + struct io_uring_params *p, struct io_ring_ctx_rings *r) { if (!(p->flags & IORING_SETUP_NO_MMAP)) { @@ -452,7 +453,7 @@ static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg) n.rings->cq_ring_entries = p.cq_entries; if (copy_to_user(arg, &p, sizeof(p))) { - io_register_free_rings(&p, &n); + io_register_free_rings(ctx, &p, &n); return -EFAULT; } @@ -461,7 +462,7 @@ static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg) else size = array_size(sizeof(struct io_uring_sqe), p.sq_entries); if (size == SIZE_MAX) { - io_register_free_rings(&p, &n); + io_register_free_rings(ctx, &p, &n); return -EOVERFLOW; } @@ -472,7 +473,7 @@ static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg) p.sq_off.user_addr, size); if (IS_ERR(ptr)) { - io_register_free_rings(&p, &n); + io_register_free_rings(ctx, &p, &n); return PTR_ERR(ptr); } @@ -562,7 +563,7 @@ static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg) out: spin_unlock(&ctx->completion_lock); mutex_unlock(&ctx->mmap_lock); - io_register_free_rings(&p, to_free); + io_register_free_rings(ctx, &p, to_free); if (ctx->sq_data) io_sq_thread_unpark(ctx->sq_data); From patchwork Sun Nov 24 21:12:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13884201 Received: from mail-wm1-f42.google.com (mail-wm1-f42.google.com [209.85.128.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ECD04156960 for ; Sun, 24 Nov 2024 21:12:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482740; cv=none; b=Q+zFOWZMu+YuxH12HZrWQJSl+sAnQ/fzYd8Fb5/RY7YjAZyPjGXsfFAz1llx3JGi5gOfyIoSON/Oq3Dd3DJomOwYRfOcOXhuixIaw7H4CzfaL9K0LHBnAy50JqvccD9pFK4bukmMDgxtgGI7+M+q6uiSrGPQnaVSRZEKBGpkR2o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482740; c=relaxed/simple; bh=7SPNydoRvUkf5XyAMfTHZ15KufbLOiwBoh4vK3N7Os0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ioXoLCaqnrII7vPfdfZhS1d3AawkyW9yRZN2HRqXk8dp1N85Yf9vjiKap7C04M+glFtQyki794NvnzZDqJxc9Gh6YiJTL4mB52leKOhbtP09lk805bMkvYIw5ljjSHpL1RVPD3zL2K27wjAkMe9Dm/HQ/aN6U0xzmUGGJ7GHgmU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=QRbEM1Tq; arc=none smtp.client-ip=209.85.128.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="QRbEM1Tq" Received: by mail-wm1-f42.google.com with SMTP id 5b1f17b1804b1-4314fa33a35so33371075e9.1 for ; Sun, 24 Nov 2024 13:12:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732482737; x=1733087537; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6Wvr3o9uNStf39InzwsOAPQ7rRYdvVvlqZ6yOr6JckQ=; b=QRbEM1TqjeoQMi3oUIO4mMHfaG8xIllldExzT7khbszsPJf75yU6bfQEjmQ44OTmPI z9vPNhZumDmUratPUL8PuyA0FrE1XZPqlN4ZD6v0JmlF2YGqWevsAyrfARIJebBE4Cbu x6oFKZaDIwctkoN7xz/GwWDAyDhJM73Ph44sLwgBXLZzikB5ql4vftYFmmeVfCHKnlj/ 0RxiomUsg7KalY5uc8GEkw3RvEUN/cDMt1N7KRXyhNT63tdSOvGWdgmppu1IF6BB0A3j dO9DUPnMFkk2bwK6m9MXO/PMDue1kWqnAjsVRRoaIofw2+XGFdGPZsAHW9SvN6cYVdUV v40A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732482737; x=1733087537; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6Wvr3o9uNStf39InzwsOAPQ7rRYdvVvlqZ6yOr6JckQ=; b=rA2C0MIP4WyF0OzleLQaEaTfOvtEo5rswsCMI5t+tKvpZvuGK10sem1H6s9eEb9wbW UDaqafcpTVK6K8Zj8k+30NqIce13qkmUmv8CWu9vMv+rPeihAhevkvpnntexytfEtthU Dowm8fPEAa3B4X9qLiYmMs2BNXka8ze2/T99oITDxoXEMpOAbQXCHg294KTos5poLJCP BCf00Lw/yFMpSSlssr+Id8WQRJhx/RN4KwduOlWCW3S1kQKeY9So63QhOH9k6uTrcZD+ FXj3CzUJ2Hms/T98BYMCOfQIMKkEKRN6t6D2eudq/SvhGuJ5peDVCnRd4asTGm0uuFfJ 8/4Q== X-Gm-Message-State: AOJu0YyC4ndqqLDa4OYrJfjcLjh+NIh+wrBU1dJ73n6X2A1YrPa1kyHf RT49X4i3w5WxulDiTut6koDbzlCm7FEryJik6zBh9vyANDP/RMxrqxRL2w== X-Gm-Gg: ASbGncsLdCzF9Ew1RGPnl9VYvmcBsz7EUmQfY4Hc4bziU26i/x5OfXiXg4Rul8GRIAX wNBak06+W9XoQRwpgv7m1HnozJn/e1PPq+h4+vKpN6ssDlblvqZqMpuFPDoKtTycOFsbwg+D6C8 X7lSDPXjBOT4b8bxJF9G1QVhPpe1ueGOB/UHAQmryz2NAP+frvMzOOy2Q+wnhegsoVBklVqA/Py 3OTtqYzM5/k7OpVXmJ6FvfZSjIY9WuVPZqH8fT2q4qQBiE7/NbrXGYvQQn11tg= X-Google-Smtp-Source: AGHT+IGqoWWD9AK+T8U7StDPyBVwYbDd9ZDsh1V5noeYDrkyuX6hk8DjVPL7e7LeOJGK4aQ1KBtXjQ== X-Received: by 2002:a05:600c:1da6:b0:434:9e17:18e5 with SMTP id 5b1f17b1804b1-4349e171d04mr19786475e9.0.1732482737272; Sun, 24 Nov 2024 13:12:17 -0800 (PST) Received: from 127.0.0.1localhost ([85.255.235.224]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-432f643b299sm132733745e9.0.2024.11.24.13.12.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 24 Nov 2024 13:12:16 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v2 13/18] io_uring: use region api for SQ Date: Sun, 24 Nov 2024 21:12:30 +0000 Message-ID: <0b85236173823848fa6207110e74914e5dba47e8.1732481694.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Convert internal parts of the SQ managment to the region API. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 3 +-- io_uring/io_uring.c | 36 +++++++++++++--------------------- io_uring/memmap.c | 3 +-- io_uring/register.c | 35 +++++++++++++++------------------ 4 files changed, 32 insertions(+), 45 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 4cee414080fd..3f353f269c6e 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -431,10 +431,9 @@ struct io_ring_ctx { * the gup'ed pages for the two rings, and the sqes. */ unsigned short n_ring_pages; - unsigned short n_sqe_pages; struct page **ring_pages; - struct page **sqe_pages; + struct io_mapped_region sq_region; /* used for optimised request parameter and wait argument passing */ struct io_mapped_region param_region; }; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index fee3b7f50176..a1dca7bce54a 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2633,29 +2633,19 @@ static void *io_rings_map(struct io_ring_ctx *ctx, unsigned long uaddr, size); } -static void *io_sqes_map(struct io_ring_ctx *ctx, unsigned long uaddr, - size_t size) -{ - return __io_uaddr_map(&ctx->sqe_pages, &ctx->n_sqe_pages, uaddr, - size); -} - static void io_rings_free(struct io_ring_ctx *ctx) { if (!(ctx->flags & IORING_SETUP_NO_MMAP)) { io_pages_unmap(ctx->rings, &ctx->ring_pages, &ctx->n_ring_pages, true); - io_pages_unmap(ctx->sq_sqes, &ctx->sqe_pages, &ctx->n_sqe_pages, - true); } else { io_pages_free(&ctx->ring_pages, ctx->n_ring_pages); ctx->n_ring_pages = 0; - io_pages_free(&ctx->sqe_pages, ctx->n_sqe_pages); - ctx->n_sqe_pages = 0; vunmap(ctx->rings); - vunmap(ctx->sq_sqes); } + io_free_region(ctx, &ctx->sq_region); + ctx->rings = NULL; ctx->sq_sqes = NULL; } @@ -3472,9 +3462,10 @@ bool io_is_uring_fops(struct file *file) static __cold int io_allocate_scq_urings(struct io_ring_ctx *ctx, struct io_uring_params *p) { + struct io_uring_region_desc rd; struct io_rings *rings; size_t size, sq_array_offset; - void *ptr; + int ret; /* make sure these are sane, as we already accounted them */ ctx->sq_entries = p->sq_entries; @@ -3510,17 +3501,18 @@ static __cold int io_allocate_scq_urings(struct io_ring_ctx *ctx, return -EOVERFLOW; } - if (!(ctx->flags & IORING_SETUP_NO_MMAP)) - ptr = io_pages_map(&ctx->sqe_pages, &ctx->n_sqe_pages, size); - else - ptr = io_sqes_map(ctx, p->sq_off.user_addr, size); - - if (IS_ERR(ptr)) { + memset(&rd, 0, sizeof(rd)); + rd.size = PAGE_ALIGN(size); + if (ctx->flags & IORING_SETUP_NO_MMAP) { + rd.user_addr = p->sq_off.user_addr; + rd.flags |= IORING_MEM_REGION_TYPE_USER; + } + ret = io_create_region(ctx, &ctx->sq_region, &rd, IORING_OFF_SQES); + if (ret) { io_rings_free(ctx); - return PTR_ERR(ptr); + return ret; } - - ctx->sq_sqes = ptr; + ctx->sq_sqes = io_region_get_ptr(&ctx->sq_region); return 0; } diff --git a/io_uring/memmap.c b/io_uring/memmap.c index 5d971ba33d5a..0a2d03bd312b 100644 --- a/io_uring/memmap.c +++ b/io_uring/memmap.c @@ -464,8 +464,7 @@ __cold int io_uring_mmap(struct file *file, struct vm_area_struct *vma) npages = min(ctx->n_ring_pages, (sz + PAGE_SIZE - 1) >> PAGE_SHIFT); return io_uring_mmap_pages(ctx, vma, ctx->ring_pages, npages); case IORING_OFF_SQES: - return io_uring_mmap_pages(ctx, vma, ctx->sqe_pages, - ctx->n_sqe_pages); + return io_region_mmap(ctx, &ctx->sq_region, vma); case IORING_OFF_PBUF_RING: return io_pbuf_mmap(file, vma); case IORING_MAP_OFF_PARAM_REGION: diff --git a/io_uring/register.c b/io_uring/register.c index 5e07205fb071..44cd64923d31 100644 --- a/io_uring/register.c +++ b/io_uring/register.c @@ -368,11 +368,11 @@ static int io_register_clock(struct io_ring_ctx *ctx, */ struct io_ring_ctx_rings { unsigned short n_ring_pages; - unsigned short n_sqe_pages; struct page **ring_pages; - struct page **sqe_pages; - struct io_uring_sqe *sq_sqes; struct io_rings *rings; + + struct io_uring_sqe *sq_sqes; + struct io_mapped_region sq_region; }; static void io_register_free_rings(struct io_ring_ctx *ctx, @@ -382,14 +382,11 @@ static void io_register_free_rings(struct io_ring_ctx *ctx, if (!(p->flags & IORING_SETUP_NO_MMAP)) { io_pages_unmap(r->rings, &r->ring_pages, &r->n_ring_pages, true); - io_pages_unmap(r->sq_sqes, &r->sqe_pages, &r->n_sqe_pages, - true); } else { io_pages_free(&r->ring_pages, r->n_ring_pages); - io_pages_free(&r->sqe_pages, r->n_sqe_pages); vunmap(r->rings); - vunmap(r->sq_sqes); } + io_free_region(ctx, &r->sq_region); } #define swap_old(ctx, o, n, field) \ @@ -404,11 +401,11 @@ static void io_register_free_rings(struct io_ring_ctx *ctx, static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg) { + struct io_uring_region_desc rd; struct io_ring_ctx_rings o = { }, n = { }, *to_free = NULL; size_t size, sq_array_offset; struct io_uring_params p; unsigned i, tail; - void *ptr; int ret; /* for single issuer, must be owner resizing */ @@ -466,16 +463,18 @@ static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg) return -EOVERFLOW; } - if (!(p.flags & IORING_SETUP_NO_MMAP)) - ptr = io_pages_map(&n.sqe_pages, &n.n_sqe_pages, size); - else - ptr = __io_uaddr_map(&n.sqe_pages, &n.n_sqe_pages, - p.sq_off.user_addr, - size); - if (IS_ERR(ptr)) { + memset(&rd, 0, sizeof(rd)); + rd.size = PAGE_ALIGN(size); + if (p.flags & IORING_SETUP_NO_MMAP) { + rd.user_addr = p.sq_off.user_addr; + rd.flags |= IORING_MEM_REGION_TYPE_USER; + } + ret = io_create_region_mmap_safe(ctx, &n.sq_region, &rd, IORING_OFF_SQES); + if (ret) { io_register_free_rings(ctx, &p, &n); - return PTR_ERR(ptr); + return ret; } + n.sq_sqes = io_region_get_ptr(&n.sq_region); /* * If using SQPOLL, park the thread @@ -506,7 +505,6 @@ static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg) * Now copy SQ and CQ entries, if any. If either of the destination * rings can't hold what is already there, then fail the operation. */ - n.sq_sqes = ptr; tail = o.rings->sq.tail; if (tail - o.rings->sq.head > p.sq_entries) goto overflow; @@ -555,9 +553,8 @@ static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg) ctx->rings = n.rings; ctx->sq_sqes = n.sq_sqes; swap_old(ctx, o, n, n_ring_pages); - swap_old(ctx, o, n, n_sqe_pages); swap_old(ctx, o, n, ring_pages); - swap_old(ctx, o, n, sqe_pages); + swap_old(ctx, o, n, sq_region); to_free = &o; ret = 0; out: From patchwork Sun Nov 24 21:12:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13884202 Received: from mail-wr1-f53.google.com (mail-wr1-f53.google.com [209.85.221.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7C946163 for ; Sun, 24 Nov 2024 21:12:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482742; cv=none; b=GleTxjcn8/pmfAv0eIYLBfQDEHS681JAUlhV4oRPeXZSSlHAt104mQjgdplfN7KvVklD9Mjx+BpAvuNULDqHDc3861Dbr+ix8/AMw0mowO7xj7dlxbicAc6FvhSM6ukbsGd9KmTDNk9hE2g8xPMJezaHNNAKu01YsuNO89Qz4+M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482742; c=relaxed/simple; bh=qunIwsik9CJrbGOX0b2lyFAD0nodsGoh4Aek47k9Smw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Xa17B9P5GmLPgzQli1IfZLX+EBIfVgWaYpxT8oqnbZEef9+RQOEptPQVg2a9LtPv42JSmJaUcbhBGH7sFsGkPNkcB9zLj9pgZlkKgh9w+f2mCTmQJBcT59a8CCAY8inOlnIFQ0S7Wc2J0iixGpuDxsmjA8p+rC7krj56camNXPE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=DhjVJrUF; arc=none smtp.client-ip=209.85.221.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="DhjVJrUF" Received: by mail-wr1-f53.google.com with SMTP id ffacd0b85a97d-382588b7a5cso2429666f8f.3 for ; Sun, 24 Nov 2024 13:12:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732482738; x=1733087538; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BSQND+9XVUkAsmMkZ/9UPwxAYa30YwMjtyKoN9vzpOY=; b=DhjVJrUFEuo7ZnQnSpOzAzdLER7vcNcLMhHvrOWt6Ich3sTDl+y67HqpYrUrmO+VOE vuTZtEH/lpgtf9TzswPfC5luD1oH33fZpyyNhH57d/dM8WlRNlBREbWOcmLcnDKS2bjN 8KtrTrDMbrc6c/cZZ1BAV8TCcYDSW3QNpXxNbTj5sOtarNm+Pn1B3gHcbv01gIuXhIPP 8+PrhkDmiN7pwxJSZ4fl/2kk7M5SjPNpuPQL/oFdUzSt5Hv9gBYgfhsVfhtNLeZE6VkE eJiiLONYlO941wjR1COseEZUyq/VouuQ0mzIwIovu1OPZus9W5bFWZrxXF7Hry0Z3V1E 2SAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732482738; x=1733087538; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BSQND+9XVUkAsmMkZ/9UPwxAYa30YwMjtyKoN9vzpOY=; b=aI4MaBCOQQt1gbiwDmGrydkC9mpeqc9bjc9i9n8wkTNEpxjdx+suJNruUMpLFmhOY2 OefqWWkk71AmYuUWlG7a1ti0yJM+PiHZuDuiY/bus8bg7qHxxG5E2dBWKtSRKUo/bBrE phd4+PK3StV8r+tuk6gCLcmWNfJUbk1hH+jWS0vv33Ry22rXSa1ngFa3jR/oihN21Eaw uWQMpf4VC6uSZe3D/GdC6DE9eUOM2cjgJMLeU8auCisgy5G/wjh9YC97epdbAXb7CwQl UIjYBYK5WLNjRtc/UYnoJ4u+kjxAGDVb4d82OWSxpCY63hAAg/aqNRmI1OFE6xE2hLAu yzPA== X-Gm-Message-State: AOJu0Yy0GHyuM/oxFtg2uGvhag9/hqykR4aRX4Pwl+dZR0RempydB+fV DFzi1NWV3/tMNG74975qcNws8W7HtImiGjryHooe1YOrWbEXdmDZJYQOEA== X-Gm-Gg: ASbGnctsapGkhI4qxtMuhiG7MrlI0fEK507NsgL5APGAfsSuHpUufca+PTJGFIe2x83 1Jy1vMmFFSNxo95KRBfC9s6ttD44C9VLBDbGH95b1FvEIoTYjpZeHhKhGwYaaGK6qOjuUGz/Exr tqwO2FaeQ4n5+1887aHUT/dlgakEVJvqZDaz484uZDDyGc0FB7Z4plA5s3nG0qfU17bMh0DfXiG wcgxIFEy4Elpg7vxL2KY4G/nC7xKV8Uc4YE8zzDKiGNtfpHiBiGlckkrqD6Clg= X-Google-Smtp-Source: AGHT+IE6FdhDbY37LYSDxMZ2H98HUCqBgPSJcqB5Iq+AWlY8RWtD9hL8JqWCaPAUMNyofrEKbiC8ig== X-Received: by 2002:a5d:5f53:0:b0:382:4bb8:e1d with SMTP id ffacd0b85a97d-38260b4d616mr8587174f8f.1.1732482738491; Sun, 24 Nov 2024 13:12:18 -0800 (PST) Received: from 127.0.0.1localhost ([85.255.235.224]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-432f643b299sm132733745e9.0.2024.11.24.13.12.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 24 Nov 2024 13:12:18 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v2 14/18] io_uring: use region api for CQ Date: Sun, 24 Nov 2024 21:12:31 +0000 Message-ID: <42b3eda88aed4b3542534747cb0ce22744042d98.1732481694.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Convert internal parts of the CQ/SQ array managment to the region API. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 8 +---- io_uring/io_uring.c | 36 +++++++--------------- io_uring/memmap.c | 55 +++++----------------------------- io_uring/memmap.h | 4 --- io_uring/register.c | 35 ++++++++++------------ 5 files changed, 36 insertions(+), 102 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 3f353f269c6e..2db252841509 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -426,14 +426,8 @@ struct io_ring_ctx { */ struct mutex mmap_lock; - /* - * If IORING_SETUP_NO_MMAP is used, then the below holds - * the gup'ed pages for the two rings, and the sqes. - */ - unsigned short n_ring_pages; - struct page **ring_pages; - struct io_mapped_region sq_region; + struct io_mapped_region ring_region; /* used for optimised request parameter and wait argument passing */ struct io_mapped_region param_region; }; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index a1dca7bce54a..b346a1f5f353 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2626,26 +2626,10 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, u32 flags, return READ_ONCE(rings->cq.head) == READ_ONCE(rings->cq.tail) ? ret : 0; } -static void *io_rings_map(struct io_ring_ctx *ctx, unsigned long uaddr, - size_t size) -{ - return __io_uaddr_map(&ctx->ring_pages, &ctx->n_ring_pages, uaddr, - size); -} - static void io_rings_free(struct io_ring_ctx *ctx) { - if (!(ctx->flags & IORING_SETUP_NO_MMAP)) { - io_pages_unmap(ctx->rings, &ctx->ring_pages, &ctx->n_ring_pages, - true); - } else { - io_pages_free(&ctx->ring_pages, ctx->n_ring_pages); - ctx->n_ring_pages = 0; - vunmap(ctx->rings); - } - io_free_region(ctx, &ctx->sq_region); - + io_free_region(ctx, &ctx->ring_region); ctx->rings = NULL; ctx->sq_sqes = NULL; } @@ -3476,15 +3460,17 @@ static __cold int io_allocate_scq_urings(struct io_ring_ctx *ctx, if (size == SIZE_MAX) return -EOVERFLOW; - if (!(ctx->flags & IORING_SETUP_NO_MMAP)) - rings = io_pages_map(&ctx->ring_pages, &ctx->n_ring_pages, size); - else - rings = io_rings_map(ctx, p->cq_off.user_addr, size); - - if (IS_ERR(rings)) - return PTR_ERR(rings); + memset(&rd, 0, sizeof(rd)); + rd.size = PAGE_ALIGN(size); + if (ctx->flags & IORING_SETUP_NO_MMAP) { + rd.user_addr = p->cq_off.user_addr; + rd.flags |= IORING_MEM_REGION_TYPE_USER; + } + ret = io_create_region(ctx, &ctx->ring_region, &rd, IORING_OFF_CQ_RING); + if (ret) + return ret; + ctx->rings = rings = io_region_get_ptr(&ctx->ring_region); - ctx->rings = rings; if (!(ctx->flags & IORING_SETUP_NO_SQARRAY)) ctx->sq_array = (u32 *)((char *)rings + sq_array_offset); rings->sq_ring_mask = p->sq_entries - 1; diff --git a/io_uring/memmap.c b/io_uring/memmap.c index 0a2d03bd312b..52afe0576be6 100644 --- a/io_uring/memmap.c +++ b/io_uring/memmap.c @@ -118,18 +118,6 @@ void io_pages_unmap(void *ptr, struct page ***pages, unsigned short *npages, *npages = 0; } -void io_pages_free(struct page ***pages, int npages) -{ - struct page **page_array = *pages; - - if (!page_array) - return; - - unpin_user_pages(page_array, npages); - kvfree(page_array); - *pages = NULL; -} - struct page **io_pin_pages(unsigned long uaddr, unsigned long len, int *npages) { unsigned long start, end, nr_pages; @@ -167,34 +155,6 @@ struct page **io_pin_pages(unsigned long uaddr, unsigned long len, int *npages) return ERR_PTR(ret); } -void *__io_uaddr_map(struct page ***pages, unsigned short *npages, - unsigned long uaddr, size_t size) -{ - struct page **page_array; - unsigned int nr_pages; - void *page_addr; - - *npages = 0; - - if (uaddr & (PAGE_SIZE - 1) || !size) - return ERR_PTR(-EINVAL); - - nr_pages = 0; - page_array = io_pin_pages(uaddr, size, &nr_pages); - if (IS_ERR(page_array)) - return page_array; - - page_addr = vmap(page_array, nr_pages, VM_MAP, PAGE_KERNEL); - if (page_addr) { - *pages = page_array; - *npages = nr_pages; - return page_addr; - } - - io_pages_free(&page_array, nr_pages); - return ERR_PTR(-ENOMEM); -} - enum { IO_REGION_F_VMAP = 1, IO_REGION_F_USER_PINNED = 2, @@ -383,9 +343,10 @@ static void *io_region_validate_mmap(struct io_ring_ctx *ctx, static int io_region_mmap(struct io_ring_ctx *ctx, struct io_mapped_region *mr, - struct vm_area_struct *vma) + struct vm_area_struct *vma, + unsigned max_pages) { - unsigned long nr_pages = mr->nr_pages; + unsigned long nr_pages = min(mr->nr_pages, max_pages); vm_flags_set(vma, VM_DONTEXPAND); return vm_insert_pages(vma, vma->vm_start, mr->pages, &nr_pages); @@ -449,7 +410,7 @@ __cold int io_uring_mmap(struct file *file, struct vm_area_struct *vma) struct io_ring_ctx *ctx = file->private_data; size_t sz = vma->vm_end - vma->vm_start; long offset = vma->vm_pgoff << PAGE_SHIFT; - unsigned int npages; + unsigned int page_limit; void *ptr; guard(mutex)(&ctx->mmap_lock); @@ -461,14 +422,14 @@ __cold int io_uring_mmap(struct file *file, struct vm_area_struct *vma) switch (offset & IORING_OFF_MMAP_MASK) { case IORING_OFF_SQ_RING: case IORING_OFF_CQ_RING: - npages = min(ctx->n_ring_pages, (sz + PAGE_SIZE - 1) >> PAGE_SHIFT); - return io_uring_mmap_pages(ctx, vma, ctx->ring_pages, npages); + page_limit = (sz + PAGE_SIZE - 1) >> PAGE_SHIFT; + return io_region_mmap(ctx, &ctx->ring_region, vma, page_limit); case IORING_OFF_SQES: - return io_region_mmap(ctx, &ctx->sq_region, vma); + return io_region_mmap(ctx, &ctx->sq_region, vma, UINT_MAX); case IORING_OFF_PBUF_RING: return io_pbuf_mmap(file, vma); case IORING_MAP_OFF_PARAM_REGION: - return io_region_mmap(ctx, &ctx->param_region, vma); + return io_region_mmap(ctx, &ctx->param_region, vma, UINT_MAX); } return -EINVAL; diff --git a/io_uring/memmap.h b/io_uring/memmap.h index 2402bca3d700..7395996eb353 100644 --- a/io_uring/memmap.h +++ b/io_uring/memmap.h @@ -4,7 +4,6 @@ #define IORING_MAP_OFF_PARAM_REGION 0x20000000ULL struct page **io_pin_pages(unsigned long ubuf, unsigned long len, int *npages); -void io_pages_free(struct page ***pages, int npages); int io_uring_mmap_pages(struct io_ring_ctx *ctx, struct vm_area_struct *vma, struct page **pages, int npages); @@ -13,9 +12,6 @@ void *io_pages_map(struct page ***out_pages, unsigned short *npages, void io_pages_unmap(void *ptr, struct page ***pages, unsigned short *npages, bool put_pages); -void *__io_uaddr_map(struct page ***pages, unsigned short *npages, - unsigned long uaddr, size_t size); - #ifndef CONFIG_MMU unsigned int io_uring_nommu_mmap_capabilities(struct file *file); #endif diff --git a/io_uring/register.c b/io_uring/register.c index 44cd64923d31..f1698c18c7cb 100644 --- a/io_uring/register.c +++ b/io_uring/register.c @@ -367,26 +367,19 @@ static int io_register_clock(struct io_ring_ctx *ctx, * either mapping or freeing. */ struct io_ring_ctx_rings { - unsigned short n_ring_pages; - struct page **ring_pages; struct io_rings *rings; - struct io_uring_sqe *sq_sqes; + struct io_mapped_region sq_region; + struct io_mapped_region ring_region; }; static void io_register_free_rings(struct io_ring_ctx *ctx, struct io_uring_params *p, struct io_ring_ctx_rings *r) { - if (!(p->flags & IORING_SETUP_NO_MMAP)) { - io_pages_unmap(r->rings, &r->ring_pages, &r->n_ring_pages, - true); - } else { - io_pages_free(&r->ring_pages, r->n_ring_pages); - vunmap(r->rings); - } io_free_region(ctx, &r->sq_region); + io_free_region(ctx, &r->ring_region); } #define swap_old(ctx, o, n, field) \ @@ -436,13 +429,18 @@ static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg) if (size == SIZE_MAX) return -EOVERFLOW; - if (!(p.flags & IORING_SETUP_NO_MMAP)) - n.rings = io_pages_map(&n.ring_pages, &n.n_ring_pages, size); - else - n.rings = __io_uaddr_map(&n.ring_pages, &n.n_ring_pages, - p.cq_off.user_addr, size); - if (IS_ERR(n.rings)) - return PTR_ERR(n.rings); + memset(&rd, 0, sizeof(rd)); + rd.size = PAGE_ALIGN(size); + if (p.flags & IORING_SETUP_NO_MMAP) { + rd.user_addr = p.cq_off.user_addr; + rd.flags |= IORING_MEM_REGION_TYPE_USER; + } + ret = io_create_region_mmap_safe(ctx, &n.ring_region, &rd, IORING_OFF_CQ_RING); + if (ret) { + io_register_free_rings(ctx, &p, &n); + return ret; + } + n.rings = io_region_get_ptr(&n.ring_region); n.rings->sq_ring_mask = p.sq_entries - 1; n.rings->cq_ring_mask = p.cq_entries - 1; @@ -552,8 +550,7 @@ static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg) ctx->rings = n.rings; ctx->sq_sqes = n.sq_sqes; - swap_old(ctx, o, n, n_ring_pages); - swap_old(ctx, o, n, ring_pages); + swap_old(ctx, o, n, ring_region); swap_old(ctx, o, n, sq_region); to_free = &o; ret = 0; From patchwork Sun Nov 24 21:12:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13884203 Received: from mail-wm1-f45.google.com (mail-wm1-f45.google.com [209.85.128.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 582CE156960 for ; Sun, 24 Nov 2024 21:12:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482743; cv=none; b=gxjjE0u7NywVvLeBWVGtsOtFXnLEbUI4+TWRMIFMF2BrkyHs+PCNcTZdpI/bnvYXpww1hJj7/cyeeB69fHIstRhEc+zn1NYXVcJJTtyysUhv5+uDXSyhi0U7Bzk8iqW+6j+YiQ882wy7Nf5aFZXIpkR/XlWBPDpn4YeHBNAJadw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482743; c=relaxed/simple; bh=6E56ks1kypMJjgmiNbwiHW0OEsH6fT3SvMKVVQrd8Go=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=d51I7S3g5d7ySL4VahISdzwGO1dr/5TsVEOvHOIPI/A1yeVnYMA7Hv9ibGz2iH1soEZ/LNo5nfF/fS19B9kEzUMk7tGLxGRKebZyBuF6WHPhClOcKvtZ/H4SqcwB2vHQ7hFYHJd5LHCXy3ZguqzqaPvjt5XLrs89cAcswmVymMo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=lRaHQGKc; arc=none smtp.client-ip=209.85.128.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="lRaHQGKc" Received: by mail-wm1-f45.google.com with SMTP id 5b1f17b1804b1-43158625112so34980275e9.3 for ; Sun, 24 Nov 2024 13:12:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732482739; x=1733087539; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UMqCVFH4bCWU9Q/bAjVPrMRtHJHe9onTMmao6gGbEpU=; b=lRaHQGKcFdiLXaW9rCXzM5IztELkN7zMxstGWiAnh8cb/8THEHyxT4c4DM6fvzmYMq Xsjym4gKzktXQJskFwyWoJ8YZULhVNXihqmGxmXb89WjBpQ8DX+YcwWC63aogRfNzEmX ETxWGp4MXhnrSZQTSvkhMF0/fEZdSK8G/I/eVr/vnMbql3jGCE23vfG0G5f5+DtrxGX7 xncTUzWwae4Gs1PptNrwW6QYKO7981l9wnNxe9lkQvFyf8PzSVgb4WLma38S0Reb31mJ HfNimDaJojp7ljsNeBhcIZeJMf0KVe6FBZeJssOG4f3mgfUT3MKuAiVGmzrU4Sb3iNw/ Jb8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732482739; x=1733087539; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UMqCVFH4bCWU9Q/bAjVPrMRtHJHe9onTMmao6gGbEpU=; b=niHXEvbirueRaKU1FmyDow+AVsx21J/jyc4+VS6yARW7LgxrCMNnWgLHJ2hh2fwH3y gCbc7w6HZJBLXYkeGBd/MUl0OB1tUfy2LAo80K+2rzU+y4p4C+vswZfyGBv7qY0BCvlS A9w2xrTpk2n+T8KfpoiwCccY/sBVs0/+IBdSXxhp4wkKyebeBOdpxJ4l/BSTYm9yq/8O TTkv5S5KRleQ1wQd37Meqj86GQ8crO84lvAcGjUzJqwz3wQPyx1BJ1j8zF0UV3/tnx6J q29hMSclDwo0AJe9lC9PJ+92MJ22yrXBjHoMplA3EYbtLg7czt4v+hV6gdjUgsDshvU3 ZHbw== X-Gm-Message-State: AOJu0YxvbpTbx9w3aDbvF8F0+c10BYEyUwqUS2dl+C5S04VriK15BPV3 MaugyWNE9J/y3Sn/a0B2HZkUozzTlux8ALATdIUW1Px4hTDf/B89bzxbiA== X-Gm-Gg: ASbGncvmxv+h0DJpA1F3/nsjZh9wPNIRuCPlzArjwi0HkQQwdJSZm4Mue1d9Mk/quXX sV29+rX2gyBxLAif4H1kN8AfeTR0xXtZgm2SdTxJ5rg9LF1arBdW27MefUzNUuIqrmcTmkD1Qjo pBgyGKxEHXoU8zn5ZjJhJXnjPgetihRkzP1Uj6oPIPOpy6BJlEb/Pr/z4tXzzyC/WrAEugrOWd6 FmPIFPJbMwTR3A1E5kEKz52ylpqrUkzlATO4NH0TuiVJseCZLcDabYgEUGj7mQ= X-Google-Smtp-Source: AGHT+IHEmzT58X5BKSfsoIJvpsCMD50MrbnUO0cXtfm0i1tSOMplwjyCTHvg1pBFtjuqga5w2vE28w== X-Received: by 2002:a05:600c:1d08:b0:431:5465:807b with SMTP id 5b1f17b1804b1-433ce4e2582mr84337325e9.32.1732482739539; Sun, 24 Nov 2024 13:12:19 -0800 (PST) Received: from 127.0.0.1localhost ([85.255.235.224]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-432f643b299sm132733745e9.0.2024.11.24.13.12.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 24 Nov 2024 13:12:19 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v2 15/18] io_uring/kbuf: use mmap_lock to sync with mmap Date: Sun, 24 Nov 2024 21:12:32 +0000 Message-ID: <62eec2e533534f4f023929fc3c3695947d45e131.1732481694.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 A preparation / cleanup patch simplifying the buf ring - mmap synchronisation. Instead of relying on RCU, which is trickier, do it by grabbing the mmap_lock when when anyone tries to publish or remove a registered buffer to / from ->io_bl_xa. Modifications of the xarray should always be protected by both ->uring_lock and ->mmap_lock, while lookups should hold either of them. While a struct io_buffer_list is in the xarray, the mmap related fields like ->flags and ->buf_pages should stay stable. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 5 +++ io_uring/kbuf.c | 56 +++++++++++++++------------------- io_uring/kbuf.h | 1 - 3 files changed, 29 insertions(+), 33 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 2db252841509..091d1eaf5ba0 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -293,6 +293,11 @@ struct io_ring_ctx { struct io_submit_state submit_state; + /* + * Modifications are protected by ->uring_lock and ->mmap_lock. + * The flags, buf_pages and buf_nr_pages fields should be stable + * once published. + */ struct xarray io_bl_xa; struct io_hash_table cancel_table; diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index d407576ddfb7..662e928cc3b0 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -45,10 +45,11 @@ static int io_buffer_add_list(struct io_ring_ctx *ctx, /* * Store buffer group ID and finally mark the list as visible. * The normal lookup doesn't care about the visibility as we're - * always under the ->uring_lock, but the RCU lookup from mmap does. + * always under the ->uring_lock, but lookups from mmap do. */ bl->bgid = bgid; atomic_set(&bl->refs, 1); + guard(mutex)(&ctx->mmap_lock); return xa_err(xa_store(&ctx->io_bl_xa, bgid, bl, GFP_KERNEL)); } @@ -388,7 +389,7 @@ void io_put_bl(struct io_ring_ctx *ctx, struct io_buffer_list *bl) { if (atomic_dec_and_test(&bl->refs)) { __io_remove_buffers(ctx, bl, -1U); - kfree_rcu(bl, rcu); + kfree(bl); } } @@ -397,10 +398,17 @@ void io_destroy_buffers(struct io_ring_ctx *ctx) struct io_buffer_list *bl; struct list_head *item, *tmp; struct io_buffer *buf; - unsigned long index; - xa_for_each(&ctx->io_bl_xa, index, bl) { - xa_erase(&ctx->io_bl_xa, bl->bgid); + while (1) { + unsigned long index = 0; + + scoped_guard(mutex, &ctx->mmap_lock) { + bl = xa_find(&ctx->io_bl_xa, &index, ULONG_MAX, XA_PRESENT); + if (bl) + xa_erase(&ctx->io_bl_xa, bl->bgid); + } + if (!bl) + break; io_put_bl(ctx, bl); } @@ -589,11 +597,7 @@ int io_provide_buffers(struct io_kiocb *req, unsigned int issue_flags) INIT_LIST_HEAD(&bl->buf_list); ret = io_buffer_add_list(ctx, bl, p->bgid); if (ret) { - /* - * Doesn't need rcu free as it was never visible, but - * let's keep it consistent throughout. - */ - kfree_rcu(bl, rcu); + kfree(bl); goto err; } } @@ -736,7 +740,7 @@ int io_register_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg) return 0; } - kfree_rcu(free_bl, rcu); + kfree(free_bl); return ret; } @@ -760,7 +764,9 @@ int io_unregister_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg) if (!(bl->flags & IOBL_BUF_RING)) return -EINVAL; - xa_erase(&ctx->io_bl_xa, bl->bgid); + scoped_guard(mutex, &ctx->mmap_lock) + xa_erase(&ctx->io_bl_xa, bl->bgid); + io_put_bl(ctx, bl); return 0; } @@ -795,29 +801,13 @@ struct io_buffer_list *io_pbuf_get_bl(struct io_ring_ctx *ctx, unsigned long bgid) { struct io_buffer_list *bl; - bool ret; - /* - * We have to be a bit careful here - we're inside mmap and cannot grab - * the uring_lock. This means the buffer_list could be simultaneously - * going away, if someone is trying to be sneaky. Look it up under rcu - * so we know it's not going away, and attempt to grab a reference to - * it. If the ref is already zero, then fail the mapping. If successful, - * the caller will call io_put_bl() to drop the the reference at at the - * end. This may then safely free the buffer_list (and drop the pages) - * at that point, vm_insert_pages() would've already grabbed the - * necessary vma references. - */ - rcu_read_lock(); bl = xa_load(&ctx->io_bl_xa, bgid); /* must be a mmap'able buffer ring and have pages */ - ret = false; - if (bl && bl->flags & IOBL_MMAP) - ret = atomic_inc_not_zero(&bl->refs); - rcu_read_unlock(); - - if (ret) - return bl; + if (bl && bl->flags & IOBL_MMAP) { + if (atomic_inc_not_zero(&bl->refs)) + return bl; + } return ERR_PTR(-EINVAL); } @@ -829,6 +819,8 @@ int io_pbuf_mmap(struct file *file, struct vm_area_struct *vma) struct io_buffer_list *bl; int bgid, ret; + lockdep_assert_held(&ctx->mmap_lock); + bgid = (pgoff & ~IORING_OFF_MMAP_MASK) >> IORING_OFF_PBUF_SHIFT; bl = io_pbuf_get_bl(ctx, bgid); if (IS_ERR(bl)) diff --git a/io_uring/kbuf.h b/io_uring/kbuf.h index 36aadfe5ac00..d5e4afcbfbb3 100644 --- a/io_uring/kbuf.h +++ b/io_uring/kbuf.h @@ -25,7 +25,6 @@ struct io_buffer_list { struct page **buf_pages; struct io_uring_buf_ring *buf_ring; }; - struct rcu_head rcu; }; __u16 bgid; From patchwork Sun Nov 24 21:12:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13884204 Received: from mail-wm1-f53.google.com (mail-wm1-f53.google.com [209.85.128.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3487E15B97D for ; Sun, 24 Nov 2024 21:12:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482743; cv=none; b=jUVgSk8MXuL94nEvmd9l8aVy+coSKrygujP65VfP29weDCdk8t+GR9zAFjyFvX/0Mpgpp9+zA1y8gErQJialTCSFhYWuDoYC0PtjbCzxAp/6d7pTlqQcHJdKAZlRDlCZWArkOve5DqLvuzJ2dXoklY/ongsWoGsvfhlfjOjVEjI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482743; c=relaxed/simple; bh=HBHMOwbntdYlLb19w1mghXVhp16gZ994kKx4hDX4zC4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=C5B2s5R9sqdmeFj9wKakuN2DvzGzmV4+aEDBgF51GA7C5aQ5mHcYqBB2I/q4Q15rbaHuLJHCDvCndR/4Zux3dav+ZBvn5EJi62BeI+pkOeLvwHUXuw4beUIKLK6yoAz7+ovbfIXNdMJPC67oS4+p4bdwDOhCMpwuYjWNyw1Ogf4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=lLAdqA8n; arc=none smtp.client-ip=209.85.128.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="lLAdqA8n" Received: by mail-wm1-f53.google.com with SMTP id 5b1f17b1804b1-4315baa51d8so34456135e9.0 for ; Sun, 24 Nov 2024 13:12:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732482740; x=1733087540; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jtAiMRa8mZj+7A/VMGZktqkrs4UQMtnl0rhZfY6g9rA=; b=lLAdqA8n/T71tJs68TEhGccPE9Jp/J7i8chibelbeH+WmFPBX2gM17n5/Y3QzbMdyN Usr9JnU5zFf7M7OeEDC+d6MKDAJftXDnRwdcSCl3CihBeUsHGFBfxOTESH9HUdsJzRZc ZH1bEbdtIG98wJGeao+FYRsErXlePD2Uc3oQj6UzM0wSJTgiT4Ip7iEdIbcb3+ZzugeZ focwRQPajVnXx5HRBPYhqsXXzrFTWJZCQ3g/569NgPClrE5tNs2SfZHAFZZ/mdwwtF54 QK7NtvFxZ6NtAP9VSmLafRcYrlcxXlx+fsR5VoFANSugXgBdXvGTO6bSi+Lg37P2/sPT ZJqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732482740; x=1733087540; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jtAiMRa8mZj+7A/VMGZktqkrs4UQMtnl0rhZfY6g9rA=; b=Sjm0g4NsVUbjY8AkbHnifVbhmrZVMoPgbtef8IB2EPYSIfF1lSIuEEwWD9UiEjUhS6 RwIu4s+WrfBf8NH7eL9od7KFwLksxPZ/8zv3lP3mFfaz5JcG+SiDToJP7hAoO0oQ8HjG S2qs+4URMWtxfs28YzxctU1g8Qd2ck5DLldSVSmw3GwzAG2DcFBwat+8H8PasU28xkSY avXMQdy9QUjx165iO0WcutCqTxPKJr+7GMUS9Z/lQC5wL8XqrpAn+oaJXeHob/BwNHIM UDrVnc41s6y7twNp//NM20Ev0cJar6rbZ00fBFE6m0n+2euiE+e/m+Si5pvBkGRzrRoR Zh2Q== X-Gm-Message-State: AOJu0YxqjAQepOSA1pw+tWEzL3OZuYRVfiXav2DYeXjPdfpFjhy/jZ/u JFy/xbKeyVsYIi0AyQcWSlCEnKbksQh9cUkVXJhHbHOkXsiRBR2hmWUofA== X-Gm-Gg: ASbGnctf3ilh34HrpVACYxVp0+ceDmuDEfETqdK5Ej0sE29Udg6zs6ICwShVujjvWgZ T3Px52G6GxJU/7duSlkuTOaP2JM6lk8BLI6X8aXpI6KaF4fzUO1uUh3ogZ6msDAsFNCXJcZEYHX SuTUGPLn4LMCo1tuit5AgObIa8zedSMmbYWiK3v1TUTN26Hs/sHZ89ZsnigpuyN+fM0JrlcwwNR rMST9rquS6GL4+7uwh7WqHk6xog7BO1Jr0W9oP6GMWQLLq1YH5DjZJ8JSfHZfg= X-Google-Smtp-Source: AGHT+IGs2+wJa4E+BCMWu7M4iOGvF0bPQa1i8zpkqXsimgkNZxOD9yRm+DsVdFBoF1HrOxrGHOrswg== X-Received: by 2002:a05:600c:1c0b:b0:431:52b7:a499 with SMTP id 5b1f17b1804b1-433ce48ea08mr81224875e9.20.1732482740408; Sun, 24 Nov 2024 13:12:20 -0800 (PST) Received: from 127.0.0.1localhost ([85.255.235.224]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-432f643b299sm132733745e9.0.2024.11.24.13.12.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 24 Nov 2024 13:12:19 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v2 16/18] io_uring/kbuf: remove pbuf ring refcounting Date: Sun, 24 Nov 2024 21:12:33 +0000 Message-ID: <88da9858b1d5cd09013125a96f42c74f9e10aa1b.1732481694.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 struct io_buffer_list refcounting was needed for RCU based sync with mmap, now we can kill it. Signed-off-by: Pavel Begunkov --- io_uring/kbuf.c | 21 +++++++-------------- io_uring/kbuf.h | 3 --- io_uring/memmap.c | 1 - 3 files changed, 7 insertions(+), 18 deletions(-) diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index 662e928cc3b0..644f61445ec9 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -48,7 +48,6 @@ static int io_buffer_add_list(struct io_ring_ctx *ctx, * always under the ->uring_lock, but lookups from mmap do. */ bl->bgid = bgid; - atomic_set(&bl->refs, 1); guard(mutex)(&ctx->mmap_lock); return xa_err(xa_store(&ctx->io_bl_xa, bgid, bl, GFP_KERNEL)); } @@ -385,12 +384,10 @@ static int __io_remove_buffers(struct io_ring_ctx *ctx, return i; } -void io_put_bl(struct io_ring_ctx *ctx, struct io_buffer_list *bl) +static void io_put_bl(struct io_ring_ctx *ctx, struct io_buffer_list *bl) { - if (atomic_dec_and_test(&bl->refs)) { - __io_remove_buffers(ctx, bl, -1U); - kfree(bl); - } + __io_remove_buffers(ctx, bl, -1U); + kfree(bl); } void io_destroy_buffers(struct io_ring_ctx *ctx) @@ -804,10 +801,8 @@ struct io_buffer_list *io_pbuf_get_bl(struct io_ring_ctx *ctx, bl = xa_load(&ctx->io_bl_xa, bgid); /* must be a mmap'able buffer ring and have pages */ - if (bl && bl->flags & IOBL_MMAP) { - if (atomic_inc_not_zero(&bl->refs)) - return bl; - } + if (bl && bl->flags & IOBL_MMAP) + return bl; return ERR_PTR(-EINVAL); } @@ -817,7 +812,7 @@ int io_pbuf_mmap(struct file *file, struct vm_area_struct *vma) struct io_ring_ctx *ctx = file->private_data; loff_t pgoff = vma->vm_pgoff << PAGE_SHIFT; struct io_buffer_list *bl; - int bgid, ret; + int bgid; lockdep_assert_held(&ctx->mmap_lock); @@ -826,7 +821,5 @@ int io_pbuf_mmap(struct file *file, struct vm_area_struct *vma) if (IS_ERR(bl)) return PTR_ERR(bl); - ret = io_uring_mmap_pages(ctx, vma, bl->buf_pages, bl->buf_nr_pages); - io_put_bl(ctx, bl); - return ret; + return io_uring_mmap_pages(ctx, vma, bl->buf_pages, bl->buf_nr_pages); } diff --git a/io_uring/kbuf.h b/io_uring/kbuf.h index d5e4afcbfbb3..dff7444026a6 100644 --- a/io_uring/kbuf.h +++ b/io_uring/kbuf.h @@ -35,8 +35,6 @@ struct io_buffer_list { __u16 mask; __u16 flags; - - atomic_t refs; }; struct io_buffer { @@ -83,7 +81,6 @@ void __io_put_kbuf(struct io_kiocb *req, int len, unsigned issue_flags); bool io_kbuf_recycle_legacy(struct io_kiocb *req, unsigned issue_flags); -void io_put_bl(struct io_ring_ctx *ctx, struct io_buffer_list *bl); struct io_buffer_list *io_pbuf_get_bl(struct io_ring_ctx *ctx, unsigned long bgid); int io_pbuf_mmap(struct file *file, struct vm_area_struct *vma); diff --git a/io_uring/memmap.c b/io_uring/memmap.c index 52afe0576be6..88428a8dc3bc 100644 --- a/io_uring/memmap.c +++ b/io_uring/memmap.c @@ -384,7 +384,6 @@ static void *io_uring_validate_mmap_request(struct file *file, loff_t pgoff, if (IS_ERR(bl)) return bl; ptr = bl->buf_ring; - io_put_bl(ctx, bl); return ptr; } case IORING_MAP_OFF_PARAM_REGION: From patchwork Sun Nov 24 21:12:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13884205 Received: from mail-wm1-f50.google.com (mail-wm1-f50.google.com [209.85.128.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5804514F136 for ; Sun, 24 Nov 2024 21:12:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482745; cv=none; b=Q+ikWWM+2pj0AG7RqFu62NxPWN9j+eb6mqNzJZ1cubNjNZVrfijp7QME6+VZciSIVtPCeO3npjLJooy+1QnfANWhjQuT2NGlAHCUtnjN57Zy5QUBQdDhBEeJ+CDmiwz1PgtZRv7X1pFrp76Yn09QMvXqVY8OzBvud57YCCbhxQM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482745; c=relaxed/simple; bh=oWkcdYZp8ZG7s5K/XtQzMBNvRindXxCIGzLjVKo/FmQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=m8kqq5eM7Z7IEV9myVkAjV3kUs5o/pdKNiTbRhrd67XFwumbwvcxtVu0sNKDuLO4cdPyuf+rhDRC9i2IVNTEM++knNL67+nm4bj3LBdRqtbdV9fODZVJT+4CZ7+HM/R/0NFO8zPRh/dexUipVuhgMCC1RUrKUNffC70siCVpYyo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Jdsuv699; arc=none smtp.client-ip=209.85.128.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Jdsuv699" Received: by mail-wm1-f50.google.com with SMTP id 5b1f17b1804b1-43162cf1eaaso48598295e9.0 for ; Sun, 24 Nov 2024 13:12:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732482741; x=1733087541; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZC2XeEP04JXajrKNiQ8haKEqXQM2GqxNCdByjG6tFAM=; b=Jdsuv69922BeCb96VYP3KUCktwiIegYE7Am4C1gLb1dQG2A5tZBYd8wD8C5DRKAr9V gDtmOKaatASR0QTpS0TrPkQ6fhoL13CM1uqgG47DtU5sHyFWF/90sDPlNRLe1Guln2KI OiMT8qSICmPUN98QuiS9mXwCNOrVrSqiHPHlwjDNshNdLgP5M3mj0vxFZShnIqbRE82i BblxM8yWqAfcNiEdFkm536bP57V/n5DNd8c6vafUpF1sw8C2dHiTLqbJC6TKAZGVmtrX aABrpG8+hdREfvg5JBRIXqOuv0E6cYUx6Pov5OilmZB2lYYr81HiA8x5qW9nU1/Ac1Qf qz7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732482741; x=1733087541; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZC2XeEP04JXajrKNiQ8haKEqXQM2GqxNCdByjG6tFAM=; b=eGK4nadSPx80jRnVonM75fuxGDCyLaJ5xTGda1fWpVB+wBLF/rS1+EgcGhISCQmJbh st/fgy9cnbNwOhLqA6RLmMdh4LxMtZOZckTFr/fHaIIuybvnPm63W+2QU7z3VTGIjUH1 8gIgO+oNik+V1hObW41kQIokbiZEivZkAEPbs1lI80sbP1B0g5EXFL1DyvFP/DLosI4Y 3ZtShm5DyNIZ7KmogYk2GEW55aoNuz9Tyfwr8ErqFxvxa3pjKepX9bL5KimLnzAlWphI ob+thBz+pY1ARnmr7fauH+ngGZ+eHxWmSJ7z4hO3KzLfpar/M9dQtEuM+LXIvVRQve1T y1nA== X-Gm-Message-State: AOJu0YxCud2M+FDYu/mckcPGxuFU5ttbrOjxZsS/Eg74dsao2QAX+Bm2 WZLrwp95gPrIUKMUnGODMSpTGUO+PaghzdUCoHnfEohNY8hXen8ufZk2Fw== X-Gm-Gg: ASbGncuK1RLVHnyf0kmzNF0ZqTUXFzSQjDJeiXpj42qMnWpqikzWNReyQ11aSthH9yN XvFR6e6GFGk84sIWN/870x1UywPDGT5FBv/kRKCoxzFvL0ngHzU67rzDy8CcCSRoPC88g4yXX0P sgO6VaTtQRtm0C5gRId75RU9gFzguzN87AUNr+PE+uPD6AGSaCtSaP7X/FKanbkv0G65hvloy+s eazEtyM+QfFHlWIL5Q2XnaZXVFW8K5S4GFU9sGD6GWocLJ9RZ+QbDaLn9kIgAk= X-Google-Smtp-Source: AGHT+IHddRzkkirMHWqvrgVVt5KAhexHpC/M/ZdqKlTjYfwIo44BKGhIsq6/evpYcW5A0IpCf8BMxw== X-Received: by 2002:a05:600c:138c:b0:431:562a:54be with SMTP id 5b1f17b1804b1-433ce41ce64mr116533755e9.9.1732482741276; Sun, 24 Nov 2024 13:12:21 -0800 (PST) Received: from 127.0.0.1localhost ([85.255.235.224]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-432f643b299sm132733745e9.0.2024.11.24.13.12.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 24 Nov 2024 13:12:20 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v2 17/18] io_uring/kbuf: use region api for pbuf rings Date: Sun, 24 Nov 2024 21:12:34 +0000 Message-ID: <981856b77fd6144508747550cb2c2c26666184d6.1732481694.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Convert internal parts of the provided buffer ring managment to the region API. It's the last non-region mapped ring we have, so it also kills a bunch of now unused memmap.c helpers. Signed-off-by: Pavel Begunkov --- io_uring/kbuf.c | 170 ++++++++++++++-------------------------------- io_uring/kbuf.h | 18 ++--- io_uring/memmap.c | 116 +++++-------------------------- io_uring/memmap.h | 7 -- 4 files changed, 73 insertions(+), 238 deletions(-) diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index 644f61445ec9..f77d14e4d598 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -351,17 +351,7 @@ static int __io_remove_buffers(struct io_ring_ctx *ctx, if (bl->flags & IOBL_BUF_RING) { i = bl->buf_ring->tail - bl->head; - if (bl->buf_nr_pages) { - int j; - - if (!(bl->flags & IOBL_MMAP)) { - for (j = 0; j < bl->buf_nr_pages; j++) - unpin_user_page(bl->buf_pages[j]); - } - io_pages_unmap(bl->buf_ring, &bl->buf_pages, - &bl->buf_nr_pages, bl->flags & IOBL_MMAP); - bl->flags &= ~IOBL_MMAP; - } + io_free_region(ctx, &bl->region); /* make sure it's seen as empty */ INIT_LIST_HEAD(&bl->buf_list); bl->flags &= ~IOBL_BUF_RING; @@ -614,75 +604,14 @@ int io_provide_buffers(struct io_kiocb *req, unsigned int issue_flags) return IOU_OK; } -static int io_pin_pbuf_ring(struct io_uring_buf_reg *reg, - struct io_buffer_list *bl) -{ - struct io_uring_buf_ring *br = NULL; - struct page **pages; - int nr_pages, ret; - - pages = io_pin_pages(reg->ring_addr, - flex_array_size(br, bufs, reg->ring_entries), - &nr_pages); - if (IS_ERR(pages)) - return PTR_ERR(pages); - - br = vmap(pages, nr_pages, VM_MAP, PAGE_KERNEL); - if (!br) { - ret = -ENOMEM; - goto error_unpin; - } - -#ifdef SHM_COLOUR - /* - * On platforms that have specific aliasing requirements, SHM_COLOUR - * is set and we must guarantee that the kernel and user side align - * nicely. We cannot do that if IOU_PBUF_RING_MMAP isn't set and - * the application mmap's the provided ring buffer. Fail the request - * if we, by chance, don't end up with aligned addresses. The app - * should use IOU_PBUF_RING_MMAP instead, and liburing will handle - * this transparently. - */ - if ((reg->ring_addr | (unsigned long) br) & (SHM_COLOUR - 1)) { - ret = -EINVAL; - goto error_unpin; - } -#endif - bl->buf_pages = pages; - bl->buf_nr_pages = nr_pages; - bl->buf_ring = br; - bl->flags |= IOBL_BUF_RING; - bl->flags &= ~IOBL_MMAP; - return 0; -error_unpin: - unpin_user_pages(pages, nr_pages); - kvfree(pages); - vunmap(br); - return ret; -} - -static int io_alloc_pbuf_ring(struct io_ring_ctx *ctx, - struct io_uring_buf_reg *reg, - struct io_buffer_list *bl) -{ - size_t ring_size; - - ring_size = reg->ring_entries * sizeof(struct io_uring_buf_ring); - - bl->buf_ring = io_pages_map(&bl->buf_pages, &bl->buf_nr_pages, ring_size); - if (IS_ERR(bl->buf_ring)) { - bl->buf_ring = NULL; - return -ENOMEM; - } - - bl->flags |= (IOBL_BUF_RING | IOBL_MMAP); - return 0; -} - int io_register_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg) { struct io_uring_buf_reg reg; struct io_buffer_list *bl, *free_bl = NULL; + struct io_uring_region_desc rd; + struct io_uring_buf_ring *br; + unsigned long mmap_offset; + unsigned long ring_size; int ret; lockdep_assert_held(&ctx->uring_lock); @@ -694,19 +623,8 @@ int io_register_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg) return -EINVAL; if (reg.flags & ~(IOU_PBUF_RING_MMAP | IOU_PBUF_RING_INC)) return -EINVAL; - if (!(reg.flags & IOU_PBUF_RING_MMAP)) { - if (!reg.ring_addr) - return -EFAULT; - if (reg.ring_addr & ~PAGE_MASK) - return -EINVAL; - } else { - if (reg.ring_addr) - return -EINVAL; - } - if (!is_power_of_2(reg.ring_entries)) return -EINVAL; - /* cannot disambiguate full vs empty due to head/tail size */ if (reg.ring_entries >= 65536) return -EINVAL; @@ -722,21 +640,47 @@ int io_register_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg) return -ENOMEM; } - if (!(reg.flags & IOU_PBUF_RING_MMAP)) - ret = io_pin_pbuf_ring(®, bl); - else - ret = io_alloc_pbuf_ring(ctx, ®, bl); + mmap_offset = reg.bgid << IORING_OFF_PBUF_SHIFT; + ring_size = flex_array_size(br, bufs, reg.ring_entries); - if (!ret) { - bl->nr_entries = reg.ring_entries; - bl->mask = reg.ring_entries - 1; - if (reg.flags & IOU_PBUF_RING_INC) - bl->flags |= IOBL_INC; + memset(&rd, 0, sizeof(rd)); + rd.size = PAGE_ALIGN(ring_size); + if (!(reg.flags & IOU_PBUF_RING_MMAP)) { + rd.user_addr = reg.ring_addr; + rd.flags |= IORING_MEM_REGION_TYPE_USER; + } + ret = io_create_region_mmap_safe(ctx, &bl->region, &rd, mmap_offset); + if (ret) + goto fail; + br = io_region_get_ptr(&bl->region); - io_buffer_add_list(ctx, bl, reg.bgid); - return 0; +#ifdef SHM_COLOUR + /* + * On platforms that have specific aliasing requirements, SHM_COLOUR + * is set and we must guarantee that the kernel and user side align + * nicely. We cannot do that if IOU_PBUF_RING_MMAP isn't set and + * the application mmap's the provided ring buffer. Fail the request + * if we, by chance, don't end up with aligned addresses. The app + * should use IOU_PBUF_RING_MMAP instead, and liburing will handle + * this transparently. + */ + if (!(reg.flags & IOU_PBUF_RING_MMAP) && + ((reg->ring_addr | (unsigned long)br) & (SHM_COLOUR - 1))) { + ret = -EINVAL; + goto fail; } +#endif + bl->nr_entries = reg.ring_entries; + bl->mask = reg.ring_entries - 1; + bl->flags |= IOBL_BUF_RING; + bl->buf_ring = br; + if (reg.flags & IOU_PBUF_RING_INC) + bl->flags |= IOBL_INC; + io_buffer_add_list(ctx, bl, reg.bgid); + return 0; +fail: + io_free_region(ctx, &bl->region); kfree(free_bl); return ret; } @@ -794,32 +738,18 @@ int io_register_pbuf_status(struct io_ring_ctx *ctx, void __user *arg) return 0; } -struct io_buffer_list *io_pbuf_get_bl(struct io_ring_ctx *ctx, - unsigned long bgid) -{ - struct io_buffer_list *bl; - - bl = xa_load(&ctx->io_bl_xa, bgid); - /* must be a mmap'able buffer ring and have pages */ - if (bl && bl->flags & IOBL_MMAP) - return bl; - - return ERR_PTR(-EINVAL); -} - -int io_pbuf_mmap(struct file *file, struct vm_area_struct *vma) +struct io_mapped_region *io_pbuf_get_region(struct io_ring_ctx *ctx, + unsigned int bgid) { - struct io_ring_ctx *ctx = file->private_data; - loff_t pgoff = vma->vm_pgoff << PAGE_SHIFT; struct io_buffer_list *bl; - int bgid; lockdep_assert_held(&ctx->mmap_lock); - bgid = (pgoff & ~IORING_OFF_MMAP_MASK) >> IORING_OFF_PBUF_SHIFT; - bl = io_pbuf_get_bl(ctx, bgid); - if (IS_ERR(bl)) - return PTR_ERR(bl); + bl = xa_load(&ctx->io_bl_xa, bgid); + if (!bl || !(bl->flags & IOBL_BUF_RING)) + return NULL; + if (WARN_ON_ONCE(!io_region_is_set(&bl->region))) + return NULL; - return io_uring_mmap_pages(ctx, vma, bl->buf_pages, bl->buf_nr_pages); + return &bl->region; } diff --git a/io_uring/kbuf.h b/io_uring/kbuf.h index dff7444026a6..bd80c44c5af1 100644 --- a/io_uring/kbuf.h +++ b/io_uring/kbuf.h @@ -3,15 +3,13 @@ #define IOU_KBUF_H #include +#include enum { /* ring mapped provided buffers */ IOBL_BUF_RING = 1, - /* ring mapped provided buffers, but mmap'ed by application */ - IOBL_MMAP = 2, /* buffers are consumed incrementally rather than always fully */ - IOBL_INC = 4, - + IOBL_INC = 2, }; struct io_buffer_list { @@ -21,10 +19,7 @@ struct io_buffer_list { */ union { struct list_head buf_list; - struct { - struct page **buf_pages; - struct io_uring_buf_ring *buf_ring; - }; + struct io_uring_buf_ring *buf_ring; }; __u16 bgid; @@ -35,6 +30,8 @@ struct io_buffer_list { __u16 mask; __u16 flags; + + struct io_mapped_region region; }; struct io_buffer { @@ -81,9 +78,8 @@ void __io_put_kbuf(struct io_kiocb *req, int len, unsigned issue_flags); bool io_kbuf_recycle_legacy(struct io_kiocb *req, unsigned issue_flags); -struct io_buffer_list *io_pbuf_get_bl(struct io_ring_ctx *ctx, - unsigned long bgid); -int io_pbuf_mmap(struct file *file, struct vm_area_struct *vma); +struct io_mapped_region *io_pbuf_get_region(struct io_ring_ctx *ctx, + unsigned int bgid); static inline bool io_kbuf_recycle_ring(struct io_kiocb *req) { diff --git a/io_uring/memmap.c b/io_uring/memmap.c index 88428a8dc3bc..22c3a20bd52b 100644 --- a/io_uring/memmap.c +++ b/io_uring/memmap.c @@ -36,88 +36,6 @@ static void *io_mem_alloc_compound(struct page **pages, int nr_pages, return page_address(page); } -static void *io_mem_alloc_single(struct page **pages, int nr_pages, size_t size, - gfp_t gfp) -{ - void *ret; - int i; - - for (i = 0; i < nr_pages; i++) { - pages[i] = alloc_page(gfp); - if (!pages[i]) - goto err; - } - - ret = vmap(pages, nr_pages, VM_MAP, PAGE_KERNEL); - if (ret) - return ret; -err: - while (i--) - put_page(pages[i]); - return ERR_PTR(-ENOMEM); -} - -void *io_pages_map(struct page ***out_pages, unsigned short *npages, - size_t size) -{ - gfp_t gfp = GFP_KERNEL_ACCOUNT | __GFP_ZERO | __GFP_NOWARN; - struct page **pages; - int nr_pages; - void *ret; - - nr_pages = (size + PAGE_SIZE - 1) >> PAGE_SHIFT; - pages = kvmalloc_array(nr_pages, sizeof(struct page *), gfp); - if (!pages) - return ERR_PTR(-ENOMEM); - - ret = io_mem_alloc_compound(pages, nr_pages, size, gfp); - if (!IS_ERR(ret)) - goto done; - - ret = io_mem_alloc_single(pages, nr_pages, size, gfp); - if (!IS_ERR(ret)) { -done: - *out_pages = pages; - *npages = nr_pages; - return ret; - } - - kvfree(pages); - *out_pages = NULL; - *npages = 0; - return ret; -} - -void io_pages_unmap(void *ptr, struct page ***pages, unsigned short *npages, - bool put_pages) -{ - bool do_vunmap = false; - - if (!ptr) - return; - - if (put_pages && *npages) { - struct page **to_free = *pages; - int i; - - /* - * Only did vmap for the non-compound multiple page case. - * For the compound page, we just need to put the head. - */ - if (PageCompound(to_free[0])) - *npages = 1; - else if (*npages > 1) - do_vunmap = true; - for (i = 0; i < *npages; i++) - put_page(to_free[i]); - } - if (do_vunmap) - vunmap(ptr); - kvfree(*pages); - *pages = NULL; - *npages = 0; -} - struct page **io_pin_pages(unsigned long uaddr, unsigned long len, int *npages) { unsigned long start, end, nr_pages; @@ -375,16 +293,14 @@ static void *io_uring_validate_mmap_request(struct file *file, loff_t pgoff, return ERR_PTR(-EFAULT); return ctx->sq_sqes; case IORING_OFF_PBUF_RING: { - struct io_buffer_list *bl; + struct io_mapped_region *region; unsigned int bgid; - void *ptr; bgid = (offset & ~IORING_OFF_MMAP_MASK) >> IORING_OFF_PBUF_SHIFT; - bl = io_pbuf_get_bl(ctx, bgid); - if (IS_ERR(bl)) - return bl; - ptr = bl->buf_ring; - return ptr; + region = io_pbuf_get_region(ctx, bgid); + if (!region) + return ERR_PTR(-EINVAL); + return io_region_validate_mmap(ctx, region); } case IORING_MAP_OFF_PARAM_REGION: return io_region_validate_mmap(ctx, &ctx->param_region); @@ -393,15 +309,6 @@ static void *io_uring_validate_mmap_request(struct file *file, loff_t pgoff, return ERR_PTR(-EINVAL); } -int io_uring_mmap_pages(struct io_ring_ctx *ctx, struct vm_area_struct *vma, - struct page **pages, int npages) -{ - unsigned long nr_pages = npages; - - vm_flags_set(vma, VM_DONTEXPAND); - return vm_insert_pages(vma, vma->vm_start, pages, &nr_pages); -} - #ifdef CONFIG_MMU __cold int io_uring_mmap(struct file *file, struct vm_area_struct *vma) @@ -425,8 +332,17 @@ __cold int io_uring_mmap(struct file *file, struct vm_area_struct *vma) return io_region_mmap(ctx, &ctx->ring_region, vma, page_limit); case IORING_OFF_SQES: return io_region_mmap(ctx, &ctx->sq_region, vma, UINT_MAX); - case IORING_OFF_PBUF_RING: - return io_pbuf_mmap(file, vma); + case IORING_OFF_PBUF_RING: { + struct io_mapped_region *region; + unsigned int bgid; + + bgid = (offset & ~IORING_OFF_MMAP_MASK) >> IORING_OFF_PBUF_SHIFT; + region = io_pbuf_get_region(ctx, bgid); + if (!region) + return -EINVAL; + + return io_region_mmap(ctx, region, vma, UINT_MAX); + } case IORING_MAP_OFF_PARAM_REGION: return io_region_mmap(ctx, &ctx->param_region, vma, UINT_MAX); } diff --git a/io_uring/memmap.h b/io_uring/memmap.h index 7395996eb353..c898dcba2b4e 100644 --- a/io_uring/memmap.h +++ b/io_uring/memmap.h @@ -4,13 +4,6 @@ #define IORING_MAP_OFF_PARAM_REGION 0x20000000ULL struct page **io_pin_pages(unsigned long ubuf, unsigned long len, int *npages); -int io_uring_mmap_pages(struct io_ring_ctx *ctx, struct vm_area_struct *vma, - struct page **pages, int npages); - -void *io_pages_map(struct page ***out_pages, unsigned short *npages, - size_t size); -void io_pages_unmap(void *ptr, struct page ***pages, unsigned short *npages, - bool put_pages); #ifndef CONFIG_MMU unsigned int io_uring_nommu_mmap_capabilities(struct file *file); From patchwork Sun Nov 24 21:12:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13884206 Received: from mail-wm1-f41.google.com (mail-wm1-f41.google.com [209.85.128.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3A4D0165EFC for ; Sun, 24 Nov 2024 21:12:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482746; cv=none; b=MJxR54CEmk7HnXkUR03L3W1i0AUK9ExNkpD9/uJUfi4JRTuATJwRHQg4PtWCJ9r5QwX49PoJoYPhW2Pq3zc2SAH6TERQlaZzqhi/aByq3OhSASlNdOllrfq8DUaVHRW2JzhZM4+RQTrwvUiQ9dvd858IULAJXaoH8OOHxvtmcRQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732482746; c=relaxed/simple; bh=fPhRBnMHePxLkkKrqQUKSYByjBb44pQDHEKtxX1xlQc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=eMRMET7nxvV+XnzD/VaFZqBZvqiVgThfrRT+jIYy+uIDq5upjmHFzA2ab77y6UeLLRFXV3XzC5fOucprvcW2a2mkp9ZbE0rSkGI51/0d8N44Ts4EZjNxcLUTOClaH3TvOrJskWiQfxTpNgBP2625N8cEVSDND3bkfM1ax2YvvkE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Ym+cWTQH; arc=none smtp.client-ip=209.85.128.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Ym+cWTQH" Received: by mail-wm1-f41.google.com with SMTP id 5b1f17b1804b1-43162cf1eaaso48598485e9.0 for ; Sun, 24 Nov 2024 13:12:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732482742; x=1733087542; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=n+ATPVfD8rX3FECsgDLHC0j20SKGDTPCcG71uC10WDA=; b=Ym+cWTQHfdNC3GOzz7w5m1JbFt2E3IHQDodFaWtvK3VblTdndhIple0jFsUw4fxmli ucS19seo3Y+zGPup9MZ6B8Pqu6k71HdFEX4Kwiy5s9AlfwistUXyAR+GVGuukkDc7TH2 2We6+bKJMsrOKszjTcz5dabm2lyTRKL6wBsQdxrt0VKWajjcTQWqvMtN7QBrU/0h+C+G jd4O2lSWlSsIsuGmBaI7fPxCCWGkDW8VneaowEvHqlEWhl6Zb3OlPE5r4gDaFocTjaNU ICzVNfVMocGP0Ys0tYP0QaXwsYtSfYZwlL7+VLMqOueAKxOJlY29GnH5AFbem4mSScwR td1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732482742; x=1733087542; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=n+ATPVfD8rX3FECsgDLHC0j20SKGDTPCcG71uC10WDA=; b=JOePySmOjIegyGThW9nOr4oD+Gxio+u6MFrN19pve4UhEv32wG09FnQdCBMSaDBA+m WbakMoNJUHxm6BpD3E5KpDpRtv85+CYgm5odIWVPHmzHK8B7xvPoOx9ZBKLZ8wnwl1L3 AoaTVtizkZjCKCT6OhvEAuliWdARYeqWCkX7KifRVivSyFWnFU6+OrY9RJp2FyVWxHxw +l24URgop1hl3dsm3PnYAC+RHeJM5HudK8lH0n3aI0p51ebsXbFPnOBrOpo0au2gRsQE xF4jd/QScv0wuxR698hGPZMIMCctE70zr28dUqAk1XPGCcY7N+ucWkngniBJkxJDAbXL LE/Q== X-Gm-Message-State: AOJu0YzvCfz/WwANc8LqXtX0PPvuEh68of1CrK4iYCMD0lK/b9YzJ/Kb P7Ho2JTksme1etPrMFRS7JikbxdHlvXibl2pgNrboRLdXyUHqmi2RKgxpA== X-Gm-Gg: ASbGnctvHZILfvMPxLOce2rTAFcSevVeZEFaLhw7rYVTii4Yv3988y1B2oT6XjC36FK LbWdHrSmWtIItMVzsLqVEcueVA98IarkiqQICS0C6jvYgtRLS8cWGsS1lj1OpY8QHf/VYBzacW6 2sTs4rDGG0F4YPQFNwwDhb2vNEQhqCfKuN6UEa/EO6zNVSrA+AkZXcjHFGskcIcRxgPVFdycfvc TbNtjbkrEdwWnXwDWm9ZccVBcjfUyiNFkgmrT5cZ4EXdPcxop8Yw/rhAQ3E3y0= X-Google-Smtp-Source: AGHT+IHiBDGpcmAYpzc6fSJLKpY+GyS3fogbrN2uXQnJCVNvfBs3+U2pd+DoE7/RMfGJ/e4DZtEwLQ== X-Received: by 2002:a05:600c:4fcc:b0:42c:ba83:3f00 with SMTP id 5b1f17b1804b1-433ce410337mr95457605e9.1.1732482742484; Sun, 24 Nov 2024 13:12:22 -0800 (PST) Received: from 127.0.0.1localhost ([85.255.235.224]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-432f643b299sm132733745e9.0.2024.11.24.13.12.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 24 Nov 2024 13:12:22 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v2 18/18] io_uring/memmap: unify io_uring mmap'ing code Date: Sun, 24 Nov 2024 21:12:35 +0000 Message-ID: <8cc1f2443b5523674f3662a8654212694c756f9e.1732481694.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 All mapped memory is now backed by regions and we can unify and clean up io_region_validate_mmap() and io_uring_mmap(). Extract a function looking up a region, the rest of the handling should be generic and just needs the region. There is one more ring type specific code, i.e. the mmaping size truncation quirk for IORING_OFF_[S,C]Q_RING, which is left as is. Signed-off-by: Pavel Begunkov --- io_uring/kbuf.c | 3 -- io_uring/memmap.c | 82 ++++++++++++++++++----------------------------- 2 files changed, 32 insertions(+), 53 deletions(-) diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index f77d14e4d598..a3847154bd99 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -748,8 +748,5 @@ struct io_mapped_region *io_pbuf_get_region(struct io_ring_ctx *ctx, bl = xa_load(&ctx->io_bl_xa, bgid); if (!bl || !(bl->flags & IOBL_BUF_RING)) return NULL; - if (WARN_ON_ONCE(!io_region_is_set(&bl->region))) - return NULL; - return &bl->region; } diff --git a/io_uring/memmap.c b/io_uring/memmap.c index 22c3a20bd52b..4b4bc4233e5a 100644 --- a/io_uring/memmap.c +++ b/io_uring/memmap.c @@ -246,6 +246,27 @@ int io_create_region_mmap_safe(struct io_ring_ctx *ctx, struct io_mapped_region return 0; } +static struct io_mapped_region *io_mmap_get_region(struct io_ring_ctx *ctx, + loff_t pgoff) +{ + loff_t offset = pgoff << PAGE_SHIFT; + unsigned int bgid; + + switch (offset & IORING_OFF_MMAP_MASK) { + case IORING_OFF_SQ_RING: + case IORING_OFF_CQ_RING: + return &ctx->ring_region; + case IORING_OFF_SQES: + return &ctx->sq_region; + case IORING_OFF_PBUF_RING: + bgid = (offset & ~IORING_OFF_MMAP_MASK) >> IORING_OFF_PBUF_SHIFT; + return io_pbuf_get_region(ctx, bgid); + case IORING_MAP_OFF_PARAM_REGION: + return &ctx->param_region; + } + return NULL; +} + static void *io_region_validate_mmap(struct io_ring_ctx *ctx, struct io_mapped_region *mr) { @@ -270,43 +291,17 @@ static int io_region_mmap(struct io_ring_ctx *ctx, return vm_insert_pages(vma, vma->vm_start, mr->pages, &nr_pages); } + static void *io_uring_validate_mmap_request(struct file *file, loff_t pgoff, size_t sz) { struct io_ring_ctx *ctx = file->private_data; - loff_t offset = pgoff << PAGE_SHIFT; - - switch ((pgoff << PAGE_SHIFT) & IORING_OFF_MMAP_MASK) { - case IORING_OFF_SQ_RING: - case IORING_OFF_CQ_RING: - /* Don't allow mmap if the ring was setup without it */ - if (ctx->flags & IORING_SETUP_NO_MMAP) - return ERR_PTR(-EINVAL); - if (!ctx->rings) - return ERR_PTR(-EFAULT); - return ctx->rings; - case IORING_OFF_SQES: - /* Don't allow mmap if the ring was setup without it */ - if (ctx->flags & IORING_SETUP_NO_MMAP) - return ERR_PTR(-EINVAL); - if (!ctx->sq_sqes) - return ERR_PTR(-EFAULT); - return ctx->sq_sqes; - case IORING_OFF_PBUF_RING: { - struct io_mapped_region *region; - unsigned int bgid; - - bgid = (offset & ~IORING_OFF_MMAP_MASK) >> IORING_OFF_PBUF_SHIFT; - region = io_pbuf_get_region(ctx, bgid); - if (!region) - return ERR_PTR(-EINVAL); - return io_region_validate_mmap(ctx, region); - } - case IORING_MAP_OFF_PARAM_REGION: - return io_region_validate_mmap(ctx, &ctx->param_region); - } + struct io_mapped_region *region; - return ERR_PTR(-EINVAL); + region = io_mmap_get_region(ctx, pgoff); + if (!region) + return ERR_PTR(-EINVAL); + return io_region_validate_mmap(ctx, region); } #ifdef CONFIG_MMU @@ -316,7 +311,8 @@ __cold int io_uring_mmap(struct file *file, struct vm_area_struct *vma) struct io_ring_ctx *ctx = file->private_data; size_t sz = vma->vm_end - vma->vm_start; long offset = vma->vm_pgoff << PAGE_SHIFT; - unsigned int page_limit; + unsigned int page_limit = UINT_MAX; + struct io_mapped_region *region; void *ptr; guard(mutex)(&ctx->mmap_lock); @@ -329,25 +325,11 @@ __cold int io_uring_mmap(struct file *file, struct vm_area_struct *vma) case IORING_OFF_SQ_RING: case IORING_OFF_CQ_RING: page_limit = (sz + PAGE_SIZE - 1) >> PAGE_SHIFT; - return io_region_mmap(ctx, &ctx->ring_region, vma, page_limit); - case IORING_OFF_SQES: - return io_region_mmap(ctx, &ctx->sq_region, vma, UINT_MAX); - case IORING_OFF_PBUF_RING: { - struct io_mapped_region *region; - unsigned int bgid; - - bgid = (offset & ~IORING_OFF_MMAP_MASK) >> IORING_OFF_PBUF_SHIFT; - region = io_pbuf_get_region(ctx, bgid); - if (!region) - return -EINVAL; - - return io_region_mmap(ctx, region, vma, UINT_MAX); - } - case IORING_MAP_OFF_PARAM_REGION: - return io_region_mmap(ctx, &ctx->param_region, vma, UINT_MAX); + break; } - return -EINVAL; + region = io_mmap_get_region(ctx, vma->vm_pgoff); + return io_region_mmap(ctx, region, vma, page_limit); } unsigned long io_uring_get_unmapped_area(struct file *filp, unsigned long addr,