From patchwork Wed Nov 20 23:33:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13881494 Received: from mail-ej1-f48.google.com (mail-ej1-f48.google.com [209.85.218.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B23481BC07B for ; Wed, 20 Nov 2024 23:33:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732145589; cv=none; b=rRXX2Vn0rbIm+5YS6j4mXO4fOTVcDlbsEkXKDRegNaBzQ+xtaa9wGPeTx5fGZxrrSaK3ISuyfs5MTrcsF+heUA3g9+7Ss6WGpJ4mVdicU6fAxZSv68REm5pOm4IkVW76RomHJMcTs9YMAdZTuWZhO3Nyf6ieF2XzaEkPFHrqUXQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732145589; c=relaxed/simple; bh=UPTbRG9neEhUTCuJVFBoQHbF7x+O860/DGHGgPzDdYI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=sS5QREKlTVhgKO/1zn2tNE3dFAGVNEQFxIsD6RI9OiIFQFSJ9Dxg4L75+Fyg2K8jYX2qYitbbabBN32QdhA50yCxiafioAMiLnMsPO1cPmQ9+crFC5DYs8IzrxjXaBzLN6h5B8HzQY9Hwrz8+n3ZWDMGxHHqxQN0gYXoHHHzL94= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=EjSJqGDY; arc=none smtp.client-ip=209.85.218.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="EjSJqGDY" Received: by mail-ej1-f48.google.com with SMTP id a640c23a62f3a-a9a6acac4c3so53726766b.0 for ; Wed, 20 Nov 2024 15:33:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732145586; x=1732750386; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=d1Ts0VuPNUivf/LvKiwdXOLn5cp6FT5YlAICwjVUTc4=; b=EjSJqGDYJ+vXpgq4nf9Sr/QlDmx5L+192QF1Mow0t7u8csPxbs9oe4q5akIAVW2KYC PQuHXyb2N8LNLBc/XoAuGABWNijG8ZvvuzJSfAoDrRMlnrxKdE8t9UBIQGcddYTZnfHq 5eMAoJUseRL8cXic5mYdKrhML0Hc/xoRlEZEv2/rtwjnqj5UHBd1+iOBWfnRT79SbcpX P64ZbZdJuU6RyJK2RWhQwZZ8fsGdvAFEJlJStBRRD4uxOjeF9FUzNGFCBfU1I94x0iWJ Q6FszRin3H29zIX4jY8tdo5KJCEcj8Xt2zeZJ1XCws3i9Rcp4IImHjEau2iT94p7+wrX qcvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732145586; x=1732750386; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=d1Ts0VuPNUivf/LvKiwdXOLn5cp6FT5YlAICwjVUTc4=; b=FPEewt8oZwzQ+G6UACjz2mRbXSiW6xDe8jABidu6AYPOIdTQEJL/hPC92iSsGo2th4 ojjtREQG5wPfcFYXafpzFQQf/Ik2T2Gl9T/cpOsFWG2hsTZinRpTiv3IDAm9IRn46mxn myel/G+xAIm9W1aUms/hTgxxTakeIREBu9NHCdS7iut7PcuefNNi0lVf61TwUKlnhJNB 5jRsmK8V/cFXfVnzCzNNX5IUTSA7Jciwod7q4ixmxBVtV0FV0eplRU5wB4prHGBl4Sha xjlTpjuMkurL2sYz+EN7l6MtoJuw/hLOUs5XXG8nUd9V/Y0Z1Qqej0zCQG9kKaUTprF4 lNsQ== X-Gm-Message-State: AOJu0Yy7ZzdEseJNeWAvTFSUrNaDPnsFY0lOAfA0VOtFnvjzMJsjWM4w q2FIQAqduNWK7vqTve4+R2rALrStU7bfdnnrviYCad1afqrmEjUK0IqMPA== X-Gm-Gg: ASbGncv04bIxqaIq2RAITmVgFLE5cg6+kXnQ6KfDpILVu2QrkCN+n8xUGLFtlgysUpL A9BwDJvWoctBD7AxI6uertljbDNCbVUy41Iq986rt9Y0R/ieX6Y9pTAwkIZ4gsNupVrYz9ywUs9 lMpsMAjCCOrBGDeSfnBRoiT0rixt9XmC8KxXGumBRscisvJMZMpeyi2ZTXSGsEUatfDwpuybqWf Wyo/92vUotzl5eusyg8bN/iW+afzucivCX2ODpzaYrLDrWp0oYodaqE9wCSehkW X-Google-Smtp-Source: AGHT+IG1/ExHUUYFnGg5SMZsUZmjMkTTrG8ySnGdqIAz5eTRFWeAWakAX7EwZ8PDawjlj9laal2Qig== X-Received: by 2002:a17:907:1c14:b0:a9a:8034:3644 with SMTP id a640c23a62f3a-aa4dd5517c4mr414818566b.14.1732145585436; Wed, 20 Nov 2024 15:33:05 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.141.165]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-aa4f415372bsm12166066b.13.2024.11.20.15.33.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Nov 2024 15:33:05 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH 01/11] io_uring: rename ->resize_lock Date: Wed, 20 Nov 2024 23:33:24 +0000 Message-ID: <6b0a570e4685d1a66bec720dd1fd3fb735a0844c.1732144783.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 ->resize_lock is used for resizing rings, but it's a good idea to reuse it in other cases as well. Rename it into mmap_lock as it's protects from races with mmap. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 2 +- io_uring/io_uring.c | 2 +- io_uring/memmap.c | 6 +++--- io_uring/register.c | 8 ++++---- 4 files changed, 9 insertions(+), 9 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index aa5f5ea98076..ac7b2b6484a9 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -422,7 +422,7 @@ struct io_ring_ctx { * side will need to grab this lock, to prevent either side from * being run concurrently with the other. */ - struct mutex resize_lock; + struct mutex mmap_lock; /* * If IORING_SETUP_NO_MMAP is used, then the below holds diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index da8fd460977b..d565b1589951 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -350,7 +350,7 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) INIT_WQ_LIST(&ctx->submit_state.compl_reqs); INIT_HLIST_HEAD(&ctx->cancelable_uring_cmd); io_napi_init(ctx); - mutex_init(&ctx->resize_lock); + mutex_init(&ctx->mmap_lock); return ctx; diff --git a/io_uring/memmap.c b/io_uring/memmap.c index 3d71756bc598..771a57a4a16b 100644 --- a/io_uring/memmap.c +++ b/io_uring/memmap.c @@ -322,7 +322,7 @@ __cold int io_uring_mmap(struct file *file, struct vm_area_struct *vma) unsigned int npages; void *ptr; - guard(mutex)(&ctx->resize_lock); + guard(mutex)(&ctx->mmap_lock); ptr = io_uring_validate_mmap_request(file, vma->vm_pgoff, sz); if (IS_ERR(ptr)) @@ -358,7 +358,7 @@ unsigned long io_uring_get_unmapped_area(struct file *filp, unsigned long addr, if (addr) return -EINVAL; - guard(mutex)(&ctx->resize_lock); + guard(mutex)(&ctx->mmap_lock); ptr = io_uring_validate_mmap_request(filp, pgoff, len); if (IS_ERR(ptr)) @@ -408,7 +408,7 @@ unsigned long io_uring_get_unmapped_area(struct file *file, unsigned long addr, struct io_ring_ctx *ctx = file->private_data; void *ptr; - guard(mutex)(&ctx->resize_lock); + guard(mutex)(&ctx->mmap_lock); ptr = io_uring_validate_mmap_request(file, pgoff, len); if (IS_ERR(ptr)) diff --git a/io_uring/register.c b/io_uring/register.c index 1e99c783abdf..ba61697d7a53 100644 --- a/io_uring/register.c +++ b/io_uring/register.c @@ -486,15 +486,15 @@ static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg) } /* - * We'll do the swap. Grab the ctx->resize_lock, which will exclude + * We'll do the swap. Grab the ctx->mmap_lock, which will exclude * any new mmap's on the ring fd. Clear out existing mappings to prevent * mmap from seeing them, as we'll unmap them. Any attempt to mmap * existing rings beyond this point will fail. Not that it could proceed * at this point anyway, as the io_uring mmap side needs go grab the - * ctx->resize_lock as well. Likewise, hold the completion lock over the + * ctx->mmap_lock as well. Likewise, hold the completion lock over the * duration of the actual swap. */ - mutex_lock(&ctx->resize_lock); + mutex_lock(&ctx->mmap_lock); spin_lock(&ctx->completion_lock); o.rings = ctx->rings; ctx->rings = NULL; @@ -561,7 +561,7 @@ static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg) ret = 0; out: spin_unlock(&ctx->completion_lock); - mutex_unlock(&ctx->resize_lock); + mutex_unlock(&ctx->mmap_lock); io_register_free_rings(&p, to_free); if (ctx->sq_data) From patchwork Wed Nov 20 23:33:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13881495 Received: from mail-ej1-f49.google.com (mail-ej1-f49.google.com [209.85.218.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B33BA1C1F12 for ; Wed, 20 Nov 2024 23:33:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732145590; cv=none; b=jNROM7GduA7gTni/9Bf4K5JhSF2QhwN/f/CwNtENbs+5R2dsyoMDagSD3SBHJjTldG9IWCwze1+bfuxln6vThsDzTM7F9b4aa1ODqypfty5M2S3ltgeNR6TNNKu/291S9FqAdy68ujCY6wF3xKMZjxDmue7MbFRQR9IrGySujP4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732145590; c=relaxed/simple; bh=1jmb/YGUScs5yrYsukVscAuiJXOxz0cRSODGRVeVGzI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FSeSV4+H2elIQx1WhnsvEDkk/sB1Ig9c0l/JzBD+/sUHR5dmpzxD0997Nqo8eBvzwhMeFzO7rzQlOqTztbCbMb5K7s+m2RcFvH1Al63DPO8Yf6dLWwkafXKes/uvc8EcHQali4sPWY+vbOgigOOlSJLjiY8dZTm6GdJdbFUD1/E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=aKrlgvtd; arc=none smtp.client-ip=209.85.218.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="aKrlgvtd" Received: by mail-ej1-f49.google.com with SMTP id a640c23a62f3a-a9ed7d8c86cso55750766b.2 for ; Wed, 20 Nov 2024 15:33:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732145587; x=1732750387; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gaPvSNc+wa5krf/6t9pRuh8xeS0CHz0dlFRi5Cy4Ht8=; b=aKrlgvtdgH2K1WA7ahLerSEHx1UKpj+Zrv/f37k71mrEPdswfb3dEl+e8YHBpjHF7U 2L1naNqlVpExDo8IRR5wUtJEBCqqc3LI5fjJdTbu2LhpQepvkGrIiXO+Ko9o2EoM8syA V7pGd2xfhEgq7rhH4TEShTHVYyLNKNsHZchRXuR76DEWzsbBUMHgOGNX9hkFHe93th1K F8+IuU2rlDfZ3lepRIsx1EJaSVpogVo1dVrA45yLXfEVXEUNydz3WjiTFyiJ93dTiydu zsdlWu4GIOG7CZYK1PupI/mAvqQGkby4GGav73jCjB/lNkAhQIuNmJRJrNpP9XGg5Xsi T4nw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732145587; x=1732750387; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gaPvSNc+wa5krf/6t9pRuh8xeS0CHz0dlFRi5Cy4Ht8=; b=QFT0y96+LQlWznGP+EOtBRUv1Mqe5XWw/xhYXN2eDBujP5LJLF+FFde2g6GKrQgbk5 DDidRz2gBw6Bx4HP4C+CQsMe4WJBVFa03suqBSA0IwELXbZRSeLWDOm8X0sN1doGQCLD vp1PyalW+LVAsmDQN9JRZID9c2i315BWSNdX9z2CG99WclqNs6/E/Cn6QYj1fd1YEXl0 tTIAzTrf0IY/K5lVhyUIcSOLIk2UuUVl6wtRlW9j8PzWQYIhnXkMwRrkU/swmJH47h8V Lb6DXbbfjVHV/4SiX68aJ3zjminT9hywBgF8WOU1VywiAesfeUjgI6WIJ7nOykSj8Nqb xBDA== X-Gm-Message-State: AOJu0YygUmnwkTJJScEEfEfA6NOd5OjkfRJ2vqIB7yqqWch6Sp2g+sXM Pxka10iwzZE6LRHA+QeICOTaVnOAFvKJKJislNFe9/qoR8KmHVfueYIbGg== X-Google-Smtp-Source: AGHT+IGBSQEtAl8P79cU/or/f2yAqVDkAV5rooLl9Yl4zSiub5mFOYSn4T4qDa2p4OX/f6pwTycqBA== X-Received: by 2002:a17:906:c10a:b0:a9d:e1d6:41fc with SMTP id a640c23a62f3a-aa4dd752639mr439733766b.53.1732145586555; Wed, 20 Nov 2024 15:33:06 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.141.165]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-aa4f415372bsm12166066b.13.2024.11.20.15.33.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Nov 2024 15:33:06 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH 02/11] io_uring/rsrc: export io_check_coalesce_buffer Date: Wed, 20 Nov 2024 23:33:25 +0000 Message-ID: <8085256a8f07fc2c11c63c810a56b8fd8fe7a5eb.1732144783.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 io_try_coalesce_buffer() is a useful helper collecting useful info about a set of pages, I want to reuse it for analysing ring/etc. mappings. I don't need the entire thing and only interested if it can be coalesced into a single page, but that's better than duplicating the parsing. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 22 ++++++++++++---------- io_uring/rsrc.h | 4 ++++ 2 files changed, 16 insertions(+), 10 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index adaae8630932..e51e5ddae728 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -626,11 +626,12 @@ static int io_buffer_account_pin(struct io_ring_ctx *ctx, struct page **pages, return ret; } -static bool io_do_coalesce_buffer(struct page ***pages, int *nr_pages, - struct io_imu_folio_data *data, int nr_folios) +static bool io_coalesce_buffer(struct page ***pages, int *nr_pages, + struct io_imu_folio_data *data) { struct page **page_array = *pages, **new_array = NULL; int nr_pages_left = *nr_pages, i, j; + int nr_folios = data->nr_folios; /* Store head pages only*/ new_array = kvmalloc_array(nr_folios, sizeof(struct page *), @@ -667,15 +668,14 @@ static bool io_do_coalesce_buffer(struct page ***pages, int *nr_pages, return true; } -static bool io_try_coalesce_buffer(struct page ***pages, int *nr_pages, - struct io_imu_folio_data *data) +bool io_check_coalesce_buffer(struct page **page_array, int nr_pages, + struct io_imu_folio_data *data) { - struct page **page_array = *pages; struct folio *folio = page_folio(page_array[0]); unsigned int count = 1, nr_folios = 1; int i; - if (*nr_pages <= 1) + if (nr_pages <= 1) return false; data->nr_pages_mid = folio_nr_pages(folio); @@ -687,7 +687,7 @@ static bool io_try_coalesce_buffer(struct page ***pages, int *nr_pages, * Check if pages are contiguous inside a folio, and all folios have * the same page count except for the head and tail. */ - for (i = 1; i < *nr_pages; i++) { + for (i = 1; i < nr_pages; i++) { if (page_folio(page_array[i]) == folio && page_array[i] == page_array[i-1] + 1) { count++; @@ -715,7 +715,8 @@ static bool io_try_coalesce_buffer(struct page ***pages, int *nr_pages, if (nr_folios == 1) data->nr_pages_head = count; - return io_do_coalesce_buffer(pages, nr_pages, data, nr_folios); + data->nr_folios = nr_folios; + return true; } static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, @@ -729,7 +730,7 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, size_t size; int ret, nr_pages, i; struct io_imu_folio_data data; - bool coalesced; + bool coalesced = false; if (!iov->iov_base) return NULL; @@ -749,7 +750,8 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, } /* If it's huge page(s), try to coalesce them into fewer bvec entries */ - coalesced = io_try_coalesce_buffer(&pages, &nr_pages, &data); + if (io_check_coalesce_buffer(pages, nr_pages, &data)) + coalesced = io_coalesce_buffer(&pages, &nr_pages, &data); imu = kvmalloc(struct_size(imu, bvec, nr_pages), GFP_KERNEL); if (!imu) diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 7a4668deaa1a..c8b093584461 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -40,6 +40,7 @@ struct io_imu_folio_data { /* For non-head/tail folios, has to be fully included */ unsigned int nr_pages_mid; unsigned int folio_shift; + unsigned int nr_folios; }; struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx, int type); @@ -66,6 +67,9 @@ int io_register_rsrc_update(struct io_ring_ctx *ctx, void __user *arg, int io_register_rsrc(struct io_ring_ctx *ctx, void __user *arg, unsigned int size, unsigned int type); +bool io_check_coalesce_buffer(struct page **page_array, int nr_pages, + struct io_imu_folio_data *data); + static inline struct io_rsrc_node *io_rsrc_node_lookup(struct io_rsrc_data *data, int index) { From patchwork Wed Nov 20 23:33:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13881496 Received: from mail-ej1-f42.google.com (mail-ej1-f42.google.com [209.85.218.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BCE6B1C7B63 for ; Wed, 20 Nov 2024 23:33:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732145591; cv=none; b=UgKrOUxCGbrOALSLSGA7FIkVQAkUAl8PGV8vJ8weSBDBuJcd+R2fIiHwn067TgMxdI6J2wgV7h7zatxOoGSqwmzUR8/Exn8rnHgzbMzxI7617RB8kFbardbXEw55JclUOcjkn905J83Y4faf4+jNOlgDlLc9NmFSrFJGo89LHQE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732145591; c=relaxed/simple; bh=3guHR+zYJfa16yyG7x/vkhKd1mBgkHtCczZEsPnZxYQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=V82/eEXfWE1iS89QXnNyvEcm57He3KiI6r8pmgK1V4r10706z0GrCT+du1LEpdtcD554ff5dhOPBxkUEiVpHkh/WVUCK7kntSuEjcGfg+4s900e6tchsbXFMWet9CA2Jt+1NPgGgEq1R4PW/lXCT2keUKxvSu2x6mRr8H5Lfgtk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Hb9CFjvS; arc=none smtp.client-ip=209.85.218.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Hb9CFjvS" Received: by mail-ej1-f42.google.com with SMTP id a640c23a62f3a-a99eb8b607aso28532166b.2 for ; Wed, 20 Nov 2024 15:33:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732145588; x=1732750388; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wymql0mU7Zf6dtk9Kzjz2Ao5/EDbYR0HVIjkz1dGWIE=; b=Hb9CFjvSs0JI1pziuG0Txvx+kuywNcJMmW0i0nhNsGlzam+KTAGTR5n5NGXUKuCHwe J2e8+J2r2PF4I/43pQBxdsbvRCjVjlz6FdHd7qhhRTFSUzL5IP4Yzb5Be0gAsejXG7ky 9a60CmUwel9FMNbkf1zI/I/3OeoraaX7wttoldOWcgOYuDgfg/FfPxCa4kEZa9cBlXLC IBNHmbaRXH9dCf+u3OP+qxk9Q7cgVepVxOmY+c6WtUDEi/UkCZFAip9kNPRnnMVXHv3l 63L3hg/MxBWQrc1HLhDuaMjVtSXNIXCSCDzCJP9alnWUdJ8g72wjehR5JNSkQfltX5aE 8S8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732145588; x=1732750388; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wymql0mU7Zf6dtk9Kzjz2Ao5/EDbYR0HVIjkz1dGWIE=; b=retT+jZOlU16KJOdI9BXeDapFgj3jvx+UZC7i3XboeBBp+1hl53SZAmvg7bYSf49F7 yVWeTvyxiA2xlOr2hLb956IPQbLmFtH9NLU1k7P3Qb+sdGi5tGLacTRlPUeOs1Xg819y juBNHQFA4DGC2sVFzfHebaZh7fFR/zJ/GZWUOCJ79BtKQlYtJ1lmQihJYRe9FWvPcMsZ CvJoMfKSNoeGOyr7kt/3JS6SvjR/p19ZmIYe0EEtAe9JkKjmMO/PxpR7gc8LylrhFPwh tvoZMfnRB2IsK5sAF0AbR+l/Hg+5Q0auZFK98zgIHzPr5Q0nPkiJbgv6xPZ36sOXV4hR VwwQ== X-Gm-Message-State: AOJu0YxmZGth9g4M+8IJoy6REQ9Er/n1WcF3fNOf6QXY2x4eU7GZaoEC OifSUyx/mVE005IWWryBMS542PUE0WzKsXLONfAcXP4ULpGAGsJe1zkGQA== X-Gm-Gg: ASbGncuDweE4zYbRN56AXBKTwZbi4PXT069t/qQnUDruizRk84hZSoM7Y1qkB1JkWXQ FgJIQ2PRlXOiaG4H1JJ29rCMuUKHyIYQZNdUPn2pIaIT49StO5t9+eD0Ws2oFbg/sISMwx5wcJV CXuFYH+Xgd75NB5Tg88R1nAotIkv2bX+rzi426IjT6qOGEGHHykPB8gwhkAiP2//p6maYjoUo55 Rs4HnWPWO/lRhacKIkxtdfxr1K3s6LyKfh6GbEZJd1rt3nRQIEDujxb36/cGwP3 X-Google-Smtp-Source: AGHT+IEWZRZ+uFLV5ycaqb0WUe4bYSdUYNejW9UCFniUzvGb5F02ZUCVb4+N8QWlxN9FECzm+VKp8g== X-Received: by 2002:a17:906:7953:b0:a9e:d4a9:2c28 with SMTP id a640c23a62f3a-aa4dd748bd2mr405608466b.53.1732145587687; Wed, 20 Nov 2024 15:33:07 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.141.165]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-aa4f415372bsm12166066b.13.2024.11.20.15.33.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Nov 2024 15:33:07 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH 03/11] io_uring/memmap: add internal region flags Date: Wed, 20 Nov 2024 23:33:26 +0000 Message-ID: X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add internal flags for struct io_mapped_region, it will help to add more functionality while not bloating struct io_mapped_region. Use it to mark if the pointer needs to be vunmap'ed. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 5 +++-- io_uring/memmap.c | 13 +++++++++---- io_uring/memmap.h | 2 +- 3 files changed, 13 insertions(+), 7 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index ac7b2b6484a9..31b420b8ecd9 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -77,8 +77,9 @@ struct io_hash_table { struct io_mapped_region { struct page **pages; - void *vmap_ptr; - size_t nr_pages; + void *ptr; + unsigned nr_pages; + unsigned flags; }; /* diff --git a/io_uring/memmap.c b/io_uring/memmap.c index 771a57a4a16b..21353ea09b39 100644 --- a/io_uring/memmap.c +++ b/io_uring/memmap.c @@ -195,14 +195,18 @@ void *__io_uaddr_map(struct page ***pages, unsigned short *npages, return ERR_PTR(-ENOMEM); } +enum { + IO_REGION_F_VMAP = 1, +}; + void io_free_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr) { if (mr->pages) { unpin_user_pages(mr->pages, mr->nr_pages); kvfree(mr->pages); } - if (mr->vmap_ptr) - vunmap(mr->vmap_ptr); + if ((mr->flags & IO_REGION_F_VMAP) && mr->ptr) + vunmap(mr->ptr); if (mr->nr_pages && ctx->user) __io_unaccount_mem(ctx->user, mr->nr_pages); @@ -218,7 +222,7 @@ int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, void *vptr; u64 end; - if (WARN_ON_ONCE(mr->pages || mr->vmap_ptr || mr->nr_pages)) + if (WARN_ON_ONCE(mr->pages || mr->ptr || mr->nr_pages)) return -EFAULT; if (memchr_inv(®->__resv, 0, sizeof(reg->__resv))) return -EINVAL; @@ -253,8 +257,9 @@ int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, } mr->pages = pages; - mr->vmap_ptr = vptr; + mr->ptr = vptr; mr->nr_pages = nr_pages; + mr->flags |= IO_REGION_F_VMAP; return 0; out_free: if (pages_accounted) diff --git a/io_uring/memmap.h b/io_uring/memmap.h index f361a635b6c7..2096a8427277 100644 --- a/io_uring/memmap.h +++ b/io_uring/memmap.h @@ -28,7 +28,7 @@ int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, static inline void *io_region_get_ptr(struct io_mapped_region *mr) { - return mr->vmap_ptr; + return mr->ptr; } static inline bool io_region_is_set(struct io_mapped_region *mr) From patchwork Wed Nov 20 23:33:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13881497 Received: from mail-ej1-f45.google.com (mail-ej1-f45.google.com [209.85.218.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0587A1C1F12 for ; Wed, 20 Nov 2024 23:33:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732145592; cv=none; b=i7cQThuxJFQwT/hdXgUILgt2VwR0KEjUD+rxreOx6WOq67c/o7CatBpEYc7DQZKnKLbPg5CLv1E+s0GMWm9bXtZ2AHN32h/L751OGAkFZ/hd4urWwSGoSWRZBkC7gibHNlgO9/UYQo4HHowJq5tAh6OPaNdYpKjjPS3Ea8hgCJM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732145592; c=relaxed/simple; bh=1gdiLeCNSwVhK3nzWHMR4xFD8ClHJFNdvlyHODABro0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=IixObkvaf8dz9XopKtF34LtVWpZpvA3m0fPkiSesjVeHd7iqPu+uHoXUKhVUx+aVY63er18NZW2EcT6HusHxhDq5eEftUNfz1eahIFijwDARFAT0+rHUCqQpH8NUcosWs2X5yvYhMqs7dHKeW2B2Wx7D4fOkeRuH6xsvUoqBJ1Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=moZgZTU9; arc=none smtp.client-ip=209.85.218.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="moZgZTU9" Received: by mail-ej1-f45.google.com with SMTP id a640c23a62f3a-a9aa8895facso51524066b.2 for ; Wed, 20 Nov 2024 15:33:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732145589; x=1732750389; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=usEgNxoUAlzm7LvnyJowdrWjygvIn9+DGkjQ7Uu9NSo=; b=moZgZTU9sxZD//zIyJ55/YNE7uAHYiXLdZOC/lrcyOSOdc1JTmLaBqrSLxTDV2Adb4 NAj9JzmxhBra4cJoL5wCsASvneufzhT7IbXTI2ERxCDt4c37Az6CWT99gFDkgTybnwQy ZiUbWRUnRd1AF4XA1rPa4YQM2U/4sPKB5l/GdRcHSmUc2N0VH9OLaL7zRccS31elyUDt 3EsHk1YF4Bw2gmkgJmW7dBcduNbnOaYd7hb4IvM2/bxwst7xEVnhy/UzXXOmsnxz44Op 8G01OLcARPW7pvMOZ4+VYohEyRjDGG33ILxSN8bKw//puzD8iSiFOLp4To8mQvD0cblh Jotg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732145589; x=1732750389; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=usEgNxoUAlzm7LvnyJowdrWjygvIn9+DGkjQ7Uu9NSo=; b=fWS8AcSqtcu0W+BbqpoA98S0b+IhSQuUHqeAc9HofgjgiYq3q6thSzSbRIP101UsOV xmEPLXqjJtpIBrJbhN5n8ihmjFunMkVMlNaZaE74uMavmHdoPK7k2WCt6aiow78oBJH2 mmlmnU/nq6q0XWPgRUFolxBMjJYVGkU7Oj8+v9ocFap/xmE4/Yn0ujf29Vhal6gfSaR5 VsN94iMJDwNimub9PAK6+DpVU0qnv3JP1vcchgK2DEZkTiqGuJNFBrEea4dQb+ZUDIj3 0cwifcpYI92JCnv9Bx8DQSTu65vYoAtHCvJR53pPrpld9XPCuuw3lOxf26x3rZ7AdESi 3Mhg== X-Gm-Message-State: AOJu0Yw0QaiR4VxpInMQIFU7EDG4De5kMMiktPEMsHJzeix085/4C17g O4nFZ50p5fHdQmLgP9FPqpDFl4M9YvmA+R7lXnhWsn7Y8iwItBeRq7hWjw== X-Gm-Gg: ASbGncu9NdwL5lUn6ES/nH8xWXHtO4Lo2lbvr8S6JDI7a+yq0ITymnOWh7HG5Q7ARzu 8yXYEWcRJ/oQV9Wo3GAX2zi14iBfK7MsRG/a2ghT3Kkpd6YH6gZMMFaFKHJy+qBuYKIBoNNtRwL U+HFax0slBajgaiY8+BQlvLK3a8jhX7OrVPCBXUqx9qOBG76VmfpSxeH9QFH/xDz8hLs8lDCU0S 0IiT+URVwJ9z1cfiozL+IAvzZ8XSg/uf+VxMFC00SGTC+Kl6M5PV1CASrFcAWhT X-Google-Smtp-Source: AGHT+IECLULtaQx4yVcVrZwGwqgVD+QoYNTQm7pgrSh+VWpVMZZXO//20z6LIYoQgZKa/d1KB8xGMw== X-Received: by 2002:a17:907:6eab:b0:a9a:7f92:782 with SMTP id a640c23a62f3a-aa4dd74b2eemr476852066b.52.1732145588951; Wed, 20 Nov 2024 15:33:08 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.141.165]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-aa4f415372bsm12166066b.13.2024.11.20.15.33.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Nov 2024 15:33:08 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH 04/11] io_uring/memmap: flag regions with user pages Date: Wed, 20 Nov 2024 23:33:27 +0000 Message-ID: <0ff910962678b2b4d896d59f592d89181309e1b8.1732144783.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 In preparation to kernel allocated regions add a flag telling if the region contains user pinned pages or not. Signed-off-by: Pavel Begunkov --- io_uring/memmap.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/io_uring/memmap.c b/io_uring/memmap.c index 21353ea09b39..f76bee5a861a 100644 --- a/io_uring/memmap.c +++ b/io_uring/memmap.c @@ -197,12 +197,16 @@ void *__io_uaddr_map(struct page ***pages, unsigned short *npages, enum { IO_REGION_F_VMAP = 1, + IO_REGION_F_USER_PINNED = 2, }; void io_free_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr) { if (mr->pages) { - unpin_user_pages(mr->pages, mr->nr_pages); + if (mr->flags & IO_REGION_F_USER_PINNED) + unpin_user_pages(mr->pages, mr->nr_pages); + else + release_pages(mr->pages, mr->nr_pages); kvfree(mr->pages); } if ((mr->flags & IO_REGION_F_VMAP) && mr->ptr) @@ -259,7 +263,7 @@ int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, mr->pages = pages; mr->ptr = vptr; mr->nr_pages = nr_pages; - mr->flags |= IO_REGION_F_VMAP; + mr->flags |= IO_REGION_F_VMAP | IO_REGION_F_USER_PINNED; return 0; out_free: if (pages_accounted) From patchwork Wed Nov 20 23:33:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13881498 Received: from mail-ej1-f50.google.com (mail-ej1-f50.google.com [209.85.218.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 90DAD1C8306 for ; Wed, 20 Nov 2024 23:33:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732145593; cv=none; b=lOSb13rA20dz/85iU0tavh/dukzjSMudlGpdmj26fTPjaW0tlGdynRtF5FdH+ycnYPKOYoI+ng1R2mDUtzIszNkD7xvqRuIQSS/Z8DEo9Cx2C11FWvouLuxbHKT33Di+73tc3D2IngJX8FSE9suN3pOOhm1YErQywuJ0q83uVnQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732145593; c=relaxed/simple; bh=A/BG30k4dPSYeGyQ9WWlDhGgvjjruOFJ2EsBN5g0230=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ClwketkB/jP0xAWNGJF7GpmH6NdXbqEsbZdqSNu8B3b9HkDbs4tTZPK4fAyXjzO0DvZGwuKQnlmoR9hxeElAIPpIqt///P48POV6M9zFNg5Dhc+TfFijvkdr+oOao4c144cejx7i+f9yhXQ7ylQYD8ig3YczjspfFWyndtfolXI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=X/LfyBht; arc=none smtp.client-ip=209.85.218.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="X/LfyBht" Received: by mail-ej1-f50.google.com with SMTP id a640c23a62f3a-a9eb3794a04so31272266b.3 for ; Wed, 20 Nov 2024 15:33:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732145590; x=1732750390; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5cQnp+lrUnHITWCsKfUZMU+VFdBNN5rUz4oDhu1P1PU=; b=X/LfyBhtExzLE4fS1ZP1Y+l0mvEbIcTXl0NK+FtPGBFpIaleNwLxJw4qV0/ie3BXst kHAPlW/fCt0LHz8ACRjrHsOzakH4w9BHakFbWuTbV82Z8gEA1RaUbXyBWIDNSXHIKer0 Vlt59v5RxGBNx1O5HnJpb5VceNjEpxapcAMjQRDzH34Lf++xoSn4uiWlFUev4bXNZLWr Th+IK15HCCcq0ptBGKWXpzMQIqJgyO0PuhR5VPejo7LDjATE7xb4Afggo2kXQSAWjIa9 bU9jp/0YFSl/dyqTWO8Z5y0aL0e9Qq7kLUMgzMDOYl/aFRj109uF5TKiB3sFNpdc3bsA yf7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732145590; x=1732750390; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5cQnp+lrUnHITWCsKfUZMU+VFdBNN5rUz4oDhu1P1PU=; b=pdkFTYhNvz8v9xAtyiosl+/zJpkjm9CIYu04YQmONOw04wpiedb2A3wWyHJa6wP7N/ Km/fv7Cf1Lo78bJplI8PoDgnZBVhpG81kgN2TuIeMsUM8OVusJolQFsp8NnqhRp6uJ2F Ws7QFq3hxtmx8jfGdol8nyLkFScILKd24f94v81Nkr61Dc1VQMfdt7u3t3EXrr7cqh0e 22sfhPl2Iqtty0q4pgEC7SZ1Wnw57w2z/sm9qxRNhwSZOwOzKap6ZlpuFy2D+tYJaIwy xaA5VHqgsrMs+JfqN+YkxutjXWn2wxL/LHZeTl73INHrliLI8hsHXWXMKtRNC/TYha8v D1Dw== X-Gm-Message-State: AOJu0YzfvnJ8IVsaqfcnwnx+UmfchOUrzyohkHZcC0fit1/a3o+6rwzu Q54MRHB083/mX3KtnC51L8w7f/A+ueIXaenNMIXGo3NFe0CNxiZZ2z6jTw== X-Google-Smtp-Source: AGHT+IHkPqIIRFu+8WyF3Vi5l/fSSbPN75nthpEilp1VBi0CJiLpfWRWgdYbYNxS1cgcLhWjfNJ4rg== X-Received: by 2002:a17:907:9493:b0:a99:5234:c56c with SMTP id a640c23a62f3a-aa4dd59f50bmr463269566b.33.1732145589768; Wed, 20 Nov 2024 15:33:09 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.141.165]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-aa4f415372bsm12166066b.13.2024.11.20.15.33.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Nov 2024 15:33:09 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH 05/11] io_uring/memmap: account memory before pinning Date: Wed, 20 Nov 2024 23:33:28 +0000 Message-ID: X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Move memory accounting before page pinning. It shouldn't even try to pin pages if it's not allowed, and accounting is also relatively inexpensive. It also give a better code structure as we do generic accounting and then can branch for different mapping types. Signed-off-by: Pavel Begunkov --- io_uring/memmap.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/io_uring/memmap.c b/io_uring/memmap.c index f76bee5a861a..cc5f6f69ee6c 100644 --- a/io_uring/memmap.c +++ b/io_uring/memmap.c @@ -243,17 +243,21 @@ int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, if (check_add_overflow(reg->user_addr, reg->size, &end)) return -EOVERFLOW; - pages = io_pin_pages(reg->user_addr, reg->size, &nr_pages); - if (IS_ERR(pages)) - return PTR_ERR(pages); - + nr_pages = reg->size >> PAGE_SHIFT; if (ctx->user) { ret = __io_account_mem(ctx->user, nr_pages); if (ret) - goto out_free; + return ret; pages_accounted = nr_pages; } + pages = io_pin_pages(reg->user_addr, reg->size, &nr_pages); + if (IS_ERR(pages)) { + ret = PTR_ERR(pages); + pages = NULL; + goto out_free; + } + vptr = vmap(pages, nr_pages, VM_MAP, PAGE_KERNEL); if (!vptr) { ret = -ENOMEM; @@ -268,7 +272,8 @@ int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, out_free: if (pages_accounted) __io_unaccount_mem(ctx->user, pages_accounted); - io_pages_free(&pages, nr_pages); + if (pages) + io_pages_free(&pages, nr_pages); return ret; } From patchwork Wed Nov 20 23:33:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13881499 Received: from mail-lf1-f46.google.com (mail-lf1-f46.google.com [209.85.167.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F39B51C8317 for ; Wed, 20 Nov 2024 23:33:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732145594; cv=none; b=f+MTige8hpQG6Llw1MSgA4wyYB8DQzO9OdiNWeO3oUcEY3GG5huBQeT6H5yx+ip236/ZafjTwaiCSAuzhq5VNHGZSOPVoLqSgdFe3HS9F3csq2HWXdpCcpnURoRgNjnJA57omhfNt9NRPC8WBTnlfwBnRvuqzNOfvfeoLFBxcYQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732145594; c=relaxed/simple; bh=dujS0aFvz1kY8S8u3h9P118Wqvzr24FydnRRbGhqzO0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=bpdRR+N6GBxyALf/5NRmjUyVJJ/ViV8baABVqiOYEcqLDsH68RdcFmWFqGY1RqfP8WUYXpOWZQ7dqw4iSn57iYwR7QkfJw/XmsaL53d3z9MJHPjISvVhzbDDwXcFv13fd5vMsOJzUkZIHQNc/3U60AoXIG7yoYdtCAAq1AHhrJY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=FpRQYszL; arc=none smtp.client-ip=209.85.167.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="FpRQYszL" Received: by mail-lf1-f46.google.com with SMTP id 2adb3069b0e04-53b13ea6b78so421983e87.2 for ; Wed, 20 Nov 2024 15:33:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732145591; x=1732750391; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sPuIWteBXaKyE6IW5ybcaZa+VOFWSvl/ffKsXytsJvI=; b=FpRQYszLLTq1CYSDLrHKIhA1lqQesgGs0Fn1rd/PqZAFv+UhieLXGl9Ac0wRqEqqoK fBop1xpCQb5scnPxdUxpOkr3mYQmKHbJeSN3SYNTfOLKCmdkoHeVNdOm1c5m92Wr0Hnu BilwTYWhh6mkiAszJX7OPAbKLMOGmJvmwSqISogLfdAdf1kc/8MAud46AknvfGoh45C/ EGtzFhuydwNNo2NS53JxXhxdUKEf09PfRfS2HkJW4kBvpZwfDepNfOsFl5b5+4wz0aYO YJP+MY6MDxHVCDsuQ3UmvYQgcRX74RVvKEeubayDNMpAu+MT2fP36oh+yNTMJdCqX9TW 5gAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732145591; x=1732750391; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sPuIWteBXaKyE6IW5ybcaZa+VOFWSvl/ffKsXytsJvI=; b=LynOsmyfIRNswODdLlBX8F/7Pkr3xePV2l9egs0Qo4ro1tlN59l5cTCVhjBxmQtFVZ 7rm5iRi7oWlDffd3E+3h878Psb5wTMnxz1rrsM/QjnRNXwWVwGnSrDsfqL4p750fztqZ rTdJD3A8orMmybw6/9MX/cK7OK4r3E0XJxgsuli9XwZUF+lUFNH9OT2gm89g8aZJ/m2L t43b1F64ONjuZXenBuFeSEvy5QPnqO0dSnPwIdFhqtfY89YuoOzReXPxmWDJuWSYcRDZ RMu0kmDeHZawn7OD8ctSB4Rp9ggCewIBIAF29KL4FHrbzsahwMTASqidWORAYNd5TAsf uHIA== X-Gm-Message-State: AOJu0YwQGYgqcQhCyk27zOG97ZlWJAVsTiO2ghkUj6LF5m1Hk5Tq16SX lzxt6BlcoICxxyipCKL2vV8cE8FIEiniJTLR5g79d/bdL9fsW0CcrCl8ug== X-Google-Smtp-Source: AGHT+IGmEUH2UNo3QdR9ZYnpzXS+DgKxoCUTT5UrXeJpfsQYbYO7ZNEpJ+aWSz2Gors6s/8AnfI8Ig== X-Received: by 2002:a19:6b19:0:b0:53d:c3aa:e4ef with SMTP id 2adb3069b0e04-53dc3aae78emr2639633e87.31.1732145590645; Wed, 20 Nov 2024 15:33:10 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.141.165]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-aa4f415372bsm12166066b.13.2024.11.20.15.33.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Nov 2024 15:33:10 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH 06/11] io_uring/memmap: reuse io_free_region for failure path Date: Wed, 20 Nov 2024 23:33:29 +0000 Message-ID: <626c9711e8facbac2d80f1852763e61d9668de1b.1732144783.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Regions are going to become more complex with allocation options and optimisations, I want to split initialisation into steps and for that it needs a sane fail path. Reuse io_free_region(), it's smart enough to undo only what's needed and leaves the structure in a consistent state. Signed-off-by: Pavel Begunkov --- io_uring/memmap.c | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/io_uring/memmap.c b/io_uring/memmap.c index cc5f6f69ee6c..2b3cb3fd3fdf 100644 --- a/io_uring/memmap.c +++ b/io_uring/memmap.c @@ -220,7 +220,6 @@ void io_free_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr) int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, struct io_uring_region_desc *reg) { - int pages_accounted = 0; struct page **pages; int nr_pages, ret; void *vptr; @@ -248,32 +247,27 @@ int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, ret = __io_account_mem(ctx->user, nr_pages); if (ret) return ret; - pages_accounted = nr_pages; } + mr->nr_pages = nr_pages; pages = io_pin_pages(reg->user_addr, reg->size, &nr_pages); if (IS_ERR(pages)) { ret = PTR_ERR(pages); - pages = NULL; goto out_free; } + mr->pages = pages; + mr->flags |= IO_REGION_F_USER_PINNED; vptr = vmap(pages, nr_pages, VM_MAP, PAGE_KERNEL); if (!vptr) { ret = -ENOMEM; goto out_free; } - - mr->pages = pages; mr->ptr = vptr; - mr->nr_pages = nr_pages; - mr->flags |= IO_REGION_F_VMAP | IO_REGION_F_USER_PINNED; + mr->flags |= IO_REGION_F_VMAP; return 0; out_free: - if (pages_accounted) - __io_unaccount_mem(ctx->user, pages_accounted); - if (pages) - io_pages_free(&pages, nr_pages); + io_free_region(ctx, mr); return ret; } From patchwork Wed Nov 20 23:33:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13881500 Received: from mail-ej1-f53.google.com (mail-ej1-f53.google.com [209.85.218.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3D9041BBBE8 for ; Wed, 20 Nov 2024 23:33:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732145595; cv=none; b=Ms49Jq6PB0of3dTAknyorWRn1RrGP9hAA9Ls+hf4/wzkLMxqk/P44cL9DUU3meiVukuV8cD6qwt+8iD8jSK4C/n4M4PCSdULaqdh32gd+3Va9QorRAzngllbUQa3SFq94iXHdsjByVDLCDETZARQxmWW5YdjEijwwxqbNhbuxzo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732145595; c=relaxed/simple; bh=55xt3/pYTVPzdu81FVZJ0x8k5BIhUfm79mNIEsz93uw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cCELjQmhp4o7ct83i+stlXJl72s/hcLLKnrRihR5SvR7iNcifD8XvDV3sPM4mWBtqfXg2DK++B1xwHrWhxYN6ZTufXQlg2+I6NZgGRBjkErvl5sOVOqWJsOHukuSxwQTEthFU2/YxBJ1ga+vFNW7Hnn6ZSeOa0bmbTUPSYP1woQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=eENBWZGP; arc=none smtp.client-ip=209.85.218.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="eENBWZGP" Received: by mail-ej1-f53.google.com with SMTP id a640c23a62f3a-a9a0ef5179dso53483966b.1 for ; Wed, 20 Nov 2024 15:33:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732145592; x=1732750392; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=K+skNG/NMNLar+6yG7jRI9A/owfpoygJmWoUnCIcEc0=; b=eENBWZGPFUt2JHickDcMVmE8SoUcmXlRat0ZQFdwT37+NMvOJWmW8p61JvrE1d9a7E 8zySLQzHsEDD9rKPxohLLsWV4PbWFznfM+C+klIF2N9nLVqIlN5BeZnBuTUww+7KIw1N 3Yfdd4nlfflIDQkXXkybO2c3SzrGN/Vmvvbx4bzMg+Cvp0G5tBkSKv9s2LWtZUwU4Gv1 H0xB1oTm34f00ErY+/+f0chH/VfPIvhNHutU+qLBzrs7PX8/WWKT9VGIhhv+z8R1f+MX qeZDDQ2eD5dE0ln0QlM5Q7NbQrRxwJ4VKR/JlhVYq2Srz4JsiwifmG2nXPyHjlmkB4Ng StgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732145592; x=1732750392; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=K+skNG/NMNLar+6yG7jRI9A/owfpoygJmWoUnCIcEc0=; b=XL2PjZS/BeWL/gHoq95gBeTwYdt4PunlwuNb1vIO17RyQohSCdZhg5x0DaqBEBp5pI VteqRGsXNRN5XpStZsLetMTXu9H4UP45bJK93XaBNUeMmYURr8xQAUHT49BlkD69EOdU DKCSrISuTzNqesjgcUZwfFHXykOkmptKvpwkdlqUQeoWNaXnA6cfl370vHf9DLchkKCO iBbnM/EFnqEeF/gyNndacKg1u0JC3IjLaknVL/J4hEVCC4rrVaMjNJfczPqyCtjY0Ulq E8eHCBXIEiqivcRclLLzF9LZp28PlputsmVTENUFgpxCHFc0c1SaSBzECZnBq7WAMxUS XGcA== X-Gm-Message-State: AOJu0YxfpIH2WGzdgEtNmmt5rdOgCxpgLrxB5b8MM4soVFDRz3DKq9xo WD5hA433RbvwZGJmx+CAQyJQosoFS9fxYsugF9dTHwzoT7jDOsbFrM6Gqg== X-Gm-Gg: ASbGnctuhRW522sLpMsoBwxaxfSCUx+n0E/zgX+xL3TU49ZbGCcejCBE88VV3+6YelR yZUfXBhO7yc65K4V1bng1gO0+nmfrfmYM62iVxkfs2ejMvSnn6xPsmoyOo5wBVUGmzdavJYd+0Y GMSriLk9YNvIw7V5M96ErMWJBEKxMtaPy4mfCKVu4h3V3QvKJdSenkwsymFItrLlNfyuQGDP/Jo vWmJy0P30hHcg5TLdPtK6UmvmyQqjbw/cdtBbFp6mpS5TcEzSozC8SIJCzjE1lo X-Google-Smtp-Source: AGHT+IEVXkw7ltgXRAqLxaJ1TfbHUd2KJqajg7amLeXytBW0fyHEX60Lq94WhzDkobfZp+kUWZz6nw== X-Received: by 2002:a17:907:9485:b0:a9a:9ab:6233 with SMTP id a640c23a62f3a-aa4dd56af99mr425929166b.34.1732145591884; Wed, 20 Nov 2024 15:33:11 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.141.165]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-aa4f415372bsm12166066b.13.2024.11.20.15.33.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Nov 2024 15:33:11 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH 07/11] io_uring/memmap: optimise single folio regions Date: Wed, 20 Nov 2024 23:33:30 +0000 Message-ID: <018f6d706b1d986630debc38e01973fe16f085a3.1732144783.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 We don't need to vmap if memory is already physically contiguous. There are two important cases it covers: PAGE_SIZE regions and huge pages. Use io_check_coalesce_buffer() to get the number of contiguous folios. Signed-off-by: Pavel Begunkov --- io_uring/memmap.c | 29 ++++++++++++++++++++++------- 1 file changed, 22 insertions(+), 7 deletions(-) diff --git a/io_uring/memmap.c b/io_uring/memmap.c index 2b3cb3fd3fdf..32d2a39aff02 100644 --- a/io_uring/memmap.c +++ b/io_uring/memmap.c @@ -217,12 +217,31 @@ void io_free_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr) memset(mr, 0, sizeof(*mr)); } +static int io_region_init_ptr(struct io_mapped_region *mr) +{ + struct io_imu_folio_data ifd; + void *ptr; + + if (io_check_coalesce_buffer(mr->pages, mr->nr_pages, &ifd)) { + if (ifd.nr_folios == 1) { + mr->ptr = page_address(mr->pages[0]); + return 0; + } + } + ptr = vmap(mr->pages, mr->nr_pages, VM_MAP, PAGE_KERNEL); + if (!ptr) + return -ENOMEM; + + mr->ptr = ptr; + mr->flags |= IO_REGION_F_VMAP; + return 0; +} + int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, struct io_uring_region_desc *reg) { struct page **pages; int nr_pages, ret; - void *vptr; u64 end; if (WARN_ON_ONCE(mr->pages || mr->ptr || mr->nr_pages)) @@ -258,13 +277,9 @@ int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, mr->pages = pages; mr->flags |= IO_REGION_F_USER_PINNED; - vptr = vmap(pages, nr_pages, VM_MAP, PAGE_KERNEL); - if (!vptr) { - ret = -ENOMEM; + ret = io_region_init_ptr(mr); + if (ret) goto out_free; - } - mr->ptr = vptr; - mr->flags |= IO_REGION_F_VMAP; return 0; out_free: io_free_region(ctx, mr); From patchwork Wed Nov 20 23:33:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13881501 Received: from mail-ej1-f52.google.com (mail-ej1-f52.google.com [209.85.218.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E16411C1F12 for ; Wed, 20 Nov 2024 23:33:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732145596; cv=none; b=YfBXL2o4RLOEsApt/hgSiDpFH+fgtJU9gD9SDgRahkPZ0JWzYQjAx7nabD2kLdxcg2bcH3Pzd9CyG4+2k3wB3L/hLaWp6oUakHI7cRk1jrukM1X5mfgMP7qS6tnO9pUA6U2bmU1h8g0ctJnhHrNjZ0ZUBTudMWc/SEu+isLUIYk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732145596; c=relaxed/simple; bh=AOfi6x19gzyu/5rrZe/6KOA7e2hddpGJfDp/n7HaVeA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=efkuSCXNJY/JeRhNb0s91dWwJA/KRFqcrQrE+mC9iAq626wWMEC20DUm+FMyxOMosstDuw2f/oDo6RTUzXoECZBCSIQOOPZxBO8MAs8p7tJflfQxtXah4lbZdXPMwq61TlWfdZkStGZ7shDDLZnFfngfzawqokdMkb3UA6v/0e8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=RaU87vA3; arc=none smtp.client-ip=209.85.218.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="RaU87vA3" Received: by mail-ej1-f52.google.com with SMTP id a640c23a62f3a-a9a0ec0a94fso49774266b.1 for ; Wed, 20 Nov 2024 15:33:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732145593; x=1732750393; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cKExDGox31CnOmhkyqcAp7yYwe/A8AEg0kEs/PZerJw=; b=RaU87vA3uBKXknmCyscOy2UQx3t6vTZyJMcxwKuhD2/jC4GGAx4Pn0aAEN1rPm1ZZw CQcp20bxXns3FCoDLyBZMNJ3b3HHcKlP+FkAZZb3ABCK2MT5XaqoEv6zKHimFs2GpfvW NMYhOdlIUY9JJvL3xDk6En/07BAEhvonKJt1QI91fPAk8tni/6//GaLNOXsSPyRIg5Lc L/jntH1w022J0/KfYMrD4gO3e1zi/1kmiuP8mEH61F5TTagW8r29xS+aTeXRLkxZnasl fiEADRhK9LzJmfLT6GbCbR1sDq80Kc88Ch8mm4ZGvF3MtKbqC4aNZVFhNKx0pC4j2z0J 2rCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732145593; x=1732750393; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cKExDGox31CnOmhkyqcAp7yYwe/A8AEg0kEs/PZerJw=; b=vFX2G3NmqXjTpFI//ieX0i9cyflMBOdUhI9dtl3si0kwLDUnz3SguMvPCELEznY/xA 5ef8tvNezrDTFANtOfV3X3t2YzJPv/fzWahg9ZT06WHrrBPr0F+txBRaIfMZAOHOGHL6 tNE1Wk5SAmMn+FmA5lVQvdbhx/JTgSHkxqU2cCiIj34GXXLCF4g0CTREFcE9Gn0md3JJ TLwemwDBoFXo36+o61xfjgSdQiQRY1G4P9a9rKFcsP3We3r7BVtbdLY8lKtBGJLHmVg0 bEdYrSuNO4oCCm1+pJ7CEu1dBJj6GlSaRpKtqgrGPXx4T2vc9yRYqR2kRIlT/+/FqCNg kOQA== X-Gm-Message-State: AOJu0YyAjlZ1h3lkW/IGK+qWffd+BY1YabmvqxWTmyuEHSj7VucLET6g NvEet8ynYMVFlBA+jmJvwyzLgGZ4abO5FVcDvMHNtJoKVL6UrAttM7Xf0Q== X-Gm-Gg: ASbGncvxK5WWZVZs9awXfsgnh05KX/UBID1B6zVn5VQwT+zzyBDTeaTm3ZnbSwCDy6h GQZmmb0L/qoxB8B4sMP703RLC0ZoABPfp/Oprkv6RMJScG9ZxFbvYFeOzz1ThNh5PkvxuXnEZJw ZODtJEJ6HBc7CB0i/UAAwUFpUqlljh0+llQgVEQmSbxWu9q5Qcehr8229AUrTd0bkoXQEW2WB54 wSGAJq24MgN4tqM+zkvNWMLZyzOpRvYjy5o748FjSj/A+ZJeFOaQDjybJG8o2P+ X-Google-Smtp-Source: AGHT+IFMVUw3mUobW2DnWAfq4vB9p+OhvmK4OsYjRpVjZ8yiIIQfTqBz+vCuM17Io5d7BqlXO/66dQ== X-Received: by 2002:a17:907:2d08:b0:a9e:d49e:c475 with SMTP id a640c23a62f3a-aa4dd57baa4mr383783866b.26.1732145593041; Wed, 20 Nov 2024 15:33:13 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.141.165]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-aa4f415372bsm12166066b.13.2024.11.20.15.33.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Nov 2024 15:33:12 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH 08/11] io_uring/memmap: helper for pinning region pages Date: Wed, 20 Nov 2024 23:33:31 +0000 Message-ID: X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 In preparation to adding kernel allocated regions extract a new helper that pins user pages. Signed-off-by: Pavel Begunkov --- io_uring/memmap.c | 29 +++++++++++++++++++++-------- 1 file changed, 21 insertions(+), 8 deletions(-) diff --git a/io_uring/memmap.c b/io_uring/memmap.c index 32d2a39aff02..15fefbed77ec 100644 --- a/io_uring/memmap.c +++ b/io_uring/memmap.c @@ -237,10 +237,28 @@ static int io_region_init_ptr(struct io_mapped_region *mr) return 0; } +static int io_region_pin_pages(struct io_ring_ctx *ctx, + struct io_mapped_region *mr, + struct io_uring_region_desc *reg) +{ + unsigned long size = mr->nr_pages << PAGE_SHIFT; + struct page **pages; + int nr_pages; + + pages = io_pin_pages(reg->user_addr, size, &nr_pages); + if (IS_ERR(pages)) + return PTR_ERR(pages); + if (WARN_ON_ONCE(nr_pages != mr->nr_pages)) + return -EFAULT; + + mr->pages = pages; + mr->flags |= IO_REGION_F_USER_PINNED; + return 0; +} + int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, struct io_uring_region_desc *reg) { - struct page **pages; int nr_pages, ret; u64 end; @@ -269,14 +287,9 @@ int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, } mr->nr_pages = nr_pages; - pages = io_pin_pages(reg->user_addr, reg->size, &nr_pages); - if (IS_ERR(pages)) { - ret = PTR_ERR(pages); + ret = io_region_pin_pages(ctx, mr, reg); + if (ret) goto out_free; - } - mr->pages = pages; - mr->flags |= IO_REGION_F_USER_PINNED; - ret = io_region_init_ptr(mr); if (ret) goto out_free; From patchwork Wed Nov 20 23:33:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13881502 Received: from mail-ej1-f41.google.com (mail-ej1-f41.google.com [209.85.218.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3A5951C8317 for ; Wed, 20 Nov 2024 23:33:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732145597; cv=none; b=r8mSovHn4x1ehWZzbCM4s6BB6cPpg5NnAtwIKhChavD+B0QgbmywxZ+kjiHBjBVmjplmydZ3YhsIBi5vOGyCw9jecoJmTHyMAEv8jzQyU4GueoNAOlxACMgiLVn5w5Nju8c1j53V+/pHy/a6Ye8tYhAOnoN3GASBYJB25FDxn4c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732145597; c=relaxed/simple; bh=tZdEZvN/5hCLNJpMtjgBHfxWGrgUbt1UcPId78L2bOo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ittnnJfgGbs5Zwd7E9zGBiBBJyk2pWdXSIYyNfTulAe7RGInUdmlFmWfL0ItjnfS9VjeLrPUw10P6g1WrllRICmRsDQeu08XiY+b8PcvGfLV02JYi2PF2ouxp1tlBtUSclJYYLBivaxyIgXkkRuz6HbV7vvpxBK67ydEZuyYqTU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=lYxOpm9z; arc=none smtp.client-ip=209.85.218.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="lYxOpm9z" Received: by mail-ej1-f41.google.com with SMTP id a640c23a62f3a-aa1e51ce601so59685966b.3 for ; Wed, 20 Nov 2024 15:33:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732145594; x=1732750394; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TOubji84NJuUbvOMUGscG3na9pQtpTrNDF4xMRHuW3Y=; b=lYxOpm9zvIP2Bg5wu7Cp2EnzHmf9H5AtHZm54v1AgLplCSi7EaEXx0OBig2fuv2MYB 7Iq3y2yOwuBgRyx6G+1gNRGCp1WbHUjwIzQXOOsU1DirsBqrpdYeb4CZhuVizGfqBW0i KUka8Wt3dx2BiEsU9pulPkIUxUNXW0SpmCsCgGoVp8GEoG5zdnLGSWJ20EyH8bvfAZZn 7xbI1cTo4wf3j46HRW/lFJOsVOAnm1+O32waYojgFpFEFZiZ46WwD7QMH9EZuCM3wIh/ V4HLwCb5mLvX8mPRVNE4BX5pUToUQaLrlQK9RWikaZM9Y1FWLFhS+bbnn9V8BL/YqX0g q8LA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732145594; x=1732750394; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TOubji84NJuUbvOMUGscG3na9pQtpTrNDF4xMRHuW3Y=; b=HnM8exXtCLix1IfuYB4pAt3QVQMu9p1dbYJsD8cvk2XPu4MFoEY95ABJn0GGccMlOu TTqw6/uILprvv0PwgG0AsmRV+xSxZsy+SGGxqSnZrPB0n9CKMmz4xYXrOH9OOznuyRDU AyvC7hsQO+k746ypof4IqHQyJxgxeq7/jSOB+7uVw7jfcgDWq9becdmALSKXpvWWvza+ 6AftkfKUYQO3VMw60sBRy2T+Jam5fiU+jbiF+zX6wjCvERG1qyxB47rifGShIlVvpwpZ 7BfE4hh6nDv+pBUek4I/6L0sYBCdLyGTLJYjAt/VCEiPYd1oLWPLG5uo2DVNEXswDs8k Q04A== X-Gm-Message-State: AOJu0YwFtT3YW9rseHyAbcQsPi0Cq52jGLukOLKHnT/uVGTN4uAfM3jx 10YL9uFkpt6Q2dl3JKZIpW0nRx7VvPXKDlwNythHJZJlEj4HyogCJaPjFg== X-Gm-Gg: ASbGnctsxsPEIDZFTtvjmfLDkha5L8lF/kmOUE7ZPRiN1Ubng/xTSe/ISqRVywsMl9P /O9ALvTmtrsep+9/RlP/IRqDC1guaeRN0yQHmIwhYKGwYoM8d96l/QBCOY67F4Hd9brcfD7LB7u s+pb6ho10tqhYBd/BDYLGqCc2XnaIe/zHbmmYK1ISGOG5Rzk8yRUNrNtTJU5Dru8sHXmwSYB92Q eTz83ZyHS1uYwp4G2vOdYs51ebZE3i1BYhu006IBRviEHwizvi+vSVi4BLu/JBE X-Google-Smtp-Source: AGHT+IGuV6comNTVE4IAoxI5lopPVb+lCI1qhsCEj78+PPDhezKWs/lHTeJ7AhWOXi7S1aKLP6jrEg== X-Received: by 2002:a17:906:eec8:b0:a99:77f0:51f7 with SMTP id a640c23a62f3a-aa4dd799d4bmr431768466b.61.1732145594136; Wed, 20 Nov 2024 15:33:14 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.141.165]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-aa4f415372bsm12166066b.13.2024.11.20.15.33.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Nov 2024 15:33:13 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH 09/11] io_uring/memmap: add IO_REGION_F_SINGLE_REF Date: Wed, 20 Nov 2024 23:33:32 +0000 Message-ID: <807340ea7288500c76fb693e8a13a3372aba1709.1732144783.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Kernel allocated compound pages will have just one reference for the entire page array, add a flag telling io_free_region about that. Signed-off-by: Pavel Begunkov --- io_uring/memmap.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/io_uring/memmap.c b/io_uring/memmap.c index 15fefbed77ec..cdd620bdd3ee 100644 --- a/io_uring/memmap.c +++ b/io_uring/memmap.c @@ -198,15 +198,22 @@ void *__io_uaddr_map(struct page ***pages, unsigned short *npages, enum { IO_REGION_F_VMAP = 1, IO_REGION_F_USER_PINNED = 2, + IO_REGION_F_SINGLE_REF = 4, }; void io_free_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr) { if (mr->pages) { + long nr_pages = mr->nr_pages; + + if (mr->flags & IO_REGION_F_SINGLE_REF) + nr_pages = 1; + if (mr->flags & IO_REGION_F_USER_PINNED) - unpin_user_pages(mr->pages, mr->nr_pages); + unpin_user_pages(mr->pages, nr_pages); else - release_pages(mr->pages, mr->nr_pages); + release_pages(mr->pages, nr_pages); + kvfree(mr->pages); } if ((mr->flags & IO_REGION_F_VMAP) && mr->ptr) From patchwork Wed Nov 20 23:33:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13881503 Received: from mail-ed1-f45.google.com (mail-ed1-f45.google.com [209.85.208.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1B6051BDA80 for ; Wed, 20 Nov 2024 23:33:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732145598; cv=none; b=JDFqrOEZhPGt8t3pyiMMNQ0EYZaMM6VNiXDckdKM8qsHOicDwmfxKGgr+xjlMn8+crZwRobcQri1OtYnYOgPxYPIKy65JDApEOV5HiNANXnck2TbiVdRzDXXe6hdJZTi825eo6DXoz6+gqmxa33gyjtOpuzMWTZtUovI25kdIrM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732145598; c=relaxed/simple; bh=Ojbg/pyG/fWFfozOjF7t6/LoeMXwPm8VkT4/RWIuwuo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=UP/9aTQoW3YPTC4gIjTkWkMzEt6S3VCArrU7PVPCgp6Yt1NLVbuVNCRsFaEUEbH5+X9BPDMdONL4Kqed7AQBA8YXmUc+ycXGqOjNzmeplvW/tlMaqacaeQ/hLF/E0Z4gyoAltPoNJJsOqtlhiY5l0G19Iy+0oxghaeXhfDwyAWc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=h9R1VwFQ; arc=none smtp.client-ip=209.85.208.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="h9R1VwFQ" Received: by mail-ed1-f45.google.com with SMTP id 4fb4d7f45d1cf-5cfc19065ffso286261a12.3 for ; Wed, 20 Nov 2024 15:33:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732145595; x=1732750395; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HuAPtcJgL3o5hKcE460Eafo6KlyiNWFiJdD3Q2BBUe8=; b=h9R1VwFQv5wt1LijTlaSJV4whMQjXPrkWV/0GC0RGc7g8s5HWFTF7TLNmc2HBwWFXj tLpDwpRmS1wBCp74B21eIoktsd3jgI6end+FWqpWC9DNwmxlESTYpv94JayMdWg4Ccln Z2d7G3OZjvN9I7Sxt4lpgC88hGTpcw8VqlC9wFdAiI+1ejEmdANIUwGo8c1sj+lrVNg5 qW4AIqDNiK0qseHNFLCgG9tB1z/RmD7mFtnAy9hpDLfhcjdZgq4KtAuEUyi72Co1TQsk ZevwiHCUq+V+xldJ4wA/bhiFNS2flZC9LjHZAcKFEP5EaP/35DsO7D5CSOiO/G8uX6oB 9FVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732145595; x=1732750395; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HuAPtcJgL3o5hKcE460Eafo6KlyiNWFiJdD3Q2BBUe8=; b=PU0sL9ApUady0RGbrPjk75816SZNGCEKuiGFxNi0uP4b48LOzNQh5KFswfXX0qCr8a IB44VCVf0/C7Jwkp99oeHe/fJ5DJyaTkoYmXg2A48cat6mTzmtMlgwOSnt17HaCaR5pX VMViegDYcLSudpF+fRpVTmd9RolGPJ6FlxHmhUpmSJdBtUiNeLLzhPLkiupXr3ab85K4 SZ4xkPGI4VTTR+Kt9k7MXikFGp5X6jmbH5DxNtLN3+/rAcZdfLX2IMZo4rJEUC/22Wup EUYhlKEGOV3C4WUiHw5/NgXgSHx6p9dZx1GPYyDwpE+9re9W58RPCRlpQ3rKuOZvJmRI 53hw== X-Gm-Message-State: AOJu0YzGmwUEvuvI0rDg0K3PVfeE9uywzDmgteAUKe7LHNFq2dGkn9mU IPabwPVjmnuRMP1rmaS4ChncPsqECwQWImH9dchUYIPORRS5CyvrM/z0Rg== X-Gm-Gg: ASbGncvrxiWs3mzPKdoFWJyozR2kexQgLpSN3DmkQqjN9HySQ57PQ2Jcj54bebK0Wvq zyERbq8QFBVhbZmGScONDHNY/8RXxLnxrKFEr7LkcbFprmj2hubD1Hw48IqFDSwbi/+DwBfV68u 6ZjZvq6RyhIreQgd8z4z9oxdvnohQ/apxuQ9oR21l0YvpwqgjjIrueANFTK/7c763bYAb32AaQG OtUvdRlSZRsSdKwUtDM4F+rnXp6LQ78sCG7w47x1l/hCaflPYAhLB2bs4dSYkOD X-Google-Smtp-Source: AGHT+IEKV+EtifM6EBVWu4s4rPv+rgk+a9A4sxkLdHwTEXFfr76ZnOCtbb5hAmcwFDvpgiSw4I6LYA== X-Received: by 2002:a17:907:7242:b0:a99:4aa7:4d6f with SMTP id a640c23a62f3a-aa4dd50d3d7mr397922966b.12.1732145595087; Wed, 20 Nov 2024 15:33:15 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.141.165]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-aa4f415372bsm12166066b.13.2024.11.20.15.33.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Nov 2024 15:33:14 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH 10/11] io_uring/memmap: implement kernel allocated regions Date: Wed, 20 Nov 2024 23:33:33 +0000 Message-ID: X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Allow the kernel to allocate memory for a region. That's the classical way SQ/CQ are allocated. It's not yet useful to user space as there is no way to mmap it, which is why it's explicitly disabled in io_register_mem_region(). Signed-off-by: Pavel Begunkov --- io_uring/memmap.c | 44 +++++++++++++++++++++++++++++++++++++++++--- io_uring/register.c | 2 ++ 2 files changed, 43 insertions(+), 3 deletions(-) diff --git a/io_uring/memmap.c b/io_uring/memmap.c index cdd620bdd3ee..8598770bc385 100644 --- a/io_uring/memmap.c +++ b/io_uring/memmap.c @@ -263,6 +263,39 @@ static int io_region_pin_pages(struct io_ring_ctx *ctx, return 0; } +static int io_region_allocate_pages(struct io_ring_ctx *ctx, + struct io_mapped_region *mr, + struct io_uring_region_desc *reg) +{ + gfp_t gfp = GFP_KERNEL_ACCOUNT | __GFP_ZERO | __GFP_NOWARN; + unsigned long size = mr->nr_pages << PAGE_SHIFT; + unsigned long nr_allocated; + struct page **pages; + void *p; + + pages = kvmalloc_array(mr->nr_pages, sizeof(*pages), gfp); + if (!pages) + return -ENOMEM; + + p = io_mem_alloc_compound(pages, mr->nr_pages, size, gfp); + if (!IS_ERR(p)) { + mr->flags |= IO_REGION_F_SINGLE_REF; + mr->pages = pages; + return 0; + } + + nr_allocated = alloc_pages_bulk_noprof(gfp, numa_node_id(), NULL, + mr->nr_pages, NULL, pages); + if (nr_allocated != mr->nr_pages) { + if (nr_allocated) + release_pages(pages, nr_allocated); + kvfree(pages); + return -ENOMEM; + } + mr->pages = pages; + return 0; +} + int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, struct io_uring_region_desc *reg) { @@ -273,9 +306,10 @@ int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, return -EFAULT; if (memchr_inv(®->__resv, 0, sizeof(reg->__resv))) return -EINVAL; - if (reg->flags != IORING_MEM_REGION_TYPE_USER) + if (reg->flags & ~IORING_MEM_REGION_TYPE_USER) return -EINVAL; - if (!reg->user_addr) + /* user_addr should be set IFF it's a user memory backed region */ + if ((reg->flags & IORING_MEM_REGION_TYPE_USER) != !!reg->user_addr) return -EFAULT; if (!reg->size || reg->mmap_offset || reg->id) return -EINVAL; @@ -294,9 +328,13 @@ int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, } mr->nr_pages = nr_pages; - ret = io_region_pin_pages(ctx, mr, reg); + if (reg->flags & IORING_MEM_REGION_TYPE_USER) + ret = io_region_pin_pages(ctx, mr, reg); + else + ret = io_region_allocate_pages(ctx, mr, reg); if (ret) goto out_free; + ret = io_region_init_ptr(mr); if (ret) goto out_free; diff --git a/io_uring/register.c b/io_uring/register.c index ba61697d7a53..f043d3f6b026 100644 --- a/io_uring/register.c +++ b/io_uring/register.c @@ -586,6 +586,8 @@ static int io_register_mem_region(struct io_ring_ctx *ctx, void __user *uarg) if (copy_from_user(&rd, rd_uptr, sizeof(rd))) return -EFAULT; + if (!(rd.flags & IORING_MEM_REGION_TYPE_USER)) + return -EINVAL; if (memchr_inv(®.__resv, 0, sizeof(reg.__resv))) return -EINVAL; if (reg.flags & ~IORING_MEM_REGION_REG_WAIT_ARG) From patchwork Wed Nov 20 23:33:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13881504 Received: from mail-ej1-f50.google.com (mail-ej1-f50.google.com [209.85.218.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 120D61BDA8A for ; Wed, 20 Nov 2024 23:33:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732145600; cv=none; b=d+YE+mjMZe4kZsxmaWYCn0zDYU4I4llen3f0KfImzofNlGDiNoZcb+EfykehCcHGGaxVT+om4QMm/kE4XhsokDvhb9Beb+WZ8mOn4iaGYsivt5I0vHIvK+Gj9E6MaK1Fj1rRvOHSIAaEOwMNGxy6klasqlqnR4M59McH9TeCEcE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732145600; c=relaxed/simple; bh=n3OStU6FcBGysp3Pci4+yNu5CjV7dxL2Qx1q/VYalHo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PitiwP2XHgBqoijkdx38M+ogMgL8uQc8bZlxYMuroQNlcjebCa+Lmb4xw19ySlw3qgP3DYk0/rJasGlClBpRlV9hi1tmsahT1wSQ1OhoPAcr4QjnIh9tgvx/Aoo8psLzoGftyUnwNUu8NkxaeSAW5QH+jpwPrXQooSjdhnKOAJ8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=bGUy9B2y; arc=none smtp.client-ip=209.85.218.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="bGUy9B2y" Received: by mail-ej1-f50.google.com with SMTP id a640c23a62f3a-a9eb3794a04so31278666b.3 for ; Wed, 20 Nov 2024 15:33:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732145596; x=1732750396; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=304V2YUK9klc+k7M6AeMClL+WGzyroOEloK7wWLTT48=; b=bGUy9B2yLl7C+6Qa+aFb3Q7DREketPhejAAt9i/CF5iYnSLevL0E1WCzCNzdxEs3Ke 8Qt+coojC2IqYtS+t7OdCop/d9KqNm2tF2V0ukJVrVkIxKAV/9RJpI89QaxqLZEoGjZQ ysr8UH2pi+NO/ybK+6gepmu9PMnRJAKUSaMtP601pThxVeGq9dlRqRntWXSS6tcvg0P5 i1Nk1he+b8tv3dxEViBxC4xW2Q0oxAoCQ21ePVaehSRNFWTUGih1QH7Sh4yfvzjx6yWa ctJKKPnGvdZPipYtZrrLB3ElampBqf8jPXz7l3FEBzixpjv+yq+cnrK88vBd/npPtGFM wdPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732145596; x=1732750396; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=304V2YUK9klc+k7M6AeMClL+WGzyroOEloK7wWLTT48=; b=T3dX2U7JULvhufcprw9SUwhoCM6806wvuS55WsEEwsTmjIh9qnJPUnJshp0J1k7jLH XwBV9xM9fhbEfR0TjNZGUwNcX6V0AfTRRzyqmiaYizkF9ZnjvYBd7YETGTT7iHiEa+cH Vx0kr6M44IORIrb9mhZfYM2txtti+Gdvt6gadkUUZibKAHhWiLQyOH2bFzDeLrKsl7BM rUPTYNMD2u7V7qvaN/GkLGqZgR8LIT8ojlS98TVuYdZ0StOVFzyRbwlLb65JqRJs6OCp zxQ5GzEIN6wm1tKdxFKv8BMREGKPUvP5eb6SCNR+UHEEN1FeUOJxc563r1W9UnAIcOI2 MQfQ== X-Gm-Message-State: AOJu0YwGHyUSptVXROXLbHwmx/eAOsD5gCOzYQMIl/i5SSs1BKe8RKHN SO9La+/Mg4ArS4ypbfpcJMESMthgPAUl+Ji+kHWbcLJQsKp8O6np0xndtg== X-Google-Smtp-Source: AGHT+IHjKqUF9hjd1+x0V5abJME8VVwecEDlONvrqbXSk0GA/blExfqBJhN42M4Rc06ATM/ddc7mQQ== X-Received: by 2002:a17:907:3f9d:b0:a99:403e:2578 with SMTP id a640c23a62f3a-aa4dd53e1a0mr454775566b.5.1732145595983; Wed, 20 Nov 2024 15:33:15 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.141.165]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-aa4f415372bsm12166066b.13.2024.11.20.15.33.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Nov 2024 15:33:15 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH 11/11] io_uring/memmap: implement mmap for regions Date: Wed, 20 Nov 2024 23:33:34 +0000 Message-ID: <461a81aac8d96a14c3054585faef7b3bdaa2a759.1732144783.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 The patch implements mmap for the param region and enables the kernel allocation mode. Internally it uses a fixed mmap offset, however the user has to use the offset returned in struct io_uring_region_desc::mmap_offset. Note, mmap doesn't and can't take ->uring_lock and the region / ring lookup is protected by ->mmap_lock, and it's directly peeking at ctx->param_region. We can't protect io_create_region() with the mmap_lock as it'd deadlock, which is why io_create_region_mmap_safe() initialises it for us in a temporary variable and then publishes it with the lock taken. It's intentionally decoupled from main region helpers, and in the future we might want to have a list of active regions, which then could be protected by the ->mmap_lock. Signed-off-by: Pavel Begunkov --- io_uring/memmap.c | 61 +++++++++++++++++++++++++++++++++++++++++---- io_uring/memmap.h | 10 +++++++- io_uring/register.c | 6 ++--- 3 files changed, 67 insertions(+), 10 deletions(-) diff --git a/io_uring/memmap.c b/io_uring/memmap.c index 8598770bc385..5d971ba33d5a 100644 --- a/io_uring/memmap.c +++ b/io_uring/memmap.c @@ -265,7 +265,8 @@ static int io_region_pin_pages(struct io_ring_ctx *ctx, static int io_region_allocate_pages(struct io_ring_ctx *ctx, struct io_mapped_region *mr, - struct io_uring_region_desc *reg) + struct io_uring_region_desc *reg, + unsigned long mmap_offset) { gfp_t gfp = GFP_KERNEL_ACCOUNT | __GFP_ZERO | __GFP_NOWARN; unsigned long size = mr->nr_pages << PAGE_SHIFT; @@ -280,8 +281,7 @@ static int io_region_allocate_pages(struct io_ring_ctx *ctx, p = io_mem_alloc_compound(pages, mr->nr_pages, size, gfp); if (!IS_ERR(p)) { mr->flags |= IO_REGION_F_SINGLE_REF; - mr->pages = pages; - return 0; + goto done; } nr_allocated = alloc_pages_bulk_noprof(gfp, numa_node_id(), NULL, @@ -292,12 +292,15 @@ static int io_region_allocate_pages(struct io_ring_ctx *ctx, kvfree(pages); return -ENOMEM; } +done: + reg->mmap_offset = mmap_offset; mr->pages = pages; return 0; } int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, - struct io_uring_region_desc *reg) + struct io_uring_region_desc *reg, + unsigned long mmap_offset) { int nr_pages, ret; u64 end; @@ -331,7 +334,7 @@ int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, if (reg->flags & IORING_MEM_REGION_TYPE_USER) ret = io_region_pin_pages(ctx, mr, reg); else - ret = io_region_allocate_pages(ctx, mr, reg); + ret = io_region_allocate_pages(ctx, mr, reg, mmap_offset); if (ret) goto out_free; @@ -344,6 +347,50 @@ int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, return ret; } +int io_create_region_mmap_safe(struct io_ring_ctx *ctx, struct io_mapped_region *mr, + struct io_uring_region_desc *reg, + unsigned long mmap_offset) +{ + struct io_mapped_region tmp_mr; + int ret; + + memcpy(&tmp_mr, mr, sizeof(tmp_mr)); + ret = io_create_region(ctx, &tmp_mr, reg, mmap_offset); + if (ret) + return ret; + + /* + * Once published mmap can find it without holding only the ->mmap_lock + * and not ->uring_lock. + */ + guard(mutex)(&ctx->mmap_lock); + memcpy(mr, &tmp_mr, sizeof(tmp_mr)); + return 0; +} + +static void *io_region_validate_mmap(struct io_ring_ctx *ctx, + struct io_mapped_region *mr) +{ + lockdep_assert_held(&ctx->mmap_lock); + + if (!io_region_is_set(mr)) + return ERR_PTR(-EINVAL); + if (mr->flags & IO_REGION_F_USER_PINNED) + return ERR_PTR(-EINVAL); + + return io_region_get_ptr(mr); +} + +static int io_region_mmap(struct io_ring_ctx *ctx, + struct io_mapped_region *mr, + struct vm_area_struct *vma) +{ + unsigned long nr_pages = mr->nr_pages; + + vm_flags_set(vma, VM_DONTEXPAND); + return vm_insert_pages(vma, vma->vm_start, mr->pages, &nr_pages); +} + static void *io_uring_validate_mmap_request(struct file *file, loff_t pgoff, size_t sz) { @@ -379,6 +426,8 @@ static void *io_uring_validate_mmap_request(struct file *file, loff_t pgoff, io_put_bl(ctx, bl); return ptr; } + case IORING_MAP_OFF_PARAM_REGION: + return io_region_validate_mmap(ctx, &ctx->param_region); } return ERR_PTR(-EINVAL); @@ -419,6 +468,8 @@ __cold int io_uring_mmap(struct file *file, struct vm_area_struct *vma) ctx->n_sqe_pages); case IORING_OFF_PBUF_RING: return io_pbuf_mmap(file, vma); + case IORING_MAP_OFF_PARAM_REGION: + return io_region_mmap(ctx, &ctx->param_region, vma); } return -EINVAL; diff --git a/io_uring/memmap.h b/io_uring/memmap.h index 2096a8427277..2402bca3d700 100644 --- a/io_uring/memmap.h +++ b/io_uring/memmap.h @@ -1,6 +1,8 @@ #ifndef IO_URING_MEMMAP_H #define IO_URING_MEMMAP_H +#define IORING_MAP_OFF_PARAM_REGION 0x20000000ULL + struct page **io_pin_pages(unsigned long ubuf, unsigned long len, int *npages); void io_pages_free(struct page ***pages, int npages); int io_uring_mmap_pages(struct io_ring_ctx *ctx, struct vm_area_struct *vma, @@ -24,7 +26,13 @@ int io_uring_mmap(struct file *file, struct vm_area_struct *vma); void io_free_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr); int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, - struct io_uring_region_desc *reg); + struct io_uring_region_desc *reg, + unsigned long mmap_offset); + +int io_create_region_mmap_safe(struct io_ring_ctx *ctx, + struct io_mapped_region *mr, + struct io_uring_region_desc *reg, + unsigned long mmap_offset); static inline void *io_region_get_ptr(struct io_mapped_region *mr) { diff --git a/io_uring/register.c b/io_uring/register.c index f043d3f6b026..5b099ec36d00 100644 --- a/io_uring/register.c +++ b/io_uring/register.c @@ -585,9 +585,6 @@ static int io_register_mem_region(struct io_ring_ctx *ctx, void __user *uarg) rd_uptr = u64_to_user_ptr(reg.region_uptr); if (copy_from_user(&rd, rd_uptr, sizeof(rd))) return -EFAULT; - - if (!(rd.flags & IORING_MEM_REGION_TYPE_USER)) - return -EINVAL; if (memchr_inv(®.__resv, 0, sizeof(reg.__resv))) return -EINVAL; if (reg.flags & ~IORING_MEM_REGION_REG_WAIT_ARG) @@ -602,7 +599,8 @@ static int io_register_mem_region(struct io_ring_ctx *ctx, void __user *uarg) !(ctx->flags & IORING_SETUP_R_DISABLED)) return -EINVAL; - ret = io_create_region(ctx, &ctx->param_region, &rd); + ret = io_create_region_mmap_safe(ctx, &ctx->param_region, &rd, + IORING_MAP_OFF_PARAM_REGION); if (ret) return ret; if (copy_to_user(rd_uptr, &rd, sizeof(rd))) {