From patchwork Wed Oct 23 16:07:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13847403 Received: from mail-io1-f50.google.com (mail-io1-f50.google.com [209.85.166.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CFDF613B792 for ; Wed, 23 Oct 2024 16:15:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729700130; cv=none; b=QQa+KUAaCOKfm0N/5TKbgkiQns4Hgokpi31cujIimqy178yhSIlntSr7oMwS7sp6SrnEO9Wu0MkED4OuckD/Kn3sW8+9ISgM5nTGl1zxp6yv+EAmoFVn6QUY7cbNz0tYzjOVG9Z+MmeYFR0fMRfH4deZZd0Cfk30bHOkIkTf3fk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729700130; c=relaxed/simple; bh=Z0kpiriFPHTK/b9vJVR+VM0EHCAbCDbXpGsQ/2t4pQM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kYJhITrWkIxSluW5pEvayEp6/l7n4jfWS0aXTgC/R4OGJWat1syW285MnZqAQUEMI4w/LR7fYQhbYxiH4yQgMpsLCaKdH5EQNu//YfQy6CtcbzcH72fh3kV37PfV3ZSzNVTiyi0G3+ffW3t2QW9+7ssjFpkuWCJQG3x+UeHthBM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=Dfzecdfv; arc=none smtp.client-ip=209.85.166.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="Dfzecdfv" Received: by mail-io1-f50.google.com with SMTP id ca18e2360f4ac-83abf71f244so192794639f.1 for ; Wed, 23 Oct 2024 09:15:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1729700126; x=1730304926; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=aHfspwQxEyRhCy9SkIRl43Tdri8nV0vWzJCiZfMrWDI=; b=Dfzecdfv/8R+qJrPWDfl4Tln/R0FqNgaVe32WcXbN11wqiyqbeiP0qtUJ2CBL+AB8U WihDjalspPUD+omR6kZsLRZDqJPMaKkytd2GbDbM//kGeKJOo04sivHcjdXUU4Zye+WQ MaVVqNkGPIb1BBVwFIfiA6s5NwJoYD4dATDrIj9heye1IiEfKUioIqqdUPgh5r0bcVSb D9X2LYmvySevY52uC4gYBM54YeUQR0L3a3qbhqcm81Uuk76+Jgu07ApfQtFTadJx61jP Q2D8g31JnCbMA1PIRzJrHzQIXK+vq96y0DfQvkwZ6CrkwKkXboVP+v+LeFpMs0Xm5K3b kwPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729700126; x=1730304926; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=aHfspwQxEyRhCy9SkIRl43Tdri8nV0vWzJCiZfMrWDI=; b=OBNvIzmrW81pkcz8nCZ0wSg6TUZ5PCSMQstEsWkCa9d1ee0/EZCAhC2BHj29CGAFu4 iIWu9HffL0v7szcLw/PHNrgRssiPydoUpDtb7rTgbZkz/YboQ1OM5cHBZNXoPNORrOnw +hv2RdQd8m/WzKItuhPDlqMlIm6oEPJDwRVwu5DBzvfHOZoe8Wt6QGL3PCxgjiqdaViO +Qmu177mkfvgd8QZLuYT9Lc0jYGgx865tgpxKs5U1klZwz6d3A/9VFnvn3/+jxq74SLE HjUDy80yNreAYjmtSHEuzmExxVCgDprvfffAy7rvnFtSPgqQXHl4xZSo9Qq0TKdZnUV3 wyaw== X-Gm-Message-State: AOJu0YzTm6KECMGHe4pAi4byW4qC9rH7TP3G2HGPgTZmFVZiTjMlJE/3 hWM/KUCOOiZzuLe99M12LVxZUReJwO54ZzqXaXGRMZipTYLl97JfJT2RzU4vLR0XLpG8lALelET J X-Google-Smtp-Source: AGHT+IGS9hDCzocVql31HNJObnEAQDK2LRkhVV8Opl77/mTwv3ZfR7kglhc2JAxDfTFkoWuRmOgb4Q== X-Received: by 2002:a05:6602:6d06:b0:83a:b7c8:a3dc with SMTP id ca18e2360f4ac-83af6155cccmr294769339f.1.1729700126240; Wed, 23 Oct 2024 09:15:26 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4dc2a556c29sm2138180173.43.2024.10.23.09.15.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Oct 2024 09:15:25 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 1/7] io_uring/kbuf: mark buf_sel_arg mode as KBUF_MODE_FREE once allocated Date: Wed, 23 Oct 2024 10:07:34 -0600 Message-ID: <20241023161522.1126423-2-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241023161522.1126423-1-axboe@kernel.dk> References: <20241023161522.1126423-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 If kbuf expands the iovec array, then it doesn't matter if the caller didn't set KBUF_MODE_FREE or not, as once allocated it should be marked as such. Ensure that this is the case. Signed-off-by: Jens Axboe --- io_uring/kbuf.c | 1 + 1 file changed, 1 insertion(+) diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index d407576ddfb7..1318b8ee2599 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -245,6 +245,7 @@ static int io_ring_buffers_peek(struct io_kiocb *req, struct buf_sel_arg *arg, kfree(arg->iovs); arg->iovs = iov; nr_iovs = nr_avail; + arg->mode |= KBUF_MODE_FREE; } else if (nr_avail < nr_iovs) { nr_iovs = nr_avail; } From patchwork Wed Oct 23 16:07:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13847406 Received: from mail-io1-f50.google.com (mail-io1-f50.google.com [209.85.166.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F37FD13B792 for ; Wed, 23 Oct 2024 16:15:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729700132; cv=none; b=RM2S+muPft9G3s2NYauU3fPcH1bB6HWtSY6k7u4OshNN0uiKsw2Tv4uuwZtKVv0E1BcTczvm6Nvp8kgoNnU5SpIohv1xt3wSCrfwVV+pkPCtobD6g3W+trIwlP5KSRp2t+z3rbqK8Ynf+AdKwaCrqVoaQEDT/HgtCzs7pCEHT1U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729700132; c=relaxed/simple; bh=4MeTIzSnWmTop4++lpux3qtYRZ5O2Q6nBnaomxslZhA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SPhsKGuo21caGhvajw9JPWiychBfEZ+rmPMbcOUMWqtKgw0CS6aiDEDLKIcdp9oeDGQptibSZwugM8ktM9+FI2kdUcvxIA3/GhXlo1hJjnEnjUFIzLKhWksMwOWjn0sFHaKx+ferAy4z+dyyrceCjWGZ6yOlL4E3VUBY/fREMqk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=Ym7xq0fY; arc=none smtp.client-ip=209.85.166.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="Ym7xq0fY" Received: by mail-io1-f50.google.com with SMTP id ca18e2360f4ac-83abcfb9f37so222406939f.1 for ; Wed, 23 Oct 2024 09:15:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1729700129; x=1730304929; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QWqkfKF2WSauZLqFPB/EzMlYFgRDbJMCOGF8XP6rgso=; b=Ym7xq0fYaJMhgTFesa2XCJcUzDsVpTkUSK5hLKctSc7t7i6+6tA8qBu0xdzE4vXmU8 8xxFvzmXzoRnLV7RABCS5pUMVzpBm/F3T4zwFO4az7TCg5QKXEuZux/bj/kDU2li759K T6N0qvjQGf0UB+NRRY5Xhw1vYptvitIfbYDUsIe7D4foAWxrzLX0ec6QS4HpjSBiEbOr FeOXZv5W3PxVqdAHlgo+Gpi4ILKfmjXSkcAw/9DCAwWDSrIE8yiulhB9890/Nz+NwdFU vlxDHR25jwdQe8uWVx40m+UkU/c1uQ0uGXtvM7HDR+G6D3jGfnuiDC+2GV++WGhQesRL exwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729700129; x=1730304929; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QWqkfKF2WSauZLqFPB/EzMlYFgRDbJMCOGF8XP6rgso=; b=oGm2y2Q9/xepv2SY0eK8VytmzM5IKL+/vdo9FQNVDYh6NVLmwFdGPcGuOAACCJNr6y 8IiYZ7HhmdxT87/4cfT05UPZtR73ZIXMZnp61/ivyv/6CRrDSuHIG62N+aqJKqDnVyhi J4NYs1tEFg12BWSaua4wskq9o2vXOHzcByK1TkxAjn4CDdOKL0t+rep9c0QTMwVXAqHn KISVQhkQZWlOdOsJoenRR/w18mNhRfqVbIHYapg8UW0VQAFr5jMjwC/kh5+dTF+maUjx gcv5KhdNq9X5FoCQqBSrT8BCL2vhP65HXONicRaZ5OdHt65Vb5TeVkUGn+V/dohIIJ8F VE6A== X-Gm-Message-State: AOJu0Yz1x1j8Hv9X2GJ23NhYGbT/z17v9vr8zR9j68EZs7iG3iv04YLo 3e2HWrThdosbm2qsotsPqAhZJU4YTvEcvRMqkJ8MEnokZrm6y3IYOsFZpIHhveBR3P60AP4Zzoa q X-Google-Smtp-Source: AGHT+IHwjlo2gODfeugfhuzPJfhdacnHfQXEOfb9Sg1uZcFOWGCbs6ZStrXBM5DBiYVwsdHmQvopLw== X-Received: by 2002:a05:6602:150d:b0:83a:b149:fcf9 with SMTP id ca18e2360f4ac-83af61f3613mr419658839f.11.1729700127774; Wed, 23 Oct 2024 09:15:27 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4dc2a556c29sm2138180173.43.2024.10.23.09.15.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Oct 2024 09:15:26 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 2/7] io_uring/kbuf: change io_provided_buffers_select() calling convention Date: Wed, 23 Oct 2024 10:07:35 -0600 Message-ID: <20241023161522.1126423-3-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241023161522.1126423-1-axboe@kernel.dk> References: <20241023161522.1126423-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Make it more like the ring provided buffers in what arguments it takes, and use similar ordering as well. This makes the code more consistent Signed-off-by: Jens Axboe --- io_uring/kbuf.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index 1318b8ee2599..42579525c4bd 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -117,10 +117,11 @@ static void __user *io_provided_buffer_select(struct io_kiocb *req, size_t *len, return NULL; } -static int io_provided_buffers_select(struct io_kiocb *req, size_t *len, - struct io_buffer_list *bl, - struct iovec *iov) +static int io_provided_buffers_select(struct io_kiocb *req, + struct buf_sel_arg *arg, + struct io_buffer_list *bl, size_t *len) { + struct iovec *iov = arg->iovs; void __user *buf; buf = io_provided_buffer_select(req, len, bl); @@ -311,7 +312,7 @@ int io_buffers_select(struct io_kiocb *req, struct buf_sel_arg *arg, io_kbuf_commit(req, bl, arg->out_len, ret); } } else { - ret = io_provided_buffers_select(req, &arg->out_len, bl, arg->iovs); + ret = io_provided_buffers_select(req, arg, bl, &arg->out_len); } out_unlock: io_ring_submit_unlock(ctx, issue_flags); @@ -338,7 +339,7 @@ int io_buffers_peek(struct io_kiocb *req, struct buf_sel_arg *arg) } /* don't support multiple buffer selections for legacy */ - return io_provided_buffers_select(req, &arg->max_len, bl, arg->iovs); + return io_provided_buffers_select(req, arg, bl, &arg->max_len); } static int __io_remove_buffers(struct io_ring_ctx *ctx, From patchwork Wed Oct 23 16:07:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13847405 Received: from mail-io1-f42.google.com (mail-io1-f42.google.com [209.85.166.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 66B2D1CACDD for ; Wed, 23 Oct 2024 16:15:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729700132; cv=none; b=D1sBxFWzfvgFX332Mb2l2wpnsPNgK3KcUxx7q7U0w+CZJGAroSBCvtny4yGH+n6OVw3uhDy/9YA1gYZwQXerPfpvUONZU1ESTMcIKJHM/mZdTUGIk60l/hYReeB+MsZNQlF5eT/Bueyti/N5aYo13H6VsxnUhvj0WxE8HyVq0HA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729700132; c=relaxed/simple; bh=dvGsUXnzAHkxsecjiCnLZAVaZ4J8zq/EiYVeNX/IvOo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=u2SAl2uHDQb1JkC5IbwL2dU7iHfZ4z1TVoEti6Y9UUPo8jlhTQW3GXrRdxa67E2OQi4TT3t3Kmwd+aDJ1ogdF/S3WsXQH2TWV0ClFEImlJH6nVP4mINwyi7o7XHvRO4ckJS5MNqJoprsJd1P0yYve4zJpt5u4AHh04K4/zaU11Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=R39TMA99; arc=none smtp.client-ip=209.85.166.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="R39TMA99" Received: by mail-io1-f42.google.com with SMTP id ca18e2360f4ac-83ab3413493so240564139f.2 for ; Wed, 23 Oct 2024 09:15:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1729700129; x=1730304929; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JCP85O6Vn4J5WfFHGwdvkrQGlciOPw4+bilPFxYPNbI=; b=R39TMA99f5dVBMXqgCCk82n+vxwG1ii/GZ8tgLUot58qw1a1bTBgjzb7B/oaEk1I4c I1riQA9uJfvE8WMi1/CsXyYy9lUp2tUjtLbEZilgpP5k+ipbxBeAh4mB9/pqgJanSw5P wnNspOInPE8XUoyD5mL3oLZUhjgAqFbQWo+TjYQ+rS670oD7uh6xX+4p7NJ+Uwe7HPBm 0oxBmcO8zEXarDCqmdJu1ZINiu2SexzoumPVIiWjcLFoL0Dm1QGARUJUni+muvKv/y/8 7KMQEc4f1rJygXPRXQgfQIF6KRqDt/LkEPy7t1gfPMs12lDOeehT4dwywa+/NzKb/UHR ZDJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729700129; x=1730304929; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JCP85O6Vn4J5WfFHGwdvkrQGlciOPw4+bilPFxYPNbI=; b=lwJ4OjiAMt62eUTSMg2D2whbOpVPppWKoNzdu/wg8bsruOnlIp52pue2FPslr10kdE xv++TRjFj1hubo00cvJug/NQW8uWIA3xqAcdJ4VzySGlOghKnqLkU+kUGVbUEmnmHeb0 C45U/TjvHA+JRFr4Yyn+k0SIgkyicGem0rEQ4N8PkUr6rWeWhOHDtCePaeZtT7yYDrUJ DCEx7UDlQ6qC0V7rG4IGXdaBXj8zombN9ITKowXO/2tUK0qYms+6XV6BgvzLm1it9rrA PaMjioOnLRwTb771kRTQUC5rio6T679rJwTyEK1EHb1dpjfzXJQjd+kLkH/6AA7kLs4T 5/7A== X-Gm-Message-State: AOJu0YxSRTZc865koJoExY/9JO3ld/LVfR6E48r2wQvyz1e72HMA5LW2 a9hM9y6Q5Ezmifx/23uBaWLiPZ6IYxVi873o/xMPp6dUQYDbnWOZ6wrXqbRGuESBuvZ+/A7uDma M X-Google-Smtp-Source: AGHT+IGMNQtlv3Cw6heF8vIB2CZGBGko3wUxeMqdvcdolJKbFZj9yhXtwYJEgi8zWsYWTPbZQffvYg== X-Received: by 2002:a05:6602:493:b0:82d:129f:acb6 with SMTP id ca18e2360f4ac-83af63f517fmr434941939f.14.1729700128605; Wed, 23 Oct 2024 09:15:28 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4dc2a556c29sm2138180173.43.2024.10.23.09.15.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Oct 2024 09:15:28 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 3/7] io_uring/net: abstract out io_send_import() helper Date: Wed, 23 Oct 2024 10:07:36 -0600 Message-ID: <20241023161522.1126423-4-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241023161522.1126423-1-axboe@kernel.dk> References: <20241023161522.1126423-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 If provided buffers are used, this helper can be used to import the necessary data from a provided buffer group. Only one user so far, but add it in preparation of adding another one. While doing so, also split the actual import into an iov_iter out into a separate helper. In preparation for needing to know the number of mapped segments, return that instead. It still returns < 0 on error. Signed-off-by: Jens Axboe --- io_uring/net.c | 75 +++++++++++++++++++++++++++++++++----------------- 1 file changed, 50 insertions(+), 25 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 2040195e33ab..13b807c729f9 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -578,28 +578,33 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) return IOU_OK; } -int io_send(struct io_kiocb *req, unsigned int issue_flags) +static int __io_send_import(struct io_kiocb *req, struct buf_sel_arg *arg, + int nsegs, unsigned int issue_flags) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); struct io_async_msghdr *kmsg = req->async_data; - struct socket *sock; - unsigned flags; - int min_ret = 0; - int ret; + int ret = nsegs; - sock = sock_from_file(req->file); - if (unlikely(!sock)) - return -ENOTSOCK; + if (nsegs == 1) { + sr->buf = arg->iovs[0].iov_base; + ret = import_ubuf(ITER_SOURCE, sr->buf, sr->len, + &kmsg->msg.msg_iter); + if (unlikely(ret < 0)) + return ret; + } else { + iov_iter_init(&kmsg->msg.msg_iter, ITER_SOURCE, arg->iovs, + nsegs, arg->out_len); + } - if (!(req->flags & REQ_F_POLLED) && - (sr->flags & IORING_RECVSEND_POLL_FIRST)) - return -EAGAIN; + return nsegs; +} - flags = sr->msg_flags; - if (issue_flags & IO_URING_F_NONBLOCK) - flags |= MSG_DONTWAIT; +static int io_send_import(struct io_kiocb *req, unsigned int issue_flags) +{ + struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); + struct io_async_msghdr *kmsg = req->async_data; + int ret = 1; -retry_bundle: if (io_do_buffer_select(req)) { struct buf_sel_arg arg = { .iovs = &kmsg->fast_iov, @@ -629,18 +634,38 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) } sr->len = arg.out_len; - if (ret == 1) { - sr->buf = arg.iovs[0].iov_base; - ret = import_ubuf(ITER_SOURCE, sr->buf, sr->len, - &kmsg->msg.msg_iter); - if (unlikely(ret)) - return ret; - } else { - iov_iter_init(&kmsg->msg.msg_iter, ITER_SOURCE, - arg.iovs, ret, arg.out_len); - } + return __io_send_import(req, &arg, ret, issue_flags); } + return ret; +} + +int io_send(struct io_kiocb *req, unsigned int issue_flags) +{ + struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); + struct io_async_msghdr *kmsg = req->async_data; + struct socket *sock; + unsigned flags; + int min_ret = 0; + int ret; + + sock = sock_from_file(req->file); + if (unlikely(!sock)) + return -ENOTSOCK; + + if (!(req->flags & REQ_F_POLLED) && + (sr->flags & IORING_RECVSEND_POLL_FIRST)) + return -EAGAIN; + + flags = sr->msg_flags; + if (issue_flags & IO_URING_F_NONBLOCK) + flags |= MSG_DONTWAIT; + +retry_bundle: + ret = io_send_import(req, issue_flags); + if (unlikely(ret < 0)) + return ret; + /* * If MSG_WAITALL is set, or this is a bundle send, then we need * the full amount. If just bundle is set, if we do a short send From patchwork Wed Oct 23 16:07:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13847407 Received: from mail-io1-f53.google.com (mail-io1-f53.google.com [209.85.166.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 45F901CACF2 for ; Wed, 23 Oct 2024 16:15:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729700134; cv=none; b=Wbgv5JRtr3v0DqnW7kZ3chbsmSHsqe3RHdNZW7KVtSlAEyUtjh0nxYFNhSxifq2LKzPOKN0e3XVpg6nAdnrvXYIzXLujLcGIXF7w+cO5o7oWS9gsnkjLXVKL5RmU4BA8RWVZiCj+0qllp9G01JRFdhDii2VXR94OkN6q4h+sVBw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729700134; c=relaxed/simple; bh=YuhWb2M5dzo6ojcH45qDoA0CSpsw64SGLAi2xdBIlAY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=daynYhIjjl2+QEhN9PInNzO6y/x4txfMU0Lb5YNMufmuMpEKWc0n3HPXQi2mkSa+s/xbgaMSfPAvQvASNybcAG0Pw2asYrK+v6XQO2i3faO/OXf8mgZuL2uqMea2cVjbxiNC5uZAVXiYF9HEwc//c5hGmOU+L/2hjbQGI4CZsFw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=boF2UW0H; arc=none smtp.client-ip=209.85.166.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="boF2UW0H" Received: by mail-io1-f53.google.com with SMTP id ca18e2360f4ac-83ac4dacaf9so167547739f.2 for ; Wed, 23 Oct 2024 09:15:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1729700131; x=1730304931; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wJyur77YgFvOdwKbBr8lEG6FQOpaV6oMhyKIQWH8OSY=; b=boF2UW0Hz4CeaoHxqJm8OMAA9hdtlwp9FbjAPvURYqXUFcUe6i7y8Fr1qgATCNrFW1 dUwGiwqk2xXQGAajE2fNG7X9D6Eus0GocGYfg3I6Jt37BlfBrBRb33QARXlcvIK4DcML 1eRwtkyqoTXypM7PaqAXp6BuqkZxD8mCpM8YjB8rdagkhaYjBYlTh3DOsiT4PGqtL3K2 Nlnbaczt82SXbdDDQ9u/IRgkToQ8thLqsxK71DSLwQgm/Zqj2iPVb5G/yFhF8LLzfxw2 vEbS/7HvsW6daX+5WoWE4rThE1ImeIc71N9h2fWLAlVuYNjLcs2ISw4CSTunUoe1uVVk X7VA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729700131; x=1730304931; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wJyur77YgFvOdwKbBr8lEG6FQOpaV6oMhyKIQWH8OSY=; b=jD2cchHuGTWasgHN+ipXfdYwcWyHhFSF0YVTV2f73nuJbQ+Lf6WtX8xa7/Ecy6zCYk Aktrt7FBlNDOwHbOLOcQa/NE+xyF1LojRlx/zf631jXg3fVvk0+GiO52krB2YNlNVgsw EtWwKv9hVqaez/V/UjaNJfJBLnnmip7mVpfHar033cD0hb66MpO1QvitEYhBD9vaJKZk 3kMOBUk1GflnOFpjU1EqgtHpkAzFKK0yy+QVgeLpI2sTybwcv/NcSTuR76oPGs8xPiXC mOQZrWGeTpKZ+WkaUXx8DWDlEkKfjilIpw3telFTqG1um0stUj22axrtf+Fru+EtnOvt wMkQ== X-Gm-Message-State: AOJu0YyU/mbPmxzeUhEwGowEsI4dD5gYvzvFVoxim+zsqhsFBdzSED8e eT1d2Zf69PRUjf+QPdaST5iSS2cwxWOB/kZKcIL7iau2rh2kyo04xq0H0jTTEWngTGceUDWQVyE p X-Google-Smtp-Source: AGHT+IHGdfQAIlpQAR42Ix7MxvZ9+JMaMIhDkpSCtqwmWZCH7Mija65LKy7SStmsTNerUaZJUjajnA== X-Received: by 2002:a05:6602:6b0f:b0:82a:4480:badc with SMTP id ca18e2360f4ac-83af63fea2fmr346953339f.10.1729700129456; Wed, 23 Oct 2024 09:15:29 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4dc2a556c29sm2138180173.43.2024.10.23.09.15.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Oct 2024 09:15:29 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 4/7] io_uring/net: move send zc fixed buffer import into helper Date: Wed, 23 Oct 2024 10:07:37 -0600 Message-ID: <20241023161522.1126423-5-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241023161522.1126423-1-axboe@kernel.dk> References: <20241023161522.1126423-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 In preparation to making the fixed buffer importing a bit more elaborate in terms of what it supports. Signed-off-by: Jens Axboe --- io_uring/net.c | 77 ++++++++++++++++++++++++++++---------------------- 1 file changed, 44 insertions(+), 33 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 13b807c729f9..dbef14aa50f9 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -81,6 +81,9 @@ struct io_sr_msg { struct io_kiocb *notif; }; +static int io_sg_from_iter(struct sk_buff *skb, struct iov_iter *from, + size_t length); + /* * Number of times we'll try and do receives if there's more data. If we * exceed this limit, then add us to the back of the queue and retry from @@ -578,6 +581,37 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) return IOU_OK; } +static int io_send_zc_import_single(struct io_kiocb *req, + unsigned int issue_flags) +{ + struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); + struct io_async_msghdr *kmsg = req->async_data; + struct io_ring_ctx *ctx = req->ctx; + struct io_mapped_ubuf *imu; + int ret; + u16 idx; + + ret = -EFAULT; + io_ring_submit_lock(ctx, issue_flags); + if (sr->buf_index < ctx->nr_user_bufs) { + idx = array_index_nospec(sr->buf_index, ctx->nr_user_bufs); + imu = READ_ONCE(ctx->user_bufs[idx]); + io_req_set_rsrc_node(sr->notif, ctx); + ret = 0; + } + io_ring_submit_unlock(ctx, issue_flags); + + if (unlikely(ret)) + return ret; + + ret = io_import_fixed(ITER_SOURCE, &kmsg->msg.msg_iter, imu, + (u64)(uintptr_t)sr->buf, sr->len); + if (unlikely(ret)) + return ret; + kmsg->msg.sg_from_iter = io_sg_from_iter; + return 0; +} + static int __io_send_import(struct io_kiocb *req, struct buf_sel_arg *arg, int nsegs, unsigned int issue_flags) { @@ -1365,40 +1399,17 @@ static int io_send_zc_import(struct io_kiocb *req, unsigned int issue_flags) struct io_async_msghdr *kmsg = req->async_data; int ret; - if (sr->flags & IORING_RECVSEND_FIXED_BUF) { - struct io_ring_ctx *ctx = req->ctx; - struct io_mapped_ubuf *imu; - int idx; - - ret = -EFAULT; - io_ring_submit_lock(ctx, issue_flags); - if (sr->buf_index < ctx->nr_user_bufs) { - idx = array_index_nospec(sr->buf_index, ctx->nr_user_bufs); - imu = READ_ONCE(ctx->user_bufs[idx]); - io_req_set_rsrc_node(sr->notif, ctx); - ret = 0; - } - io_ring_submit_unlock(ctx, issue_flags); + if (sr->flags & IORING_RECVSEND_FIXED_BUF) + return io_send_zc_import_single(req, issue_flags); - if (unlikely(ret)) - return ret; - - ret = io_import_fixed(ITER_SOURCE, &kmsg->msg.msg_iter, imu, - (u64)(uintptr_t)sr->buf, sr->len); - if (unlikely(ret)) - return ret; - kmsg->msg.sg_from_iter = io_sg_from_iter; - } else { - ret = import_ubuf(ITER_SOURCE, sr->buf, sr->len, &kmsg->msg.msg_iter); - if (unlikely(ret)) - return ret; - ret = io_notif_account_mem(sr->notif, sr->len); - if (unlikely(ret)) - return ret; - kmsg->msg.sg_from_iter = io_sg_from_iter_iovec; - } - - return ret; + ret = import_ubuf(ITER_SOURCE, sr->buf, sr->len, &kmsg->msg.msg_iter); + if (unlikely(ret)) + return ret; + ret = io_notif_account_mem(sr->notif, sr->len); + if (unlikely(ret)) + return ret; + kmsg->msg.sg_from_iter = io_sg_from_iter_iovec; + return 0; } int io_send_zc(struct io_kiocb *req, unsigned int issue_flags) From patchwork Wed Oct 23 16:07:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13847408 Received: from mail-io1-f42.google.com (mail-io1-f42.google.com [209.85.166.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C685B1CB31C for ; Wed, 23 Oct 2024 16:15:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729700134; cv=none; b=aIIPac4LjfrKVRUQ/y3YneRjprh5YwxE4ZpQSZTKIKrgNBPP0j6iMfaxanGo0p97AvLjAl09RKnbjd0SCCy/90k8sBAPNBOBl2WYMShliygc8frRZzh+dcCYU9IC3W/1NkphtZYj9k2Dt522djcriz3h8yBULb9dvp+DUAmCZDg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729700134; c=relaxed/simple; bh=O7ILj4Waqkz06uXTxRShQ3wnBOigvCdfeeJ0N9Bhr78=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=lQb/oFwvV4Ce3cTYKj5u97Vhr9JA4YHK/Tbt075tCvm3L6LZiMStrigdb7e54J9TNh7C54V8sWlJDWYe2vWOGdu7EfGaYFgNkKSJVcATKsmjCBMD6qS3oTgEnpPPJ5J3caLjnkFWXwLEPIGXdAi1yTv02iVmlTPO4f/wpmY8xeI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=jbyhWYkK; arc=none smtp.client-ip=209.85.166.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="jbyhWYkK" Received: by mail-io1-f42.google.com with SMTP id ca18e2360f4ac-8377fd760b0so278304039f.2 for ; Wed, 23 Oct 2024 09:15:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1729700131; x=1730304931; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JAVLaZMsYlaRm01XXXR+pTcArBexY2/Nbj1JvQNG4R4=; b=jbyhWYkKqHRCgu4RMwSVJPaTOkVxBoEo5w2exaTb9ivqnh8YUweJz7gG/ZikXr60O3 QfsY+cN0GIZJHyHdiwxvnUNGVyHclBcwkSo4qqZhoQc8VT79mAALCpsHNM49MfiyVplV ZXCs6+B6fF5uBj3npxomF3j/7o3eCICsHJl12IHqLYCqp8ND+gpwb/PzEThjBYHoY0Jp JYkgoICX7E7t7W9QK/kUYFeeKXGJKWeQCXKVgYngLE/kWHN+dzSwlOoGvZxijmD5aIuX Cq5ZxM+aEGgllfO/vDbQfFOY6WwbdoLd9wy5SgQc7vct+DeQpRQYPmCccge/oBPvS6wy ANYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729700131; x=1730304931; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JAVLaZMsYlaRm01XXXR+pTcArBexY2/Nbj1JvQNG4R4=; b=r0KctkIqNbBhtWlJb+uQJa//v6irbBJ2BTBRYyOEVYxCutU3atWINvDFcND9qqwxmm gKmB1Jfg+a65ztyl3fcc0TRFXlcsy5997fHz+MyXTTPUK2Dxi47QeqfUxaRsBGJp6xER EX9yATWRZYFtHFdp3E9Cpkv8jL1Wb0n36pL3f/dmSP381RUcntnXz1Q4DIzfGxu0t1cr DJ2apXZBqNirjq+Bvm90CLf6ieh2jMOQkTjOMGcglTDgd4a/9woDj5e1c68eir+mecAL N6aEk4J2m3Vj4uj9/Un6teas5y+/ycje7wteEDZf+y2psR93RFB4r1VUo3ES2PTfBzvy Kuzg== X-Gm-Message-State: AOJu0Yx2WfxOsEdOby6E94rEwqM0626pbsHa4UEV9hT8ZUyysWmhJeDP dZwqDef5+ZXpoyPOUyMPn6xwvmFG3LtVz1hneyBnwI51D52VIQNIpLdBnwuL5tpXwRx+LR/wv0b y X-Google-Smtp-Source: AGHT+IGpcof+4P2x6ot3Owb47UAToc10m33JHusxd38kxYnIrfs/U84HpvJFwXGWMqSc4ZdWiSxyvA== X-Received: by 2002:a05:6602:6b82:b0:835:4d27:edf6 with SMTP id ca18e2360f4ac-83af6192858mr292528839f.7.1729700131270; Wed, 23 Oct 2024 09:15:31 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4dc2a556c29sm2138180173.43.2024.10.23.09.15.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Oct 2024 09:15:29 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 5/7] io_uring: add ability for provided buffer to index registered buffers Date: Wed, 23 Oct 2024 10:07:38 -0600 Message-ID: <20241023161522.1126423-6-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241023161522.1126423-1-axboe@kernel.dk> References: <20241023161522.1126423-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 This just adds the necessary shifts that define what a provided buffer that is merely an index into a registered buffer looks like. A provided buffer looks like the following: struct io_uring_buf { __u64 addr; __u32 len; __u16 bid; __u16 resv; }; where 'addr' holds a userspace address, 'len' is the length of the buffer, and 'bid' is the buffer ID identifying the buffer. This works fine for a virtual address, but it cannot be used efficiently denote a registered buffer. Registered buffers are pre-mapped into the kernel for more efficient IO, avoiding a get_user_pages() and page(s) inc+dec, and are used for things like O_DIRECT on storage and zero copy send. Particularly for the send case, it'd be useful to support a mix of provided and registered buffers. This enables the use of using a provided ring buffer to serialize sends, and also enables the use of send bundles, where a send can pick multiple buffers and send them all at once. If provided buffers are used as an index into registered buffers, the meaning of buf->addr changes. If registered buffer index 'regbuf_index' is desired, with a length of 'len' and the offset 'regbuf_offset' from the start of the buffer, then the application would fill out the entry as follows: buf->addr = ((__u64) regbuf_offset << IOU_BUF_OFFSET_BITS) | regbuf_index; buf->len = len; and otherwise add it to the buffer ring as usual. The kernel will then first pick a buffer from the desired buffer group ID, and then decode which registered buffer to use for the transfer. This provides a way to use both registered and provided buffers at the same time. Signed-off-by: Jens Axboe --- include/uapi/linux/io_uring.h | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 86cb385fe0b5..eef88d570cb4 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -733,6 +733,14 @@ struct io_uring_buf_ring { }; }; +/* + * When provided buffers are used as indices into registered buffers, the + * lower IOU_BUF_REGBUF_BITS indicate the index into the registered buffers, + * and the upper IOU_BUF_OFFSET_BITS indicate the offset into that buffer. + */ +#define IOU_BUF_REGBUF_BITS (32ULL) +#define IOU_BUF_OFFSET_BITS (32ULL) + /* * Flags for IORING_REGISTER_PBUF_RING. * From patchwork Wed Oct 23 16:07:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13847410 Received: from mail-io1-f49.google.com (mail-io1-f49.google.com [209.85.166.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B5C5F1C3045 for ; Wed, 23 Oct 2024 16:15:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729700139; cv=none; b=NKA/Nd/VrEcgO/oIlsnFw6IlwCuag5oXzFbZQhs5Ck63IS0HuOsocZbIR55+jUCgme7vEWb0UE35hqaF69J1BrAvOVD2IzQUIk0gGoBui6UlrMZnpZIuyhcPDkqO4Eq7OmgIx0V4x7RY9ltl3iWr7Tqmi9sZxP8fZzFd12gELk4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729700139; c=relaxed/simple; bh=ECixvDA0CV9DbzgGfS1vLwyWdlt6qzfwBdUA5lTD+0E=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YKcIZcxlvQeg6AsooOerHRODx6OfkQGFaB+/lSv4HUZHRoJqekf2Iya98w1YL5ZWJO7Vg1DtpmpuOnjUTsEflUb4ePBeHMHZ2NokZgYG/lJ24G88GJzsbePDVpo4WUoo6+1cnDv72haq54zg+ZdCRJxLFBlpAPNkXUlaZlmmStc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=tXpO1Azf; arc=none smtp.client-ip=209.85.166.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="tXpO1Azf" Received: by mail-io1-f49.google.com with SMTP id ca18e2360f4ac-83abe7fc77eso205017139f.0 for ; Wed, 23 Oct 2024 09:15:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1729700136; x=1730304936; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4kOPdgM0vwRvx2gbkp5/pjZOR3nL5VqvejKEQrCn8qI=; b=tXpO1AzfDVI+rtMblfcUPNa9PpdzdZAHnDS+32sIyi+9qDwDm1kXRee95u54cDfgkZ VqUYpGUrxNFRRH5DyW5Ir5tsRjZv2EmV13jETgNw0Hwb2w1mHPPzWLRbUaiftL1XNRQH I3Yi7qhc5izC5s9kFQtIx/8vvt+81voWrUmCjB58bTWyMRZQF+3BguSorJ5SLf6Lzsao diGH+Oh7ic5hGgA8ecYp1J5COXpmp0QJRa5fvVfz9oa4hBFCLSQI2bsaLw0TX1db+rxz /f3Hp4UmhAtZ/MvpCsLQbH2dOs/5CoXequsxzKnQKbrJ9Qnw2qcjrPPcAeUFdOZ1e68S V1yg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729700136; x=1730304936; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4kOPdgM0vwRvx2gbkp5/pjZOR3nL5VqvejKEQrCn8qI=; b=unfBI8A8ne9J0zXLHef+lc/kgmA9YKKXlNJ4RhaXs6PIj4sD1qiJqeAo3RpeHEklcS onejvkajjp2Gr5S/FbOrNoUyBojGq6FVh9GuVFv6SYR1veVHNGPF2CQE2LwZNdjRdBMb TXeObL02CCzApZDVl38QRErJNUKbUubRRGuRAWW38woeS5UHIJaVzPCzuMA1SzdiAeQO UeQRiGbOGKpjmcf8erla/6bS+egYyTwSmWxAl+e7uIJOZk5/i+pIkbp1TTHKS6f25wMw MyHjbrEHViF/Gej4HgTZV44VHtYfulQW+208Keoh/AkBlAY1loWgbOuA6d/HG729osNs oKsg== X-Gm-Message-State: AOJu0YyhQlRhe3KfWbhymHTAEwQWzDr953Ri595HVH8a0FQikZ2Y/7XB y3RLEry47yDbiSOr8xdZu2wJcEVtecSmr21gy2UqVlMywCehU3J38EEWQZp30WwrA+uS+8tGYJn V X-Google-Smtp-Source: AGHT+IFPv8t+UOSRBRrYN44onrF8tB2bt6NJr4qWaJqi2pmkQnFwKoFoF+dsRBAYW4pMcI6S92dyCQ== X-Received: by 2002:a05:6602:1483:b0:83a:b79c:66b6 with SMTP id ca18e2360f4ac-83af615132bmr358639039f.2.1729700132749; Wed, 23 Oct 2024 09:15:32 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4dc2a556c29sm2138180173.43.2024.10.23.09.15.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Oct 2024 09:15:31 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 6/7] io_uring/kbuf: add support for mapping type KBUF_MODE_BVEC Date: Wed, 23 Oct 2024 10:07:39 -0600 Message-ID: <20241023161522.1126423-7-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241023161522.1126423-1-axboe@kernel.dk> References: <20241023161522.1126423-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 The provided buffer helpers always map to iovecs. Add a new mode, KBUF_MODE_BVEC, which instead maps it to a bio_vec array instead. For use with zero-copy scenarios, where the caller would want to turn it into a bio_vec anyway, and this avoids first iterating and filling out and iovec array, only for the caller to then iterate it again and turn it into a bio_vec array. Since it's now managing both iovecs and bvecs, change the naming of buf_sel_arg->nr_iovs member to nr_vecs instead. Signed-off-by: Jens Axboe --- io_uring/kbuf.c | 170 +++++++++++++++++++++++++++++++++++++++++++----- io_uring/kbuf.h | 9 ++- io_uring/net.c | 10 +-- 3 files changed, 165 insertions(+), 24 deletions(-) diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index 42579525c4bd..10a3a7a27e9a 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -16,6 +16,7 @@ #include "opdef.h" #include "kbuf.h" #include "memmap.h" +#include "rsrc.h" /* BIDs are addressed by a 16-bit field in a CQE */ #define MAX_BIDS_PER_BGID (1 << 16) @@ -117,20 +118,135 @@ static void __user *io_provided_buffer_select(struct io_kiocb *req, size_t *len, return NULL; } +static struct io_mapped_ubuf *io_ubuf_from_buf(struct io_ring_ctx *ctx, + u64 addr, unsigned int *offset) +{ + struct io_mapped_ubuf *imu; + u16 idx; + + /* + * Get registered buffer index and offset, encoded into the + * addr base value. + */ + idx = addr & ((1ULL << IOU_BUF_REGBUF_BITS) - 1); + addr >>= IOU_BUF_REGBUF_BITS; + *offset = addr & ((1ULL << IOU_BUF_OFFSET_BITS) - 1); + + if (unlikely(idx >= ctx->nr_user_bufs)) + return ERR_PTR(-EFAULT); + + idx = array_index_nospec(idx, ctx->nr_user_bufs); + imu = READ_ONCE(ctx->user_bufs[idx]); + if (unlikely(*offset >= imu->len)) + return ERR_PTR(-EFAULT); + + return imu; +} + +static bool io_expand_bvecs(struct buf_sel_arg *arg) +{ + int nvecs = arg->nr_vecs + 8; + struct bio_vec *bv; + + if (!(arg->mode & KBUF_MODE_EXPAND)) + return false; + + bv = kmalloc_array(nvecs, sizeof(struct bio_vec), GFP_KERNEL); + if (unlikely(!bv)) + return false; + memcpy(bv, arg->bvecs, arg->nr_vecs * sizeof(*bv)); + if (arg->mode & KBUF_MODE_FREE) + kfree(arg->bvecs); + arg->bvecs = bv; + arg->nr_vecs = nvecs; + arg->mode |= KBUF_MODE_FREE; + return true; +} + +static int io_fill_bvecs(struct io_ring_ctx *ctx, u64 addr, + struct buf_sel_arg *arg, unsigned int len, + int *vec_off) +{ + struct bio_vec *src, *src_prv = NULL; + struct io_mapped_ubuf *imu; + unsigned int llen = len; + unsigned int offset; + + imu = io_ubuf_from_buf(ctx, addr, &offset); + if (unlikely(IS_ERR(imu))) + return PTR_ERR(imu); + + if (unlikely(offset >= imu->len || len > imu->len)) + return -EOVERFLOW; + if (unlikely(offset > imu->len - len)) + return -EOVERFLOW; + + src = imu->bvec; + if (offset > src->bv_len) { + unsigned long seg_skip; + + offset -= src->bv_len; + seg_skip = 1 + (offset >> imu->folio_shift); + offset &= ((1UL << imu->folio_shift) - 1); + src += seg_skip; + } + + do { + unsigned int this_len = len; + + if (this_len + offset > src->bv_len) + this_len = src->bv_len - offset; + + /* + * If contig with previous bio_vec, merge it to minimize the + * number of segments needed. If not, then add a new segment, + * expanding the number of available slots, if needed. + */ + if (src_prv && + page_folio(src_prv->bv_page) == page_folio(src->bv_page) && + src_prv->bv_page + 1 == src->bv_page) { + arg->bvecs[*vec_off - 1].bv_len += this_len; + } else { + struct bio_vec *dst; + + if (*vec_off == arg->nr_vecs && !io_expand_bvecs(arg)) + break; + + dst = &arg->bvecs[*vec_off]; + dst->bv_page = src->bv_page; + dst->bv_len = this_len; + dst->bv_offset = offset; + (*vec_off)++; + } + offset = 0; + len -= this_len; + src_prv = src++; + } while (len); + + return llen - len; +} + static int io_provided_buffers_select(struct io_kiocb *req, struct buf_sel_arg *arg, struct io_buffer_list *bl, size_t *len) { - struct iovec *iov = arg->iovs; void __user *buf; + int ret; buf = io_provided_buffer_select(req, len, bl); if (unlikely(!buf)) return -ENOBUFS; - iov[0].iov_base = buf; - iov[0].iov_len = *len; - return 1; + if (arg->mode & KBUF_MODE_BVEC) { + u64 addr = (unsigned long)(uintptr_t) buf; + + *len = io_fill_bvecs(req->ctx, addr, arg, *len, &ret); + } else { + arg->iovs[0].iov_base = buf; + arg->iovs[0].iov_len = *len; + ret = 1; + } + return ret; } static void __user *io_ring_buffer_select(struct io_kiocb *req, size_t *len, @@ -196,13 +312,16 @@ void __user *io_buffer_select(struct io_kiocb *req, size_t *len, #define PEEK_MAX_IMPORT 256 static int io_ring_buffers_peek(struct io_kiocb *req, struct buf_sel_arg *arg, - struct io_buffer_list *bl) + struct io_buffer_list *bl, int *nbufs) { struct io_uring_buf_ring *br = bl->buf_ring; struct iovec *iov = arg->iovs; - int nr_iovs = arg->nr_iovs; + int nr_iovs = arg->nr_vecs; __u16 nr_avail, tail, head; struct io_uring_buf *buf; + int vec_off; + + BUILD_BUG_ON(sizeof(struct iovec) > sizeof(struct bio_vec)); tail = smp_load_acquire(&br->tail); head = bl->head; @@ -236,10 +355,12 @@ static int io_ring_buffers_peek(struct io_kiocb *req, struct buf_sel_arg *arg, /* * only alloc a bigger array if we know we have data to map, eg not - * a speculative peek operation. + * a speculative peek operation. Note that struct bio_vec and + * struct iovec are the same size, so we can use them interchangably + * here as it's just for sizing purposes. */ if (arg->mode & KBUF_MODE_EXPAND && nr_avail > nr_iovs && arg->max_len) { - iov = kmalloc_array(nr_avail, sizeof(struct iovec), GFP_KERNEL); + iov = kmalloc_array(nr_avail, sizeof(struct bio_vec), GFP_KERNEL); if (unlikely(!iov)) return -ENOMEM; if (arg->mode & KBUF_MODE_FREE) @@ -255,6 +376,7 @@ static int io_ring_buffers_peek(struct io_kiocb *req, struct buf_sel_arg *arg, if (!arg->max_len) arg->max_len = INT_MAX; + vec_off = 0; req->buf_index = buf->bid; do { u32 len = buf->len; @@ -266,15 +388,25 @@ static int io_ring_buffers_peek(struct io_kiocb *req, struct buf_sel_arg *arg, buf->len = len; } - iov->iov_base = u64_to_user_ptr(buf->addr); - iov->iov_len = len; - iov++; + if (arg->mode & KBUF_MODE_BVEC) { + int ret; + + ret = io_fill_bvecs(req->ctx, buf->addr, arg, len, &vec_off); + if (unlikely(ret < 0)) + return ret; + len = ret; + } else { + iov->iov_base = u64_to_user_ptr(buf->addr); + iov->iov_len = len; + iov++; + vec_off++; + } arg->out_len += len; arg->max_len -= len; + (*nbufs)++; if (!arg->max_len) break; - buf = io_ring_head_to_buf(br, ++head, bl->mask); } while (--nr_iovs); @@ -283,7 +415,7 @@ static int io_ring_buffers_peek(struct io_kiocb *req, struct buf_sel_arg *arg, req->flags |= REQ_F_BUFFER_RING; req->buf_list = bl; - return iov - arg->iovs; + return vec_off; } int io_buffers_select(struct io_kiocb *req, struct buf_sel_arg *arg, @@ -299,7 +431,9 @@ int io_buffers_select(struct io_kiocb *req, struct buf_sel_arg *arg, goto out_unlock; if (bl->flags & IOBL_BUF_RING) { - ret = io_ring_buffers_peek(req, arg, bl); + int nbufs = 0; + + ret = io_ring_buffers_peek(req, arg, bl, &nbufs); /* * Don't recycle these buffers if we need to go through poll. * Nobody else can use them anyway, and holding on to provided @@ -307,9 +441,9 @@ int io_buffers_select(struct io_kiocb *req, struct buf_sel_arg *arg, * side anyway with normal buffers. Besides, we already * committed them, they cannot be put back in the queue. */ - if (ret > 0) { + if (nbufs) { req->flags |= REQ_F_BUFFERS_COMMIT | REQ_F_BL_NO_RECYCLE; - io_kbuf_commit(req, bl, arg->out_len, ret); + io_kbuf_commit(req, bl, arg->out_len, nbufs); } } else { ret = io_provided_buffers_select(req, arg, bl, &arg->out_len); @@ -332,7 +466,9 @@ int io_buffers_peek(struct io_kiocb *req, struct buf_sel_arg *arg) return -ENOENT; if (bl->flags & IOBL_BUF_RING) { - ret = io_ring_buffers_peek(req, arg, bl); + int nbufs = 0; + + ret = io_ring_buffers_peek(req, arg, bl, &nbufs); if (ret > 0) req->flags |= REQ_F_BUFFERS_COMMIT; return ret; diff --git a/io_uring/kbuf.h b/io_uring/kbuf.h index 36aadfe5ac00..7c56ba994f21 100644 --- a/io_uring/kbuf.h +++ b/io_uring/kbuf.h @@ -53,13 +53,18 @@ enum { KBUF_MODE_EXPAND = 1, /* if bigger vec allocated, free old one */ KBUF_MODE_FREE = 2, + /* turn into bio_vecs, not iovecs */ + KBUF_MODE_BVEC = 4, }; struct buf_sel_arg { - struct iovec *iovs; + union { + struct iovec *iovs; + struct bio_vec *bvecs; + }; size_t out_len; size_t max_len; - unsigned short nr_iovs; + unsigned short nr_vecs; unsigned short mode; }; diff --git a/io_uring/net.c b/io_uring/net.c index dbef14aa50f9..154756762a46 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -643,17 +643,17 @@ static int io_send_import(struct io_kiocb *req, unsigned int issue_flags) struct buf_sel_arg arg = { .iovs = &kmsg->fast_iov, .max_len = min_not_zero(sr->len, INT_MAX), - .nr_iovs = 1, + .nr_vecs = 1, }; if (kmsg->free_iov) { - arg.nr_iovs = kmsg->free_iov_nr; + arg.nr_vecs = kmsg->free_iov_nr; arg.iovs = kmsg->free_iov; arg.mode = KBUF_MODE_FREE; } if (!(sr->flags & IORING_RECVSEND_BUNDLE)) - arg.nr_iovs = 1; + arg.nr_vecs = 1; else arg.mode |= KBUF_MODE_EXPAND; @@ -1140,12 +1140,12 @@ static int io_recv_buf_select(struct io_kiocb *req, struct io_async_msghdr *kmsg sr->flags & IORING_RECVSEND_BUNDLE) { struct buf_sel_arg arg = { .iovs = &kmsg->fast_iov, - .nr_iovs = 1, + .nr_vecs = 1, .mode = KBUF_MODE_EXPAND, }; if (kmsg->free_iov) { - arg.nr_iovs = kmsg->free_iov_nr; + arg.nr_vecs = kmsg->free_iov_nr; arg.iovs = kmsg->free_iov; arg.mode |= KBUF_MODE_FREE; } From patchwork Wed Oct 23 16:07:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13847409 Received: from mail-io1-f45.google.com (mail-io1-f45.google.com [209.85.166.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 594971CBE8F for ; Wed, 23 Oct 2024 16:15:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729700137; cv=none; b=eHxilNSiWue8qCIJr4ZkaIDoUwSvgVPL9Q2BCamNOsQFgSEU8NnCHcap+wlz6Q3d7qBkzpg/TJIVtsv+loPdAzJNQX+mBp/F6HM80Mm4oep7i3IG0SZWucQS4XVbre3S7lZWLagLQuMfPdd99DidcIzwpWiKN5eMc8aYGxnFypI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729700137; c=relaxed/simple; bh=Uqwki1LuUEifgHw2K0BHZmq0KvpknnbwvrTuucKTFDA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=aohG4iItAmVPia+vVVoW5ue/fzvNPeFTLmBw5WtfULlkMNJorQPfhfLxIpWkvcTQ68Ow+xJMAvIFM2jGJz7HltrEHlOyTTy7oV97B/YmnHLgGq3fdmkA0TS2Tto6PD7IOiAyothAX9oo3a0PWWYT2WbDphkTC3LOEeAEEXKamY8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=0S/Xifnu; arc=none smtp.client-ip=209.85.166.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="0S/Xifnu" Received: by mail-io1-f45.google.com with SMTP id ca18e2360f4ac-83ac817aac3so171897839f.0 for ; Wed, 23 Oct 2024 09:15:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1729700134; x=1730304934; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CXtdybHKthx98tLnmZIwgxVHacDnHFHxsladV/YkWd8=; b=0S/XifnuEU2Q8H/wEp4nq5/G5VztuzzmReqJFK9Hpc5glWvJNCQNg5IySQxbcO0XI6 EP9IgbVKcGDxM+3CrlDgD0kyhY4vTL6plIDQXHxMtPuFjONcZ1LX9Ue6QIbyZ3AySlLi UtHCQHuq8kGNM0IGCUPfotoRyM3wmA1BqnW4a7WbnH0A78Ff6AsSULYchXYUkVf4zyt2 98Pb6lBsL0C81Uyhl9pHSJ/IlhT4bxbRj1IkGlzQALdh62lz+WQpSM25sRIu7A6Q8fqj 5Dd3x7AarRXbsH/rYVttYF6UhtL4WcqSsaNBp9TN6ngYWjdHPPm69bNctd753ty8ESdk 2Bmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729700134; x=1730304934; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CXtdybHKthx98tLnmZIwgxVHacDnHFHxsladV/YkWd8=; b=xG/S/ouytTp2nlk3TEZck9TewGMy9Nf/7c6cmBAfvzFXSGARCcVNZd05xrqRPKPARG UCz114okOzVvBwNmEwuGUyipWfIdCmob1suKTvApNPOPsKUqnKEIEnGjM8GkLZNCAMXq tq2OBa1qORXKSv2vHiOPZVjX3vWoZ/IoZNnBtI+uyQqhKm9e6uiYsquuUGAon23mXlCE 6cNuyBufI5fMkb39CjfUynpWjmyZfBzDP9nK1yVjzqMlzMoTFRJqZz3W3zxDQ+PfMoCg MnsgFTfDabe72RgWPfwwtD0eGQ+/hcJe6qxmIHend3p1hruld7nkDo5hYXBhQjktgC+s XBqg== X-Gm-Message-State: AOJu0Yzt/QDaQNhjgqF0rMWe9eDa4jGZjt9NUFZjPSK1NnTxAX0VUQev h3h02iqxQH0LTcJu9JktzEEU/R2ueIOy6OZNjxGYz68ndVIMgVZUd/eShECcm7C2KqBP3capPZv 9 X-Google-Smtp-Source: AGHT+IEKCM/588BO9fNZE0ipi0WMXOQlC9giondG4Enc3gQp6KnI8hjxhZE668FiCFEtJANLnABguQ== X-Received: by 2002:a05:6602:1603:b0:83a:a25a:cfaf with SMTP id ca18e2360f4ac-83af6155756mr303252739f.3.1729700133630; Wed, 23 Oct 2024 09:15:33 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4dc2a556c29sm2138180173.43.2024.10.23.09.15.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Oct 2024 09:15:33 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 7/7] io_uring/net: add provided buffer and bundle support to send zc Date: Wed, 23 Oct 2024 10:07:40 -0600 Message-ID: <20241023161522.1126423-8-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241023161522.1126423-1-axboe@kernel.dk> References: <20241023161522.1126423-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Provided buffers inform the kernel which buffer group ID to pick a buffer from for transfer. Normally that buffer contains the usual addr + length information, as well as a buffer ID that is passed back at completion time to inform the application of which buffer was used for the transfer. However, if registered and provided buffers are combined, then the provided buffer must instead tell the kernel which registered buffer index should be used, and the length/offset within that buffer. Rather than store the addr + length, the application must instead store this information instead. If provided buffers are used with send zc, then those buffers must be an index into a registered buffer. Change the mapping type to use KBUF_MODE_BVEC, which tells the kbuf handlers to turn the mappings into bio_vecs rather than iovecs. Then all that is needed is to setup our iov_iterator to use iov_iter_bvec(). Signed-off-by: Jens Axboe --- io_uring/net.c | 64 +++++++++++++++++++++++++++++++++--------------- io_uring/net.h | 10 ++++++-- io_uring/opdef.c | 1 + 3 files changed, 53 insertions(+), 22 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 154756762a46..c062b1c685bd 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -83,6 +83,8 @@ struct io_sr_msg { static int io_sg_from_iter(struct sk_buff *skb, struct iov_iter *from, size_t length); +static int io_sg_from_iter_iovec(struct sk_buff *skb, struct iov_iter *from, + size_t length); /* * Number of times we'll try and do receives if there's more data. If we @@ -581,33 +583,34 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) return IOU_OK; } -static int io_send_zc_import_single(struct io_kiocb *req, - unsigned int issue_flags) +static int __io_send_zc_import(struct io_kiocb *req, + struct io_async_msghdr *kmsg, int nsegs) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); - struct io_async_msghdr *kmsg = req->async_data; struct io_ring_ctx *ctx = req->ctx; struct io_mapped_ubuf *imu; int ret; u16 idx; - ret = -EFAULT; - io_ring_submit_lock(ctx, issue_flags); - if (sr->buf_index < ctx->nr_user_bufs) { + if (req->flags & REQ_F_BUFFER_SELECT) { + struct bio_vec *bv = kmsg->free_bvec ?: &kmsg->fast_bvec; + + WARN_ON_ONCE(bv == &kmsg->fast_bvec && nsegs > 1); + iov_iter_bvec(&kmsg->msg.msg_iter, ITER_SOURCE, bv, nsegs, sr->len); + } else { + if (WARN_ON_ONCE(nsegs != 1)) + return -EFAULT; + if (unlikely(sr->buf_index >= ctx->nr_user_bufs)) + return -EFAULT; idx = array_index_nospec(sr->buf_index, ctx->nr_user_bufs); imu = READ_ONCE(ctx->user_bufs[idx]); - io_req_set_rsrc_node(sr->notif, ctx); - ret = 0; - } - io_ring_submit_unlock(ctx, issue_flags); - if (unlikely(ret)) - return ret; + ret = io_import_fixed(ITER_SOURCE, &kmsg->msg.msg_iter, imu, + (u64)(uintptr_t)sr->buf, sr->len); + if (unlikely(ret)) + return ret; + } - ret = io_import_fixed(ITER_SOURCE, &kmsg->msg.msg_iter, imu, - (u64)(uintptr_t)sr->buf, sr->len); - if (unlikely(ret)) - return ret; kmsg->msg.sg_from_iter = io_sg_from_iter; return 0; } @@ -619,6 +622,16 @@ static int __io_send_import(struct io_kiocb *req, struct buf_sel_arg *arg, struct io_async_msghdr *kmsg = req->async_data; int ret = nsegs; + if (sr->flags & IORING_RECVSEND_FIXED_BUF) { + io_ring_submit_lock(req->ctx, issue_flags); + io_req_set_rsrc_node(sr->notif, req->ctx); + ret = __io_send_zc_import(req, kmsg, nsegs); + io_ring_submit_unlock(req->ctx, issue_flags); + if (unlikely(ret < 0)) + return ret; + return nsegs; + } + if (nsegs == 1) { sr->buf = arg->iovs[0].iov_base; ret = import_ubuf(ITER_SOURCE, sr->buf, sr->len, @@ -646,10 +659,13 @@ static int io_send_import(struct io_kiocb *req, unsigned int issue_flags) .nr_vecs = 1, }; + if (sr->flags & IORING_RECVSEND_FIXED_BUF) + arg.mode |= KBUF_MODE_BVEC; + if (kmsg->free_iov) { arg.nr_vecs = kmsg->free_iov_nr; arg.iovs = kmsg->free_iov; - arg.mode = KBUF_MODE_FREE; + arg.mode |= KBUF_MODE_FREE; } if (!(sr->flags & IORING_RECVSEND_BUNDLE)) @@ -1280,7 +1296,8 @@ void io_send_zc_cleanup(struct io_kiocb *req) } } -#define IO_ZC_FLAGS_COMMON (IORING_RECVSEND_POLL_FIRST | IORING_RECVSEND_FIXED_BUF) +#define IO_ZC_FLAGS_COMMON (IORING_RECVSEND_POLL_FIRST | \ + IORING_RECVSEND_FIXED_BUF | IORING_RECVSEND_BUNDLE) #define IO_ZC_FLAGS_VALID (IO_ZC_FLAGS_COMMON | IORING_SEND_ZC_REPORT_USAGE) int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) @@ -1399,8 +1416,13 @@ static int io_send_zc_import(struct io_kiocb *req, unsigned int issue_flags) struct io_async_msghdr *kmsg = req->async_data; int ret; + ret = io_send_import(req, issue_flags); + if (unlikely(ret < 0)) + return ret; + if (req->flags & REQ_F_BUFFER_SELECT) + return 0; if (sr->flags & IORING_RECVSEND_FIXED_BUF) - return io_send_zc_import_single(req, issue_flags); + return __io_send_zc_import(req, kmsg, 1); ret = import_ubuf(ITER_SOURCE, sr->buf, sr->len, &kmsg->msg.msg_iter); if (unlikely(ret)) @@ -1416,6 +1438,7 @@ int io_send_zc(struct io_kiocb *req, unsigned int issue_flags) { struct io_sr_msg *zc = io_kiocb_to_cmd(req, struct io_sr_msg); struct io_async_msghdr *kmsg = req->async_data; + unsigned int cflags; struct socket *sock; unsigned msg_flags; int ret, min_ret = 0; @@ -1476,7 +1499,8 @@ int io_send_zc(struct io_kiocb *req, unsigned int issue_flags) io_notif_flush(zc->notif); io_req_msg_cleanup(req, 0); } - io_req_set_res(req, ret, IORING_CQE_F_MORE); + cflags = io_put_kbuf(req, ret, issue_flags); + io_req_set_res(req, ret, cflags | IORING_CQE_F_MORE); return IOU_OK; } diff --git a/io_uring/net.h b/io_uring/net.h index 52bfee05f06a..e052762cf85d 100644 --- a/io_uring/net.h +++ b/io_uring/net.h @@ -5,9 +5,15 @@ struct io_async_msghdr { #if defined(CONFIG_NET) - struct iovec fast_iov; + union { + struct iovec fast_iov; + struct bio_vec fast_bvec; + }; /* points to an allocated iov, if NULL we use fast_iov instead */ - struct iovec *free_iov; + union { + struct iovec *free_iov; + struct bio_vec *free_bvec; + }; int free_iov_nr; int namelen; __kernel_size_t controllen; diff --git a/io_uring/opdef.c b/io_uring/opdef.c index a2be3bbca5ff..6203a7dd5052 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -422,6 +422,7 @@ const struct io_issue_def io_issue_defs[] = { .needs_file = 1, .unbound_nonreg_file = 1, .pollout = 1, + .buffer_select = 1, .audit_skip = 1, .ioprio = 1, #if defined(CONFIG_NET)