From patchwork Tue Feb 18 22:01:33 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Wei X-Patchwork-Id: 13980965 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 894821B87FB for ; Tue, 18 Feb 2025 22:01:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739916104; cv=none; b=UWweB9MEJpgyZbfWai0fNCvsMSvzmXRaWvD0nXfGY/mIYIth03ZQzbtLDbxKFRYhEQjLBSSi8gZ2AtXnRyzuBK3ItL7w3iURJDrkTJ34FrVgwf2/Pah3EEFh2a6762Mf9spTfLJT6Ygeb2R5t5E/5vMnkyyAynn8GuxOLevTAoY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739916104; c=relaxed/simple; bh=U9GBcTbEhnvXwzkuE+H/Yp0pys+0CVLQaKSQ1CPjWqk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=S/nSOFEBpyxkBOoiz6RikK6+lX1/z/zkd2gLN5dUS0M8wOVHv4MtaHqEQOYEyCzNaHx2EHRtsrOrlKdvKKvwsJkmUj9Lst+S60V7FfvzL3d2yxhvaJQDdQjTd6j4+ZGiR1HY520I1Z4rZhsN/NWRE847jAtKID/gjb20isF2KrY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk; spf=none smtp.mailfrom=davidwei.uk; dkim=pass (2048-bit key) header.d=davidwei-uk.20230601.gappssmtp.com header.i=@davidwei-uk.20230601.gappssmtp.com header.b=kTE8upDE; arc=none smtp.client-ip=209.85.214.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=davidwei-uk.20230601.gappssmtp.com header.i=@davidwei-uk.20230601.gappssmtp.com header.b="kTE8upDE" Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-220d39a5627so88763495ad.1 for ; Tue, 18 Feb 2025 14:01:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=davidwei-uk.20230601.gappssmtp.com; s=20230601; t=1739916102; x=1740520902; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=e6p0eNHlExhjC1VzHu83sWwv40KIotSYsW7eaEIok/w=; b=kTE8upDEJu/7lQzUXOFD/MJBmEPXadjOjjP95CSg1SotBMYg6m8WrhB8YLbJQit/QK GyZcR06dZqypzX7gage5LnAPXd8iieWhA6up4V+tofZ+WCS/wwFILcB3dCPg2rLovQvZ Fi/Fmj4NjShBCDAHELlzU6G7Qvzs8giZzoEgyMVgigxokwsBe437B13PWYS8PM6Ba1V1 EqpibI4pid3//Z3cDQsl2fHFZvYMAhdrQv2OwIj/xptSOJMjvwTFVqa39TmN/E7ZJvg2 zx/T8o94WIZ8ZjthmD3OCKpnjtd9fk19HkF+bNUR3I6By0fimob6G0OSYToU4QtpQpCv 1uVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739916102; x=1740520902; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=e6p0eNHlExhjC1VzHu83sWwv40KIotSYsW7eaEIok/w=; b=jVLOQtOIqIH1djr+ORZ+DAYPgG3XqE/8J73AO92hrFmjhMsjYxsq+Xypbu/1iF7pAw Uc0FK0X7nwDfhcdFYd0U5VLvtoYo4rlxAPBAfTNHDUWHSilLiDYConRsYN7pLLLDvQAT Bw4CZwsE90riUY4OHQJpEpgifnXBZRdYvlwLi+Vu3VcAnqOmKfkmgZBFi/wPrVV43VRT 9h7u/pqlDUa+SYOOnmGSR39t6w8X/AFF73A0t+Q/apclmT+nIDxFTsocj52lbDKD5k5T WiY4WLpY09qqnbsAkFkVDeCvJEAabZB1u4Y63tBi3t8DXabaBnpNRpPfQDj/TaxJ3mMd +rOw== X-Gm-Message-State: AOJu0Yzc9Y6QvbU4cd9IFxlk+VRzicN0xYgN7haL6+r4QAO2eG+RMc5F g/0NPq0Z1JfXW2JL1TKHfopiiI1+IJ8jzievbQ5OpobuP5JgjFKeQxf3EA+mOB6pwI570O7Uv+G Y X-Gm-Gg: ASbGnct/Xhzl2npyqFn+3OS3TBe+prI4NG5FVuVEG4XWxV7YpO/TrRc9KA0q2gOO5En Qg99UJTmNPuz9iEHdxYSkjMEdGXOu6Kps+IqQpwfK5xW/RwhUYxC0hHBto78CrXEyrMkrCASSi/ nzhyd4SFVEFQw1zbAUG8IDhLgHqW1JHxmV38L2SARB3M7aEXtVWyirInasy1C5NBuBqBVcV8ad1 1gLRONswNHcHmNPtRIH8lIXyZcV8AcS+spnuiDfstVYKizfLF7F+sFmIBZrNwedX5GRsS1lLkxE X-Google-Smtp-Source: AGHT+IGlx3m5MCiO8Zg9auf9bVZQjrQsAmNJsaRQnBxzGUecJ8W3oGwRoNCcE2J4ejoTZvCbg7UJZw== X-Received: by 2002:a17:902:ec84:b0:220:d532:834e with SMTP id d9443c01a7336-221040255d9mr233647425ad.19.1739916101788; Tue, 18 Feb 2025 14:01:41 -0800 (PST) Received: from localhost ([2a03:2880:ff:73::]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-220d545d4c9sm92906565ad.140.2025.02.18.14.01.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Feb 2025 14:01:41 -0800 (PST) From: David Wei To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH liburing v2 1/4] liburing: sync io_uring headers Date: Tue, 18 Feb 2025 14:01:33 -0800 Message-ID: <20250218220136.2238838-2-dw@davidwei.uk> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20250218220136.2238838-1-dw@davidwei.uk> References: <20250218220136.2238838-1-dw@davidwei.uk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 In preparation of syncing headers w/ zero copy rx changes, first sync other unrelated changes. Signed-off-by: David Wei --- src/include/liburing/io_uring.h | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/src/include/liburing/io_uring.h b/src/include/liburing/io_uring.h index 765919883cff..452240a6ebb4 100644 --- a/src/include/liburing/io_uring.h +++ b/src/include/liburing/io_uring.h @@ -364,7 +364,7 @@ enum io_uring_op { * result will be the number of buffers send, with * the starting buffer ID in cqe->flags as per * usual for provided buffer usage. The buffers - * will be contiguous from the starting buffer ID. + * will be contigious from the starting buffer ID. */ #define IORING_RECVSEND_POLL_FIRST (1U << 0) #define IORING_RECV_MULTISHOT (1U << 1) @@ -424,7 +424,7 @@ enum io_uring_msg_ring_flags { * IO completion data structure (Completion Queue Entry) */ struct io_uring_cqe { - __u64 user_data; /* sqe->user_data submission passed back */ + __u64 user_data; /* sqe->user_data value passed back */ __s32 res; /* result code for this event */ __u32 flags; @@ -616,6 +616,10 @@ enum io_uring_register_op { /* clone registered buffers from source ring to current ring */ IORING_REGISTER_CLONE_BUFFERS = 30, + /* send MSG_RING without having a ring */ + IORING_REGISTER_SEND_MSG_RING = 31, + + /* resize CQ ring */ IORING_REGISTER_RESIZE_RINGS = 33, IORING_REGISTER_MEM_REGION = 34, From patchwork Tue Feb 18 22:01:34 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Wei X-Patchwork-Id: 13980967 Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 73D8E1C701B for ; Tue, 18 Feb 2025 22:01:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739916104; cv=none; b=nG2J1f0K0dIqe/+vCJzgHGGKipX+NiYPpVlxMFCSVlzcFXdczjPHupQisnJAtAaCmRkNln+9esDDURyJQ6rLN005kPvDYOA9LuYbxTRlSXOeBu7437eK6RkTjoRQoE9e3QjOsd/uOn+mmwGmyhuD0Ca6R1f4srTh/JN83sjEbOo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739916104; c=relaxed/simple; bh=VCN7bDMu1wbb30x4oD5ih9cJjZx9VYiqK6rTypUeh0Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=OmeVdiKoRp0/q3ORNmT0RPcfy+xADDLkxOczAFLh2/1enzPA301IDMKLqlD3VzBJrme7t8dSCirozWdKLyxZHFUUJ8AD4gPNpdKQBkTf5F3nAn0KAtYwTUtNb6QxGpXjW/odyfhkwC8/CuzPhD7/Q3QboE7mYJlpKV1nFRSFBP0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk; spf=none smtp.mailfrom=davidwei.uk; dkim=pass (2048-bit key) header.d=davidwei-uk.20230601.gappssmtp.com header.i=@davidwei-uk.20230601.gappssmtp.com header.b=KEPbfVlO; arc=none smtp.client-ip=209.85.216.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=davidwei-uk.20230601.gappssmtp.com header.i=@davidwei-uk.20230601.gappssmtp.com header.b="KEPbfVlO" Received: by mail-pj1-f47.google.com with SMTP id 98e67ed59e1d1-2fc11834404so8508711a91.0 for ; Tue, 18 Feb 2025 14:01:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=davidwei-uk.20230601.gappssmtp.com; s=20230601; t=1739916103; x=1740520903; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vhYVfmU8RD9dhfxIQEmEnt4HeDToaTcnJxE8nhiL6RQ=; b=KEPbfVlOdZTfSmf8cNbnIPMhEOOpL/eWeLez/V7Zi69zMfANXD2YccXlSt9jfEbSQj WMqy3FtlLtFm3Bl+L7hY9BLt8yFKcRD0NOmY90J0z+4WrcUvXKDdHeHdDO3Kgc6MufT9 NipB9z4Zw51Fek12kcw3nKmSFqAB2b7Vqh97rg56umIRAwdVDz2azGaDPQj1UTL1Y7jG /+9Y+POhfuifEct0bvsrgPF4Q1VuLRE9/o0hLLwBpsuwQH5DmlgRBAjh7mvxIiFa+iq6 HVR/IbUbgLlM75fnFK6hQJsvQseD+qACZ4Af/uJpFwVfDGNupULTpmgnGrZG6VaaJA8E kF6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739916103; x=1740520903; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vhYVfmU8RD9dhfxIQEmEnt4HeDToaTcnJxE8nhiL6RQ=; b=buNn7mV5v6hEzJh5FedkFIh3fFPbqLb4r50qaqqIioQcO1X9D7yhBSvemoUAUTyp4z alsl+0n6b9YqMU6O+3JcBPqF9+Ny+6Pjj8SDMm8dqKwGOgChWAogkwx1IB7VB2a7CviV uQoOuASN0mkZe7uhOLQJp1X2NE1b0iGYHc1cAL6Rn6a9LORWd6u9BbR11vyVWRpp4z6c x0J06Xvd2RzIOMJfXT8XgcTZNO0I+JwMj7SOnptC/+xYniStzC9y1KzW8lmNwm46m7CV u8qVf+UxPX9ok7QSQy0oquXITAnMdcdIYGaCs6c4ICW8ULfxVb4yT04QIcOHP7+QWbw+ L7vw== X-Gm-Message-State: AOJu0YziYREFpgYr4X4B6v5u0lHUL0oGrdbB5PL1d5KW4CFIyV3ZGnZ6 ttKellPdpcs+PydavW6AipPXSZFMk8Qm4gem2hR6iYsdejt0eP39ZlVFLpOfPiccTqnRN6f2naF 3 X-Gm-Gg: ASbGncv6vFd7BZPEWHAYrOtmzuW13dPUZgbN30sidBMdtEJpjzD1dbZEw6tVu6q00Nn 4Fx62K9iXlY3ZJ10SXWwfkKWqCdc4SDuJMHRewjRCblNsemflaNd8cw8KBunlhiY0QateV8zJeu 87LARnaq/wRNaKSnsvELLPltfTXLZYihnpNK117MKDJQEgmgztTf0lUtt/zg7hyuvbxQGoR6R2z JlIuNMUSAqczqbLcSrY/q9YaHqw4UxAUnkWTvfV6YCAWBcEQt00J2vFsSS9t3ZLofSLMGrbs7jR X-Google-Smtp-Source: AGHT+IEc8tTJw6hJefBk5vsffgKksqLjfH9jVWVNBedmvlG2W3Y0bm0gFIRiN+qyxdBfXEDxQmS9XA== X-Received: by 2002:a17:90b:17c7:b0:2ee:cd83:8fe7 with SMTP id 98e67ed59e1d1-2fc411679e8mr23399633a91.35.1739916102701; Tue, 18 Feb 2025 14:01:42 -0800 (PST) Received: from localhost ([2a03:2880:ff:47::]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2fc13ad35absm10481291a91.25.2025.02.18.14.01.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Feb 2025 14:01:42 -0800 (PST) From: David Wei To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH liburing v2 2/4] zcrx: sync io_uring headers Date: Tue, 18 Feb 2025 14:01:34 -0800 Message-ID: <20250218220136.2238838-3-dw@davidwei.uk> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20250218220136.2238838-1-dw@davidwei.uk> References: <20250218220136.2238838-1-dw@davidwei.uk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Sync linux/io_uring.h with zcrx changes. Signed-off-by: David Wei --- src/include/liburing/io_uring.h | 54 +++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/src/include/liburing/io_uring.h b/src/include/liburing/io_uring.h index 452240a6ebb4..d2fcd1d22ea0 100644 --- a/src/include/liburing/io_uring.h +++ b/src/include/liburing/io_uring.h @@ -87,6 +87,7 @@ struct io_uring_sqe { union { __s32 splice_fd_in; __u32 file_index; + __u32 zcrx_ifq_idx; __u32 optlen; struct { __u16 addr_len; @@ -262,6 +263,7 @@ enum io_uring_op { IORING_OP_FTRUNCATE, IORING_OP_BIND, IORING_OP_LISTEN, + IORING_OP_RECV_ZC, /* this goes last, obviously */ IORING_OP_LAST, @@ -619,6 +621,9 @@ enum io_uring_register_op { /* send MSG_RING without having a ring */ IORING_REGISTER_SEND_MSG_RING = 31, + /* register a netdev hw rx queue for zerocopy */ + IORING_REGISTER_ZCRX_IFQ = 32, + /* resize CQ ring */ IORING_REGISTER_RESIZE_RINGS = 33, @@ -920,6 +925,55 @@ enum io_uring_socket_op { SOCKET_URING_OP_SETSOCKOPT, }; +/* Zero copy receive refill queue entry */ +struct io_uring_zcrx_rqe { + __u64 off; + __u32 len; + __u32 __pad; +}; + +struct io_uring_zcrx_cqe { + __u64 off; + __u64 __pad; +}; + +/* The bit from which area id is encoded into offsets */ +#define IORING_ZCRX_AREA_SHIFT 48 +#define IORING_ZCRX_AREA_MASK (~(((__u64)1 << IORING_ZCRX_AREA_SHIFT) - 1)) + +struct io_uring_zcrx_offsets { + __u32 head; + __u32 tail; + __u32 rqes; + __u32 __resv2; + __u64 __resv[2]; +}; + +struct io_uring_zcrx_area_reg { + __u64 addr; + __u64 len; + __u64 rq_area_token; + __u32 flags; + __u32 __resv1; + __u64 __resv2[2]; +}; + +/* + * Argument for IORING_REGISTER_ZCRX_IFQ + */ +struct io_uring_zcrx_ifq_reg { + __u32 if_idx; + __u32 if_rxq; + __u32 rq_entries; + __u32 flags; + + __u64 area_ptr; /* pointer to struct io_uring_zcrx_area_reg */ + __u64 region_ptr; /* struct io_uring_region_desc * */ + + struct io_uring_zcrx_offsets offsets; + __u64 __resv[4]; +}; + #ifdef __cplusplus } #endif From patchwork Tue Feb 18 22:01:35 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Wei X-Patchwork-Id: 13980968 Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5D04C1C701B for ; Tue, 18 Feb 2025 22:01:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739916106; cv=none; b=m+OCTFBjYnaEN+tarqO6LdxUsyRJEI2M5vACOQajIJj46xQl9liaZ2J7TZxCNp/jgSyLlHKxiowR91htYx2MxpEZt7gGKNxyMElZF2xaX+MTPgRt2LbzBj+XUhZN4mARHktcjDl/IIUC2CcYeblnlg5jLlqXStC34pbNzJ3iF5A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739916106; c=relaxed/simple; bh=nBoWU3+DWLHvBK++LtK+tqnpWIBKU2BFtAP2zUqTfOA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QTXiKg5oViPJPcBrVKjnIjH5dycxfhzZ+3JtgPt0U+8daUdOqJRjmRaJDm92gR6v7AVd4rrVyS3pYBQZ2YY7AcT5u86eIEm2BDBIMW6Azn5j2dOMWdox7rwlClGcJW7SZXWhYN8gbH15H90FB+MzOr1pMeq42z3ER9Wr17i+FKU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk; spf=none smtp.mailfrom=davidwei.uk; dkim=pass (2048-bit key) header.d=davidwei-uk.20230601.gappssmtp.com header.i=@davidwei-uk.20230601.gappssmtp.com header.b=QDQ50ZsL; arc=none smtp.client-ip=209.85.216.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=davidwei-uk.20230601.gappssmtp.com header.i=@davidwei-uk.20230601.gappssmtp.com header.b="QDQ50ZsL" Received: by mail-pj1-f54.google.com with SMTP id 98e67ed59e1d1-2fc20e0f0ceso7806579a91.3 for ; Tue, 18 Feb 2025 14:01:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=davidwei-uk.20230601.gappssmtp.com; s=20230601; t=1739916103; x=1740520903; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=E2IslT3YuiEU7iZyJVnV/hcP9Zor4bYpNa5m5cjTKws=; b=QDQ50ZsLQPc4uC7y3NRngS2fYWROjBsvXE+elx/6eWMmlpz8CqXi1rt57CzGheUHQU 2TVSX3m7fTlVULuX90lusiBXoc+141UE/T9etP6i3o5SghnaHtbQd0mnRQG1TMTjqZ9o UULzzCVdpRrohhBRcVgjgUoEwkoUw4/RVnUaAFqQXxjG9IyLQPvF/Yfp071IBF2IuDDn zUnt4bN5dcnfC9EF4KYEoMV0ANm84UFE8/JRno4dv9Twh0OdXeObhL8Ti8fyKfDdudtf HjvpfyhkZSEiSn8pR1P67uMeCHoDMmWB2LQYSfMbW2l6htCZ7WIi9u0lfYKSlAggkwpK MuAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739916103; x=1740520903; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=E2IslT3YuiEU7iZyJVnV/hcP9Zor4bYpNa5m5cjTKws=; b=Qg0fRqAsYywdYwQHwJ2yEoOC5of9bu/qG7g3awF0bGMKXDyCguVDbYccMO5rQ8mXWg rutLMjOEP4xb/lahSwG+YDwPv1IsGJS7wvvWwVQUkJvQVtHEorcSFZMk7oC3iqymw7tw UXoW5wzsCcgwwOUwGKS1pOxwqTuMZYdF/UcN0Gd+124KqKq3lUc9Uhgge9JHRsiHMtC4 XYPaCThTU2UCXeN/YFPMDPrVTO4hIgeMaYKJqWDjKJaL3wGBx4QTQ1NvWKl5uTcagAq4 +XGherfg4qohhPJuGptIYeG7Ufwu4bxZKVBNmj65nudIa59TBPD/w5QuUb41tmu6F881 TV+g== X-Gm-Message-State: AOJu0YxL2DbrCnlm3CNQ11c2UQuIqBLiFkv+haj4ulJpghRXFEyZf7LR JS1UM83UHicQ+T+1a9qlKFUUtMUIrHON1Z4mcUwP/1PpjfjGc9fRDcbPm4gaU80qnmMxIRLFE8P R X-Gm-Gg: ASbGncsm+tIY2wR9OdWq6yjsp4fZzr2Lpap8Et94PLrDkE9zQSydrUNe8vfvcGQVSIA sG8mZRXWy96Vj3YLJwCqgtjFGot1dBtwv+KZEcb+tGj3L/E64AKC4v58pmJOq/8jtU2H07zqqyo BjjMgS9vfY3IuAKipq8KL1mTPUL+RPS0d5nL9zG1zmoayZb2li6QFUEvEZYS4q7R9R0WL9aNjH2 dqJW9sdGFj6OfBHKF0d4YbiL2W499+gcpo7jPBa5efmNb+/TUar3LLetXw1IMvqe/KkMsf89oQ= X-Google-Smtp-Source: AGHT+IGAP9HxUHN9WtFGAaADlSkgC+Wq6FK0DkgieufwL9d+iMSDDwSeUi2rb5yAEwA9GnTExJ7aOw== X-Received: by 2002:a17:90b:3d0f:b0:2ee:c918:cd60 with SMTP id 98e67ed59e1d1-2fcb5a39f7dmr1715793a91.20.1739916103562; Tue, 18 Feb 2025 14:01:43 -0800 (PST) Received: from localhost ([2a03:2880:ff:c::]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2fbf98cfd19sm12723656a91.14.2025.02.18.14.01.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Feb 2025 14:01:43 -0800 (PST) From: David Wei To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH liburing v2 3/4] zcrx: add basic support Date: Tue, 18 Feb 2025 14:01:35 -0800 Message-ID: <20250218220136.2238838-4-dw@davidwei.uk> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20250218220136.2238838-1-dw@davidwei.uk> References: <20250218220136.2238838-1-dw@davidwei.uk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add basic support for zcrx with a thin wrapper around IORING_REGISTER_ZCRX_IFQ and a struct for the refill queue. Signed-off-by: David Wei --- src/include/liburing.h | 12 ++++++++++++ src/liburing-ffi.map | 2 ++ src/liburing.map | 2 ++ src/register.c | 6 ++++++ 4 files changed, 22 insertions(+) diff --git a/src/include/liburing.h b/src/include/liburing.h index 49b4edf437b2..6393599cb3bf 100644 --- a/src/include/liburing.h +++ b/src/include/liburing.h @@ -132,6 +132,16 @@ struct io_uring { unsigned pad2; }; +struct io_uring_zcrx_rq { + __u32 *khead; + __u32 *ktail; + __u32 rq_tail; + unsigned ring_entries; + + struct io_uring_zcrx_rqe *rqes; + void *ring_ptr; +}; + /* * Library interface */ @@ -265,6 +275,8 @@ int io_uring_register_file_alloc_range(struct io_uring *ring, int io_uring_register_napi(struct io_uring *ring, struct io_uring_napi *napi); int io_uring_unregister_napi(struct io_uring *ring, struct io_uring_napi *napi); +int io_uring_register_ifq(struct io_uring *ring, + struct io_uring_zcrx_ifq_reg *reg); int io_uring_register_clock(struct io_uring *ring, struct io_uring_clock_register *arg); diff --git a/src/liburing-ffi.map b/src/liburing-ffi.map index 968ccbc67366..fe14adb6d83f 100644 --- a/src/liburing-ffi.map +++ b/src/liburing-ffi.map @@ -227,4 +227,6 @@ LIBURING_2.9 { } LIBURING_2.8; LIBURING_2.10 { + global: + io_uring_register_ifq; } LIBURING_2.9; diff --git a/src/liburing.map b/src/liburing.map index 264a94946e90..d1661d9d61f9 100644 --- a/src/liburing.map +++ b/src/liburing.map @@ -113,4 +113,6 @@ LIBURING_2.9 { } LIBURING_2.8; LIBURING_2.10 { + global: + io_uring_register_ifq; } LIBURING_2.9; diff --git a/src/register.c b/src/register.c index 0fff208cd5f5..99337d13135d 100644 --- a/src/register.c +++ b/src/register.c @@ -422,6 +422,12 @@ int io_uring_clone_buffers(struct io_uring *dst, struct io_uring *src) return io_uring_clone_buffers_offset(dst, src, 0, 0, 0, 0); } +int io_uring_register_ifq(struct io_uring *ring, + struct io_uring_zcrx_ifq_reg *reg) +{ + return do_register(ring, IORING_REGISTER_ZCRX_IFQ, reg, 1); +} + int io_uring_resize_rings(struct io_uring *ring, struct io_uring_params *p) { unsigned sq_head, sq_tail; From patchwork Tue Feb 18 22:01:36 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Wei X-Patchwork-Id: 13980969 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8AB351D5CD3 for ; Tue, 18 Feb 2025 22:01:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739916107; cv=none; b=O5SvAw6wUj9+LqLTzaOwAiZbgyc9+IjLv6giNZWNOx6dPpr6YUuu5YedPdk6wowuCgol23d9gHPPv/Kx5J+mTZtToTeZSaOEx2WG76LluuqnOq61a3GrYy3/8oOJoLMIzX+aFAe1O4J8kUcRzcflbHUXukekKoMogBZZqh57zcs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739916107; c=relaxed/simple; bh=hx0Ol2zC53ViGWqkI6U1E08lp17uCfzWySD+Jv53ckc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DNdH3SXaYV2goUuZGZHoBYgkfWH9whc6laUqIAnaIuBv6DQaOitCnRRdsZ2UnKjbUvBTvpNRapCozfP4ffJF1PROSKVbI5dvZ1ZGYqrzys7Qyyv1sZ9hOPUvFtm+Iny6AdUnwlzS2Ki29dWYcavD50lC89PSjdBjEiKA1lVFrJI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk; spf=none smtp.mailfrom=davidwei.uk; dkim=pass (2048-bit key) header.d=davidwei-uk.20230601.gappssmtp.com header.i=@davidwei-uk.20230601.gappssmtp.com header.b=gPz4s2CO; arc=none smtp.client-ip=209.85.214.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=davidwei-uk.20230601.gappssmtp.com header.i=@davidwei-uk.20230601.gappssmtp.com header.b="gPz4s2CO" Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-22114b800f7so57412445ad.2 for ; Tue, 18 Feb 2025 14:01:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=davidwei-uk.20230601.gappssmtp.com; s=20230601; t=1739916105; x=1740520905; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ptPRb7uguGR3+ra0APMzhcXk6fGHS7pif0n+0v0yMKI=; b=gPz4s2COSHZbaubiMVzKl3FCaeSHv0Y7EGBeDXFTREEyET9HN9uxY6+ZqFAfA372nY l2dH35uk8cHrB7U5WK5udWAigj14KuRNxMB1FRlalohLtLfEIh68Yo2reVEqa2PfWHY3 YkaMbC3245kKU8tHRgSGCry5SBtu4+wZtFFE4xFX3+V9dekefWpSkJt7ZAUMrsCI4UMC nfIkTAYP9hlZqCcvLcYqk/xOb5j5GYOBiicTkUBZ6QyW+Poh+HOh/uthxltbqzBVHuYE XpGgKX1rhGUVDguhO8onmQ81Mi+haX3SmHg8LlsEoKxdCQF3qERe/fe3PN+Z6CTMse0t NGBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739916105; x=1740520905; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ptPRb7uguGR3+ra0APMzhcXk6fGHS7pif0n+0v0yMKI=; b=jwEqd8dOfRCwxfpz6//y9htTbnzvGGHfXzPuy6z/ejsR5PxQQq38zl+OAXupUd079A Hnc9iHuj+GRQmNKfv9gSonzccESySQgx1onJYB6MeWZvgCZSnJaos5dGIvcUl9+x3xcP DGBQ2UGGnUGE2Gbq+6xPi887Imwss/gDvzqYeXhdyHuBzY5fRvCaH5LPsS8GURDInJgR wSPA+D8cvqxJseboPEKtepw5tUH3kkE2OgLyjNIQ4bj6tu3E8ziDTRoJA+eRaE4lE8iz vQ8oDj4gsdhZXoWtUXOBKq/9gCKgwMeCsRBfwHvuw9iW1mq+S2WMEdnY5r4s8MC9RgC4 xs/w== X-Gm-Message-State: AOJu0YwMDvwxF6pcMPEitoggj5xrelDl3OXaY9OwEzwQPERcuLP9Ogu9 exJ6pNPjSH14tKn/4sdswTyxeiPgqmKpxbysX5os0SHxUKWuzNfcQ2jSv/jQIdBc1eITJgyFL5k W X-Gm-Gg: ASbGnctbJRk5aE60d4YQ+duVQR+uFNgBnK0lA6DKKcHtCVyEj5aGMWuxAvMvZnT4y2s uS+pbK4ET5cUtN6mGW6LSxEP7qXKJY5ac7w48KGX0tmkuehPefUlkuWeblWZqAym1Zek8ZdX0qJ vvNyA5c2IONbcxv1j/VfWoF5fyqBahr5OV70Q6elkWBrtYgzr6h7ff90wn04o2ozXg53blV9E25 fUy2fxCdHWaZBwpv3bA98FHklWyqP7CA54FYLsp02H4n2Y7XapOZNzwji/H8ITBPRV8RJIoQMgy X-Google-Smtp-Source: AGHT+IHYL19PJ4GFVZf2DQQxoKrsQN68oGJessiHsM+2OZG2wO+ChD7idgctGrlB+WLyOvrd5asBVA== X-Received: by 2002:a17:902:ccca:b0:220:e362:9b1a with SMTP id d9443c01a7336-22170987b55mr18615455ad.25.1739916104643; Tue, 18 Feb 2025 14:01:44 -0800 (PST) Received: from localhost ([2a03:2880:ff:13::]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-220d545cff5sm92034245ad.125.2025.02.18.14.01.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Feb 2025 14:01:44 -0800 (PST) From: David Wei To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH liburing v2 4/4] zcrx: add unit test Date: Tue, 18 Feb 2025 14:01:36 -0800 Message-ID: <20250218220136.2238838-5-dw@davidwei.uk> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20250218220136.2238838-1-dw@davidwei.uk> References: <20250218220136.2238838-1-dw@davidwei.uk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Tests for registration and a basic recv test. No zero copy is actually happening but it does test the copy fallback. Signed-off-by: Pavel Begunkov Signed-off-by: David Wei --- test/Makefile | 1 + test/zcrx.c | 918 ++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 919 insertions(+) create mode 100644 test/zcrx.c diff --git a/test/Makefile b/test/Makefile index 64e1480867d4..d92379ad9ed0 100644 --- a/test/Makefile +++ b/test/Makefile @@ -239,6 +239,7 @@ test_srcs := \ wakeup-hang.c \ wq-aff.c \ xattr.c \ + zcrx.c \ # EOL # Please keep this list sorted alphabetically. diff --git a/test/zcrx.c b/test/zcrx.c new file mode 100644 index 000000000000..d3221d3c2b83 --- /dev/null +++ b/test/zcrx.c @@ -0,0 +1,918 @@ +/* SPDX-License-Identifier: MIT */ +/* + * Simple test case showing using send and recv through io_uring + */ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "liburing.h" +#include "helpers.h" + +static unsigned int ifidx, rxq; + +/* the hw rxq must consume 128 of these pages, leaving 4 left */ +#define AREA_PAGES 132 +#define PAGE_SIZE 4096 +#define AREA_SZ AREA_PAGES * PAGE_SIZE +#define RQ_ENTRIES 128 +/* this is one more than the # of free pages after filling hw rxq */ +#define LOOP_COUNT 5 +#define DEV_ENV_VAR "NETIF" +#define RXQ_ENV_VAR "NETRXQ" +#define RING_FLAGS (IORING_SETUP_DEFER_TASKRUN | \ + IORING_SETUP_CQE32 | \ + IORING_SETUP_SINGLE_ISSUER) + +static char str[] = "iv5t4dl500w7wsrf14fsuq8thptto0z7i2q62z1p8dwrv5u4kaxpqhm2rb7bapddi5gfkh7f9695eh46t2o5yap2y43gstbsq3n90bg1i7zx1m4wojoqbuxhsrw4s4y3sh9qp57ovbaa2o9yaqa7d4to2vak1otvgkoxs5t0ovjbe6roginrjeh906kmjn1289jlho9a1bud02ex4xr3cvfcybpl6axnr117p0aesb3070wlvj91en7tpf8nyb1e"; + +#define MSG_SIZE 512 + +#define PORT 10202 +#define HOST "127.0.0.1" + +static int probe_zcrx(void *area) +{ + struct io_uring_zcrx_area_reg area_reg = { + .addr = (__u64)(unsigned long)area, + .len = AREA_SZ, + .flags = 0, + }; + struct io_uring_zcrx_ifq_reg reg = { + .if_idx = ifidx, + .if_rxq = rxq, + .rq_entries = RQ_ENTRIES, + .area_ptr = (__u64)(unsigned long)&area_reg, + }; + struct io_uring ring; + int ret; + + ret = t_create_ring(8, &ring, RING_FLAGS); + if (ret == T_SETUP_SKIP) { + fprintf(stderr, "required ring flags are not supported, skip\n"); + return T_EXIT_SKIP; + } + if (ret) { + fprintf(stderr, "probe ring setup failure\n"); + return T_EXIT_FAIL; + } + + ret = io_uring_register_ifq(&ring, ®); + if (ret == -EINVAL) { + fprintf(stderr, "zcrx is not supported, skip\n"); + return T_EXIT_SKIP; + } + if (ret) { + fprintf(stderr, "probe zcrx register fail %i\n", ret); + return T_EXIT_FAIL; + } + io_uring_queue_exit(&ring); + return T_EXIT_PASS; +} + +static int try_register_ifq(struct io_uring_zcrx_ifq_reg *reg) +{ + struct io_uring ring; + int ret; + + ret = t_create_ring(8, &ring, RING_FLAGS); + if (ret != T_SETUP_OK) { + fprintf(stderr, "ring create failed: %d\n", ret); + exit(T_EXIT_FAIL); + } + + ret = io_uring_register_ifq(&ring, reg); + io_uring_queue_exit(&ring); + return ret; +} + +static int test_invalid_if(void *area) +{ + int ret; + struct io_uring_zcrx_area_reg area_reg = { + .addr = (__u64)(unsigned long)area, + .len = AREA_SZ, + .flags = 0, + }; + struct io_uring_zcrx_ifq_reg reg = { + .if_idx = -1, + .if_rxq = rxq, + .rq_entries = RQ_ENTRIES, + .area_ptr = (__u64)(unsigned long)&area_reg, + }; + + ret = try_register_ifq(®); + if (ret != -EINVAL && ret != -ENODEV) { + fprintf(stderr, "registered invalid IF %i\n", ret); + return T_EXIT_FAIL; + } + + reg.if_idx = ifidx; + reg.if_rxq = -1; + + ret = try_register_ifq(®); + if (ret != -EINVAL) { + fprintf(stderr, "registered invalid IFQ %i\n", ret); + return T_EXIT_FAIL; + } + return T_EXIT_PASS; +} + +static int test_invalid_ifq_collision(void *area) +{ + struct io_uring ring, ring2; + struct io_uring_zcrx_area_reg area_reg = { + .addr = (__u64)(unsigned long)area, + .len = AREA_SZ, + .flags = 0, + }; + struct io_uring_zcrx_ifq_reg reg = { + .if_idx = ifidx, + .if_rxq = rxq, + .rq_entries = RQ_ENTRIES, + .area_ptr = (__u64)(unsigned long)&area_reg, + }; + int ret; + + ret = t_create_ring(8, &ring, RING_FLAGS); + if (ret != T_SETUP_OK) { + fprintf(stderr, "ring create failed: %d\n", ret); + return T_EXIT_FAIL; + } + ret = t_create_ring(8, &ring2, RING_FLAGS); + if (ret != T_SETUP_OK) { + fprintf(stderr, "ring2 create failed: %d\n", ret); + return T_EXIT_FAIL; + } + + ret = io_uring_register_ifq(&ring, ®); + if (ret) { + fprintf(stderr, "initial registration failed %i\n", ret); + return T_EXIT_FAIL; + } + + /* register taken ifq */ + ret = io_uring_register_ifq(&ring, ®); + if (!ret) { + fprintf(stderr, "registered taken queue\n"); + return T_EXIT_FAIL; + } + + ret = io_uring_register_ifq(&ring2, ®); + if (!ret) { + fprintf(stderr, "registered taken queue ring2\n"); + return T_EXIT_FAIL; + } + + io_uring_queue_exit(&ring); + io_uring_queue_exit(&ring2); + return T_EXIT_PASS; +} + +static int test_rq_setup(void *area) +{ + int ret; + struct io_uring_zcrx_area_reg area_reg = { + .addr = (__u64)(unsigned long)area, + .len = AREA_SZ, + .flags = 0, + }; + + struct io_uring_zcrx_ifq_reg reg = { + .if_idx = ifidx, + .if_rxq = rxq, + .rq_entries = 0, + .area_ptr = (__u64)(unsigned long)&area_reg, + }; + + ret = try_register_ifq(®); + if (ret != -EINVAL) { + fprintf(stderr, "registered 0 rq entries\n"); + return T_EXIT_FAIL; + } + + reg.rq_entries = (__u32)-1; + + ret = try_register_ifq(®); + if (ret != -EINVAL) { + fprintf(stderr, "registered unlimited nr of rq entries\n"); + return T_EXIT_FAIL; + } + + reg.rq_entries = RQ_ENTRIES - 1; + ret = try_register_ifq(®); + if (ret != 0) { + fprintf(stderr, "ifq registration failed %i\n", ret); + return T_EXIT_FAIL; + } + + if (reg.rq_entries == RQ_ENTRIES - 1) { + fprintf(stderr, "registered non pow2 refill entries %i\n", + reg.rq_entries); + return T_EXIT_FAIL; + } + + return T_EXIT_PASS; +} + +static int test_null_area_reg_struct(void) +{ + int ret; + + struct io_uring_zcrx_ifq_reg reg = { + .if_idx = ifidx, + .if_rxq = rxq, + .rq_entries = RQ_ENTRIES, + .area_ptr = (__u64)(unsigned long)0, + }; + + ret = try_register_ifq(®); + return ret ? T_EXIT_PASS : T_EXIT_FAIL; +} + +static int test_null_area(void) +{ + int ret; + + struct io_uring_zcrx_area_reg area_reg = { + .addr = (__u64)(unsigned long)0, + .len = AREA_SZ, + .flags = 0, + }; + + struct io_uring_zcrx_ifq_reg reg = { + .if_idx = ifidx, + .if_rxq = rxq, + .rq_entries = RQ_ENTRIES, + .area_ptr = (__u64)(unsigned long)&area_reg, + }; + + ret = try_register_ifq(®); + return ret ? T_EXIT_PASS : T_EXIT_FAIL; +} + +static int test_misaligned_area(void *area) +{ + int ret; + struct io_uring_zcrx_area_reg area_reg = { + .addr = (__u64)(unsigned long)(area + 1), + .len = AREA_SZ, + .flags = 0, + }; + + struct io_uring_zcrx_ifq_reg reg = { + .if_idx = ifidx, + .if_rxq = rxq, + .rq_entries = RQ_ENTRIES, + .area_ptr = (__u64)(unsigned long)&area_reg, + }; + + if (!try_register_ifq(®)) + return T_EXIT_FAIL; + + area_reg.addr = (__u64)(unsigned long)area; + area_reg.len = AREA_SZ - 1; + ret = try_register_ifq(®); + return ret ? T_EXIT_PASS : T_EXIT_FAIL; +} + +static int test_larger_than_alloc_area(void *area) +{ + int ret; + struct io_uring_zcrx_area_reg area_reg = { + .addr = (__u64)(unsigned long)area, + .len = AREA_SZ + 4096, + .flags = 0, + }; + + struct io_uring_zcrx_ifq_reg reg = { + .if_idx = ifidx, + .if_rxq = rxq, + .rq_entries = RQ_ENTRIES, + .area_ptr = (__u64)(unsigned long)&area_reg, + }; + + ret = try_register_ifq(®); + return ret ? T_EXIT_PASS : T_EXIT_FAIL; +} + +static int test_area_access(void) +{ + struct io_uring_zcrx_area_reg area_reg = { + .len = AREA_SZ, + .flags = 0, + }; + struct io_uring_zcrx_ifq_reg reg = { + .if_idx = ifidx, + .if_rxq = rxq, + .rq_entries = RQ_ENTRIES, + .area_ptr = (__u64)(unsigned long)&area_reg, + }; + int i, ret; + void *area; + + for (i = 0; i < 2; i++) { + int ro = i & 1; + int prot = ro ? PROT_READ : PROT_WRITE; + + area = mmap(NULL, AREA_SZ, prot, + MAP_ANONYMOUS | MAP_PRIVATE | MAP_FIXED, -1, 0); + if (area == MAP_FAILED) { + perror("mmap"); + return T_EXIT_FAIL; + } + + area_reg.addr = (__u64)(unsigned long)area; + + ret = try_register_ifq(®); + if (ret != -EFAULT) { + fprintf(stderr, "registered unaccessible memory\n"); + return T_EXIT_FAIL; + } + + munmap(area, AREA_SZ); + } + + return T_EXIT_PASS; +} + +static int create_ring_with_ifq(struct io_uring *ring, void *area) +{ + struct io_uring_zcrx_area_reg area_reg = { + .addr = (__u64)(unsigned long)area, + .len = AREA_SZ, + .flags = 0, + }; + struct io_uring_zcrx_ifq_reg reg = { + .if_idx = ifidx, + .if_rxq = rxq, + .rq_entries = RQ_ENTRIES, + .area_ptr = (__u64)(unsigned long)&area_reg, + }; + int ret; + + ret = t_create_ring(128, ring, RING_FLAGS); + if (ret != T_SETUP_OK) { + fprintf(stderr, "ring create failed: %d\n", ret); + return T_EXIT_FAIL; + } + + ret = io_uring_register_ifq(ring, ®); + if (ret) { + io_uring_queue_exit(ring); + fprintf(stderr, "ifq register failed %d\n", ret); + return T_EXIT_FAIL; + } + return 0; +} + +static void test_io_uring_prep_zcrx(struct io_uring_sqe *sqe, int fd, int ifq) +{ + io_uring_prep_rw(IORING_OP_RECV_ZC, sqe, fd, NULL, 0, 0); + sqe->zcrx_ifq_idx = ifq; + sqe->ioprio |= IORING_RECV_MULTISHOT; +} + +static struct io_uring_cqe *submit_and_wait_one(struct io_uring *ring) +{ + struct io_uring_cqe *cqe; + int ret; + + ret = io_uring_submit(ring); + if (ret != 1) { + fprintf(stderr, "sqe submit failed: %d\n", ret); + return NULL; + } + + ret = io_uring_wait_cqe(ring, &cqe); + if (ret < 0) { + fprintf(stderr, "wait completion %d\n", ret); + return NULL; + } + + return cqe; +} + +static int test_invalid_invalid_request(void *area) +{ + struct io_uring_cqe *cqe; + struct io_uring_sqe *sqe; + struct io_uring ring; + int ret, fds[2]; + + ret = create_ring_with_ifq(&ring, area); + if (ret != T_SETUP_OK) { + fprintf(stderr, "ifq-ring create failed: %d\n", ret); + return T_EXIT_FAIL; + } + + ret = t_create_socket_pair(fds, true); + if (ret) { + fprintf(stderr, "t_create_socket_pair failed: %d\n", ret); + return ret; + } + + /* invalid file */ + sqe = io_uring_get_sqe(&ring); + test_io_uring_prep_zcrx(sqe, ring.ring_fd, 0); + + cqe = submit_and_wait_one(&ring); + if (!cqe) { + fprintf(stderr, "submit_and_wait_one failed\n"); + return T_EXIT_FAIL; + } + if (cqe->flags & IORING_CQE_F_MORE) { + fprintf(stderr, "unexpected F_MORE for invalid fd\n"); + return T_EXIT_FAIL; + } + if (cqe->res != -ENOTSOCK) { + fprintf(stderr, "zcrx for non-socket file\n"); + return T_EXIT_FAIL; + } + io_uring_cqe_seen(&ring, cqe); + + /* invalid ifq idx */ + sqe = io_uring_get_sqe(&ring); + test_io_uring_prep_zcrx(sqe, fds[0], 1); + + cqe = submit_and_wait_one(&ring); + if (!cqe) { + fprintf(stderr, "submit_and_wait_one failed\n"); + return T_EXIT_FAIL; + } + if (cqe->flags & IORING_CQE_F_MORE) { + fprintf(stderr, "unexpected F_MORE for invalid fd\n"); + return T_EXIT_FAIL; + } + if (cqe->res != -EINVAL) { + fprintf(stderr, "zcrx recv with non-existent zcrx ifq\n"); + return T_EXIT_FAIL; + } + io_uring_cqe_seen(&ring, cqe); + + close(fds[0]); + close(fds[1]); + io_uring_queue_exit(&ring); + return 0; +} + +struct recv_data { + pthread_barrier_t connect; + pthread_barrier_t startup; + pthread_barrier_t barrier; + pthread_barrier_t finish; + + int accept_fd; + char buf[MSG_SIZE]; + void *area; + void *ring_ptr; + unsigned int ring_sz; + struct io_uring_zcrx_rq rq_ring; +}; + +static int recv_prep(struct io_uring *ring, struct recv_data *rd, int *sock) +{ + struct sockaddr_in saddr; + struct io_uring_sqe *sqe; + int sockfd, ret, val, use_fd; + socklen_t socklen; + + memset(&saddr, 0, sizeof(saddr)); + saddr.sin_family = AF_INET; + saddr.sin_addr.s_addr = htonl(INADDR_ANY); + saddr.sin_port = htons(PORT); + + sockfd = socket(AF_INET, SOCK_STREAM, 0); + if (sockfd < 0) { + perror("socket"); + return 1; + } + + val = 1; + setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &val, sizeof(val)); + + ret = bind(sockfd, (struct sockaddr *)&saddr, sizeof(saddr)); + if (ret < 0) { + perror("bind"); + goto err; + } + + ret = listen(sockfd, 1); + if (ret < 0) { + perror("listen"); + goto err; + } + + pthread_barrier_wait(&rd->connect); + + socklen = sizeof(saddr); + use_fd = accept(sockfd, (struct sockaddr *)&saddr, &socklen); + if (use_fd < 0) { + perror("accept"); + goto err; + } + + rd->accept_fd = use_fd; + pthread_barrier_wait(&rd->startup); + pthread_barrier_wait(&rd->barrier); + + sqe = io_uring_get_sqe(ring); + io_uring_prep_rw(IORING_OP_RECV_ZC, sqe, use_fd, NULL, 0, 0); + sqe->zcrx_ifq_idx = 0; + sqe->ioprio |= IORING_RECV_MULTISHOT; + sqe->user_data = 2; + + ret = io_uring_submit(ring); + if (ret <= 0) { + fprintf(stderr, "submit failed: %d\n", ret); + goto err; + } + + *sock = sockfd; + return 0; +err: + close(sockfd); + return 1; +} + +static struct io_uring_zcrx_rqe* get_refill_entry(struct io_uring_zcrx_rq *rq_ring) +{ + unsigned mask = rq_ring->ring_entries - 1; + struct io_uring_zcrx_rqe* rqe; + + rqe = &rq_ring->rqes[rq_ring->rq_tail & mask]; + rq_ring->rq_tail++; + return rqe; +} + +static void refill_garbage(struct recv_data *rd, uint64_t area_token) +{ + struct io_uring_zcrx_rq *rq_ring = &rd->rq_ring; + struct io_uring_zcrx_rqe* rqe; + int i = 0; + + /* invalid area */ + rqe = get_refill_entry(rq_ring); + rqe->off = (area_token + 1) << IORING_ZCRX_AREA_SHIFT; + i++; + + /* invalid area offset */ + rqe = get_refill_entry(rq_ring); + rqe->off = AREA_SZ | (area_token << IORING_ZCRX_AREA_SHIFT); + rqe->off += AREA_SZ; + i++; + + for (; i < rq_ring->ring_entries; i++) { + rqe = get_refill_entry(rq_ring); + rqe->off = ((uint64_t)1 << IORING_ZCRX_AREA_SHIFT) - 1; + } + + io_uring_smp_store_release(rq_ring->ktail, rq_ring->rq_tail); +} + +static int do_recv(struct io_uring *ring, struct recv_data *rd, + uint64_t refill_area_token) +{ + struct io_uring_cqe *cqe; + struct io_uring_zcrx_cqe *zcqe; + int i, ret; + + refill_garbage(rd, refill_area_token); + + for (i = 0; i < LOOP_COUNT - 1; i++) { + ret = io_uring_wait_cqe(ring, &cqe); + if (ret) { + fprintf(stdout, "wait_cqe: %d\n", ret); + return 1; + } + if (cqe->res == -EINVAL) { + fprintf(stdout, "recv not supported, skipping\n"); + goto out; + } + if (cqe->res < 0) { + fprintf(stderr, "failed recv cqe: %d\n", cqe->res); + goto err; + } + if (cqe->res - 1 != strlen(str)) { + fprintf(stderr, "got wrong length: %d/%d\n", cqe->res, + (int) strlen(str) + 1); + goto err; + } + + zcqe = (struct io_uring_zcrx_cqe *)(cqe + 1); + uint64_t mask = (1ULL << IORING_ZCRX_AREA_SHIFT) - 1; + uint64_t off = zcqe->off & mask; + void *addr = (char *)rd->area + off; + ret = strncmp(str, addr, sizeof(str)); + if (ret != 0) { + fprintf(stderr, "recv incorrect payload: %s\n", (const char *)addr); + goto err; + } + + io_uring_cqe_seen(ring, cqe); + } + + ret = io_uring_wait_cqe(ring, &cqe); + if (ret) { + fprintf(stdout, "wait_cqe: %d\n", ret); + return 1; + } + if (cqe->res != -ENOMEM) { + fprintf(stdout, "final recv cqe did not return ENOMEM\n"); + goto err; + } + +out: + io_uring_cqe_seen(ring, cqe); + pthread_barrier_wait(&rd->finish); + return 0; +err: + io_uring_cqe_seen(ring, cqe); + pthread_barrier_wait(&rd->finish); + return 1; +} + +static void *recv_fn(void *data) +{ + struct recv_data *rd = data; + struct io_uring_params p = { }; + struct io_uring ring; + int ret, sock; + + p.flags = RING_FLAGS; + ret = t_create_ring_params(8, &ring, &p); + if (ret == T_SETUP_SKIP) { + ret = 0; + goto err; + } else if (ret < 0) { + goto err; + } + + struct io_uring_zcrx_area_reg area_reg = { + .addr = (__u64)(unsigned long)rd->area, + .len = AREA_SZ, + .flags = 0, + }; + + struct io_uring_zcrx_ifq_reg reg = { + .if_idx = ifidx, + .if_rxq = rxq, + .rq_entries = RQ_ENTRIES, + .area_ptr = (__u64)(unsigned long)&area_reg, + }; + + ret = io_uring_register_ifq(&ring, ®); + if (ret != 0) { + fprintf(stderr, "register_ifq failed: %d\n", ret); + goto err_ring_exit; + } + + /* + rd->ring_ptr = mmap( + 0, + reg.offsets.mmap_sz, + PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_POPULATE, + ring.enter_ring_fd, + IORING_OFF_RQ_RING + ); + + rd->ring_sz = reg.offsets.mmap_sz; + */ + rd->rq_ring.khead = (__u32*)((char*)rd->ring_ptr + reg.offsets.head); + rd->rq_ring.ktail = (__u32*)((char*)rd->ring_ptr + reg.offsets.tail); + rd->rq_ring.rqes = (struct io_uring_zcrx_rqe*)((char*)rd->ring_ptr + reg.offsets.rqes); + rd->rq_ring.rq_tail = 0; + rd->rq_ring.ring_entries = reg.rq_entries; + + ret = recv_prep(&ring, rd, &sock); + if (ret) { + fprintf(stderr, "recv_prep failed: %d\n", ret); + goto err; + } + ret = do_recv(&ring, rd, area_reg.rq_area_token); + + close(sock); + close(rd->accept_fd); +err_ring_exit: + io_uring_queue_exit(&ring); +err: + return (void *)(intptr_t)ret; +} + +static int do_send(struct recv_data *rd) +{ + struct sockaddr_in saddr; + struct iovec iov = { + .iov_base = str, + .iov_len = sizeof(str), + }; + struct io_uring ring; + struct io_uring_cqe *cqe; + struct io_uring_sqe *sqe; + int i, sockfd, ret; + + ret = io_uring_queue_init(8, &ring, 0); + if (ret) { + fprintf(stderr, "queue init failed: %d\n", ret); + return 1; + } + + memset(&saddr, 0, sizeof(saddr)); + saddr.sin_family = AF_INET; + saddr.sin_port = htons(PORT); + inet_pton(AF_INET, HOST, &saddr.sin_addr); + + sockfd = socket(AF_INET, SOCK_STREAM, 0); + if (sockfd < 0) { + perror("socket"); + goto err2; + } + + pthread_barrier_wait(&rd->connect); + + ret = connect(sockfd, (struct sockaddr *)&saddr, sizeof(saddr)); + if (ret < 0) { + perror("connect"); + goto err; + } + + pthread_barrier_wait(&rd->startup); + + for (i = 0; i < LOOP_COUNT; i++) { + sqe = io_uring_get_sqe(&ring); + io_uring_prep_send(sqe, sockfd, iov.iov_base, iov.iov_len, 0); + sqe->user_data = 1; + } + + ret = io_uring_submit(&ring); + if (ret <= 0) { + fprintf(stderr, "submit failed: %d\n", ret); + goto err; + } + + pthread_barrier_wait(&rd->barrier); + + ret = io_uring_wait_cqe(&ring, &cqe); + if (cqe->res == -EINVAL) { + fprintf(stdout, "send not supported, skipping\n"); + goto err; + } + if (cqe->res != iov.iov_len) { + fprintf(stderr, "failed cqe: %d\n", cqe->res); + goto err; + } + + pthread_barrier_wait(&rd->finish); + + close(sockfd); + io_uring_queue_exit(&ring); + return 0; + +err: + close(sockfd); +err2: + io_uring_queue_exit(&ring); + pthread_barrier_wait(&rd->finish); + return 1; +} + +static int test_recv(void *area) +{ + pthread_t recv_thread; + struct recv_data rd; + int ret; + void *retval; + + memset(&rd, 0, sizeof(rd)); + pthread_barrier_init(&rd.connect, NULL, 2); + pthread_barrier_init(&rd.startup, NULL, 2); + pthread_barrier_init(&rd.barrier, NULL, 2); + pthread_barrier_init(&rd.finish, NULL, 2); + rd.area = area; + + ret = pthread_create(&recv_thread, NULL, recv_fn, &rd); + if (ret) { + fprintf(stderr, "Thread create failed: %d\n", ret); + return 1; + } + + do_send(&rd); + pthread_join(recv_thread, &retval); + return (intptr_t)retval; +} + +int main(int argc, char *argv[]) +{ + char *dev, *rxq_str, *rxq_end; + void *area_outer, *area; + int ret; + + if (argc > 1) + return 0; + + area_outer = mmap(NULL, AREA_SZ + 8192, PROT_NONE, + MAP_ANONYMOUS | MAP_PRIVATE | MAP_NORESERVE, -1, 0); + if (area_outer == MAP_FAILED) { + perror("mmap"); + return T_EXIT_FAIL; + } + + area = mmap(area_outer, AREA_SZ, PROT_READ | PROT_WRITE, + MAP_ANONYMOUS | MAP_PRIVATE | MAP_FIXED, -1, 0); + if (area == MAP_FAILED) { + perror("mmap"); + return T_EXIT_FAIL; + } + + dev = getenv(DEV_ENV_VAR); + if (!dev) + return T_EXIT_SKIP; + + ifidx = if_nametoindex(dev); + if (!ifidx) + return T_EXIT_SKIP; + + rxq_str = getenv(RXQ_ENV_VAR); + if (!rxq_str) + return T_EXIT_SKIP; + + rxq = strtol(rxq_str, &rxq_end, 10); + if (rxq_end == rxq_str || *rxq_end != '\0') + return T_EXIT_SKIP; + + ret = probe_zcrx(area); + if (ret != T_EXIT_PASS) + return ret; + + ret = test_rq_setup(area); + if (ret) { + fprintf(stderr, "test_invalid_reg_struct failed\n"); + return ret; + } + + ret = test_null_area_reg_struct(); + if (ret) { + fprintf(stderr, "test_null_area_reg_struct failed\n"); + return ret; + } + + ret = test_null_area(); + if (ret) { + fprintf(stderr, "test_null_area failed\n"); + return ret; + } + + ret = test_misaligned_area(area); + if (ret) { + fprintf(stderr, "test_misaligned_area failed\n"); + return ret; + } + + ret = test_larger_than_alloc_area(area); + if (ret) { + fprintf(stderr, "test_larger_than_alloc_area failed\n"); + return ret; + } + + ret = test_area_access(); + if (ret) { + fprintf(stderr, "test_area_access failed\n"); + return ret; + } + + ret = test_invalid_if(area); + if (ret) { + fprintf(stderr, "test_invalid_if failed\n"); + return ret; + } + + ret = test_invalid_ifq_collision(area); + if (ret) { + fprintf(stderr, "test_invalid_ifq_collision failed\n"); + return ret; + } + + ret = test_invalid_invalid_request(area); + if (ret) { + fprintf(stderr, "test_invalid_ifq_collision failed\n"); + return ret; + } + + ret = test_recv(area); + if (ret) { + fprintf(stderr, "test_recv failed\n"); + return ret; + } + + munmap(area, AREA_SZ); + return 0; +}