From patchwork Fri Jun 14 06:39:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13697902 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-112.freemail.mail.aliyun.com (out30-112.freemail.mail.aliyun.com [115.124.30.112]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 708401482E7; Fri, 14 Jun 2024 06:39:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.112 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718347181; cv=none; b=bO+1x1bUcBPPla2JZd4ZoC0RrfPjw+Glc9T+qI0jLavtG0cU9T8WQSH+NUQgtHCcFKNH4skfOjKGkyMGz+uqlDV5RhiHPSPtYwFyGN3vSr4kKJAy8yQ99tkjgy3wKzvR0VEYMtYRaLv8WimFIYxMTu13X2ntvGlO3QvM3awGxXM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718347181; c=relaxed/simple; bh=qssruXUXONS3tsc1QH9lqllrnqAJvB/oa7NZvPwxBrA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=mm48kde2WDRmwi/jl3uVomsQpZWyk9R00pNioY7A4K0dEhzmrP/mHFlweRRG4HkE3YOMlGW/t8FD9UlSpZb/fRcNFfijkgaAxIuL9EWf56EfZ1FnLtRanpJN+qQ0RX7K55S/ggBdJq/kBsNS9KVI5clKfBn/8CKLCHhUytAySOw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=dmudE51H; arc=none smtp.client-ip=115.124.30.112 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="dmudE51H" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1718347176; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=Zx8xhm7YYWjC4UDKc1EygC2VUHUPZNPB/q3JyfYPdoU=; b=dmudE51HNYlSk4tM4VKIaclQaQ9LYFSDQ5sZCNKeaqffpLT3HIZwnQw9JrATw/eics3/+F2HnnYzEAVbuUaa6S97+CpSgShbKoxSHk9axGvmsJNRQjOpybODhK0NGsohECelRrfUlzpJFusdHBV32NIWkMiYW2AwsaQINGf0A/I= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R121e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033037067113;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=15;SR=0;TI=SMTPD_---0W8Q9u7h_1718347174; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W8Q9u7h_1718347174) by smtp.aliyun-inc.com; Fri, 14 Jun 2024 14:39:35 +0800 From: Xuan Zhuo To: netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , =?utf-8?q?Eugenio_P=C3=A9rez?= , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , virtualization@lists.linux.dev, bpf@vger.kernel.org Subject: [PATCH net-next v5 01/15] virtio_ring: introduce dma map api for page Date: Fri, 14 Jun 2024 14:39:19 +0800 Message-Id: <20240614063933.108811-2-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240614063933.108811-1-xuanzhuo@linux.alibaba.com> References: <20240614063933.108811-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: e008fb4a0943 X-Patchwork-Delegate: kuba@kernel.org The virtio-net sq will use these APIs to map the scatterlist. For scatterlist, the page dma APIs are more appropriate. dma_addr_t virtqueue_dma_map_page_attrs(struct virtqueue *_vq, struct page *page, size_t offset, size_t size, enum dma_data_direction dir, unsigned long attrs); void virtqueue_dma_unmap_page_attrs(struct virtqueue *_vq, dma_addr_t addr, size_t size, enum dma_data_direction dir, unsigned long attrs); Signed-off-by: Xuan Zhuo --- drivers/virtio/virtio_ring.c | 54 ++++++++++++++++++++++++++++++++++++ include/linux/virtio.h | 7 +++++ 2 files changed, 61 insertions(+) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 2a972752ff1b..e8caa6f24704 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -3152,6 +3152,60 @@ void virtqueue_dma_unmap_single_attrs(struct virtqueue *_vq, dma_addr_t addr, } EXPORT_SYMBOL_GPL(virtqueue_dma_unmap_single_attrs); +/** + * virtqueue_dma_map_page_attrs - map DMA for _vq + * @_vq: the struct virtqueue we're talking about. + * @page: the page to do dma + * @offset: the offset inside the page + * @size: the size of the page to do dma + * @dir: DMA direction + * @attrs: DMA Attrs + * + * The caller calls this to do dma mapping in advance. The DMA address can be + * passed to this _vq when it is in pre-mapped mode. + * + * Return: DMA address. Caller should check that by virtqueue_dma_mapping_error(). + */ +dma_addr_t virtqueue_dma_map_page_attrs(struct virtqueue *_vq, struct page *page, + size_t offset, size_t size, + enum dma_data_direction dir, + unsigned long attrs) +{ + struct vring_virtqueue *vq = to_vvq(_vq); + + if (!vq->use_dma_api) { + kmsan_handle_dma(page, offset, size, dir); + return page_to_phys(page) + offset; + } + + return dma_map_page_attrs(vring_dma_dev(vq), page, offset, size, dir, attrs); +} +EXPORT_SYMBOL_GPL(virtqueue_dma_map_page_attrs); + +/** + * virtqueue_dma_unmap_page_attrs - unmap DMA for _vq + * @_vq: the struct virtqueue we're talking about. + * @addr: the dma address to unmap + * @size: the size of the buffer + * @dir: DMA direction + * @attrs: DMA Attrs + * + * Unmap the address that is mapped by the virtqueue_dma_map_* APIs. + * + */ +void virtqueue_dma_unmap_page_attrs(struct virtqueue *_vq, dma_addr_t addr, + size_t size, enum dma_data_direction dir, + unsigned long attrs) +{ + struct vring_virtqueue *vq = to_vvq(_vq); + + if (!vq->use_dma_api) + return; + + dma_unmap_page_attrs(vring_dma_dev(vq), addr, size, dir, attrs); +} +EXPORT_SYMBOL_GPL(virtqueue_dma_unmap_page_attrs); + /** * virtqueue_dma_mapping_error - check dma address * @_vq: the struct virtqueue we're talking about. diff --git a/include/linux/virtio.h b/include/linux/virtio.h index 96fea920873b..ca318a66a7e1 100644 --- a/include/linux/virtio.h +++ b/include/linux/virtio.h @@ -234,6 +234,13 @@ dma_addr_t virtqueue_dma_map_single_attrs(struct virtqueue *_vq, void *ptr, size void virtqueue_dma_unmap_single_attrs(struct virtqueue *_vq, dma_addr_t addr, size_t size, enum dma_data_direction dir, unsigned long attrs); +dma_addr_t virtqueue_dma_map_page_attrs(struct virtqueue *_vq, struct page *page, + size_t offset, size_t size, + enum dma_data_direction dir, + unsigned long attrs); +void virtqueue_dma_unmap_page_attrs(struct virtqueue *_vq, dma_addr_t addr, + size_t size, enum dma_data_direction dir, + unsigned long attrs); int virtqueue_dma_mapping_error(struct virtqueue *_vq, dma_addr_t addr); bool virtqueue_dma_need_sync(struct virtqueue *_vq, dma_addr_t addr); From patchwork Fri Jun 14 06:39:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13697907 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-118.freemail.mail.aliyun.com (out30-118.freemail.mail.aliyun.com [115.124.30.118]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AFCD21487DA; Fri, 14 Jun 2024 06:39:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.118 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718347187; cv=none; b=ECRUcBDTdYgVvONtRcyY/5emwugUSsx++9o6aKJsaDeP9hWAjEDh3sNiQVsqCsExTdPHgpXNl4Vf+UXjhsL1/Qnq8ByvJr9x+svh/C3K9OPmrVv3MH0ce/7VZOX+pyW4EqmeyGtCowtytRM0bQ61MAG7+a7b5La20NnmGAuMOYg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718347187; c=relaxed/simple; bh=azwKvLrDj+U1MX9tcdt5NVQKEQx9FFum6cYUX0Ap8Ok=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=mSYPihPdkDZ31Ww/8LFMCWVGSuwpY+2/+HmCxHqB45LFHJ88lfEMiuYqgx96XW3og06AkcAaxi2GnKYrEgb5V+Pjaq9cH+cHdydQoZnk7YBagb9CDVD60kpD21oRFgssz6SLFMRSe2dyairDB2Xd1lFw1n6W/2TaxEzH4T2zXnE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=lsfEo7KU; arc=none smtp.client-ip=115.124.30.118 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="lsfEo7KU" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1718347176; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=wz/gNoMpTBu5PmKOqW/HPAECwFWfSfMba3ie/30qSS4=; b=lsfEo7KU1zKNA1s7f1PFna5iYsbWxI8EUcUpK+TWE8i/oOf+6abz6WwStF+2payFkyu2PAPA127UuyCzBwDrsDSVsnwN5AYPFxBfR8xRBKw21yeg16cK/RTY4LZrcrb/8q4RghswuImFDM9KramgcM7htLFb8+AAh+YK7D+VT7c= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R941e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033037067111;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=15;SR=0;TI=SMTPD_---0W8QEk8Y_1718347175; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W8QEk8Y_1718347175) by smtp.aliyun-inc.com; Fri, 14 Jun 2024 14:39:36 +0800 From: Xuan Zhuo To: netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , =?utf-8?q?Eugenio_P=C3=A9rez?= , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , virtualization@lists.linux.dev, bpf@vger.kernel.org Subject: [PATCH net-next v5 02/15] virtio_ring: introduce vring_need_unmap_buffer Date: Fri, 14 Jun 2024 14:39:20 +0800 Message-Id: <20240614063933.108811-3-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240614063933.108811-1-xuanzhuo@linux.alibaba.com> References: <20240614063933.108811-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: e008fb4a0943 X-Patchwork-Delegate: kuba@kernel.org To make the code readable, introduce vring_need_unmap_buffer() to replace do_unmap. use_dma_api premapped -> vring_need_unmap_buffer() 1. false false false 2. true false true 3. true true false Signed-off-by: Xuan Zhuo Acked-by: Jason Wang --- drivers/virtio/virtio_ring.c | 27 ++++++++++++--------------- 1 file changed, 12 insertions(+), 15 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index e8caa6f24704..d0d3004a408a 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -175,11 +175,6 @@ struct vring_virtqueue { /* Do DMA mapping by driver */ bool premapped; - /* Do unmap or not for desc. Just when premapped is False and - * use_dma_api is true, this is true. - */ - bool do_unmap; - /* Head of free buffer list. */ unsigned int free_head; /* Number we've added since last sync. */ @@ -297,6 +292,11 @@ static bool vring_use_dma_api(const struct virtio_device *vdev) return false; } +static bool vring_need_unmap_buffer(const struct vring_virtqueue *vring) +{ + return vring->use_dma_api && !vring->premapped; +} + size_t virtio_max_dma_size(const struct virtio_device *vdev) { size_t max_segment_size = SIZE_MAX; @@ -445,7 +445,7 @@ static void vring_unmap_one_split_indirect(const struct vring_virtqueue *vq, { u16 flags; - if (!vq->do_unmap) + if (!vring_need_unmap_buffer(vq)) return; flags = virtio16_to_cpu(vq->vq.vdev, desc->flags); @@ -475,7 +475,7 @@ static unsigned int vring_unmap_one_split(const struct vring_virtqueue *vq, (flags & VRING_DESC_F_WRITE) ? DMA_FROM_DEVICE : DMA_TO_DEVICE); } else { - if (!vq->do_unmap) + if (!vring_need_unmap_buffer(vq)) goto out; dma_unmap_page(vring_dma_dev(vq), @@ -643,7 +643,7 @@ static inline int virtqueue_add_split(struct virtqueue *_vq, } /* Last one doesn't continue. */ desc[prev].flags &= cpu_to_virtio16(_vq->vdev, ~VRING_DESC_F_NEXT); - if (!indirect && vq->do_unmap) + if (!indirect && vring_need_unmap_buffer(vq)) vq->split.desc_extra[prev & (vq->split.vring.num - 1)].flags &= ~VRING_DESC_F_NEXT; @@ -802,7 +802,7 @@ static void detach_buf_split(struct vring_virtqueue *vq, unsigned int head, VRING_DESC_F_INDIRECT)); BUG_ON(len == 0 || len % sizeof(struct vring_desc)); - if (vq->do_unmap) { + if (vring_need_unmap_buffer(vq)) { for (j = 0; j < len / sizeof(struct vring_desc); j++) vring_unmap_one_split_indirect(vq, &indir_desc[j]); } @@ -1236,7 +1236,7 @@ static void vring_unmap_extra_packed(const struct vring_virtqueue *vq, (flags & VRING_DESC_F_WRITE) ? DMA_FROM_DEVICE : DMA_TO_DEVICE); } else { - if (!vq->do_unmap) + if (!vring_need_unmap_buffer(vq)) return; dma_unmap_page(vring_dma_dev(vq), @@ -1251,7 +1251,7 @@ static void vring_unmap_desc_packed(const struct vring_virtqueue *vq, { u16 flags; - if (!vq->do_unmap) + if (!vring_need_unmap_buffer(vq)) return; flags = le16_to_cpu(desc->flags); @@ -1632,7 +1632,7 @@ static void detach_buf_packed(struct vring_virtqueue *vq, if (!desc) return; - if (vq->do_unmap) { + if (vring_need_unmap_buffer(vq)) { len = vq->packed.desc_extra[id].len; for (i = 0; i < len / sizeof(struct vring_packed_desc); i++) @@ -2091,7 +2091,6 @@ static struct virtqueue *vring_create_virtqueue_packed( vq->dma_dev = dma_dev; vq->use_dma_api = vring_use_dma_api(vdev); vq->premapped = false; - vq->do_unmap = vq->use_dma_api; vq->indirect = virtio_has_feature(vdev, VIRTIO_RING_F_INDIRECT_DESC) && !context; @@ -2636,7 +2635,6 @@ static struct virtqueue *__vring_new_virtqueue(unsigned int index, vq->dma_dev = dma_dev; vq->use_dma_api = vring_use_dma_api(vdev); vq->premapped = false; - vq->do_unmap = vq->use_dma_api; vq->indirect = virtio_has_feature(vdev, VIRTIO_RING_F_INDIRECT_DESC) && !context; @@ -2799,7 +2797,6 @@ int virtqueue_set_dma_premapped(struct virtqueue *_vq) } vq->premapped = true; - vq->do_unmap = false; END_USE(vq); From patchwork Fri Jun 14 06:39:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13697909 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-97.freemail.mail.aliyun.com (out30-97.freemail.mail.aliyun.com [115.124.30.97]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A8FD51487E7; Fri, 14 Jun 2024 06:39:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.97 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718347187; cv=none; b=nm55GA/iP9wAW02DBasxxxWeKOIsv2UQ381yLNS34dhJ22aFycgcFd+788qNc3wJK6jEGZfRJc5iW81DB7iRq/GR4B67gd5mGNgGbeKkDtDElAVUpx05J29M5IzKISAFGUTXgpuw9OSzQktkwvqnY9Pw8Fu9UkDbz9D/PGP1+UI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718347187; c=relaxed/simple; bh=tcxopaJoD+u815bFN/f9f0HYtcCX8rUPp/jcsEYNy3E=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Mts2awH53MRs44TVRJlnxeM+7TTKIYJ5e6E1HJ0GQSVlQPgDs8+U+9SiYqw/q7p2zOnnsjmKI2LVFSFDkQOAvBrRajYUwCoFz1O1NU7fXjvKwBbZb6ZDVErirhu4FSwJVDdgEfk8axlblN6jQy5Hs67XqoYrr8q0tEBgmPSkOYs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=NASL+KsT; arc=none smtp.client-ip=115.124.30.97 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="NASL+KsT" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1718347177; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=TukRhtN8QeiEObGWmfADeZDtr0qZu2kwqlq7Wk+V02Y=; b=NASL+KsToaV+3ibjQmDWusJZlxMe7kFbekIghkSzC11dQqSwxdB/7J412vRGnSlpb5Bl6nBTVzHch2QsE3x9QjjhJKA7HmFrRjBcQ5/DpZ6DWo/uf3gcEOaJTPRP6S8PRg70GMo8ECOUd3/PXlF19DMm2tTVlzgyaHJISyPzWtk= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R141e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033032014031;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=15;SR=0;TI=SMTPD_---0W8QEk9._1718347176; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W8QEk9._1718347176) by smtp.aliyun-inc.com; Fri, 14 Jun 2024 14:39:37 +0800 From: Xuan Zhuo To: netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , =?utf-8?q?Eugenio_P=C3=A9rez?= , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , virtualization@lists.linux.dev, bpf@vger.kernel.org Subject: [PATCH net-next v5 03/15] virtio_ring: virtqueue_set_dma_premapped() support to disable Date: Fri, 14 Jun 2024 14:39:21 +0800 Message-Id: <20240614063933.108811-4-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240614063933.108811-1-xuanzhuo@linux.alibaba.com> References: <20240614063933.108811-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: e008fb4a0943 X-Patchwork-Delegate: kuba@kernel.org virtio-net sq will only enable premapped mode when sq is bound to the af-xdp. So we need the helper (virtqueue_set_dma_premapped) to enable the premapped mode when af-xdp binds to sq. And to disable the premapped mode when af-xdp unbinds to sq. Signed-off-by: Xuan Zhuo --- drivers/net/virtio_net.c | 2 +- drivers/virtio/virtio_ring.c | 7 ++++--- include/linux/virtio.h | 2 +- 3 files changed, 6 insertions(+), 5 deletions(-) diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index 61a57d134544..838b450d9591 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -938,7 +938,7 @@ static void virtnet_rq_set_premapped(struct virtnet_info *vi) for (i = 0; i < vi->max_queue_pairs; i++) /* error should never happen */ - BUG_ON(virtqueue_set_dma_premapped(vi->rq[i].vq)); + BUG_ON(virtqueue_set_dma_premapped(vi->rq[i].vq, true)); } static void virtnet_rq_unmap_free_buf(struct virtqueue *vq, void *buf) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index d0d3004a408a..12083a0e6052 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -2764,8 +2764,9 @@ EXPORT_SYMBOL_GPL(virtqueue_resize); /** * virtqueue_set_dma_premapped - set the vring premapped mode * @_vq: the struct virtqueue we're talking about. + * @premapped: bool enable/disable the premapped mode * - * Enable the premapped mode of the vq. + * Enable/disable the premapped mode of the vq. * * The vring in premapped mode does not do dma internally, so the driver must * do dma mapping in advance. The driver must pass the dma_address through @@ -2782,7 +2783,7 @@ EXPORT_SYMBOL_GPL(virtqueue_resize); * 0: success. * -EINVAL: too late to enable premapped mode, the vq already contains buffers. */ -int virtqueue_set_dma_premapped(struct virtqueue *_vq) +int virtqueue_set_dma_premapped(struct virtqueue *_vq, bool premapped) { struct vring_virtqueue *vq = to_vvq(_vq); u32 num; @@ -2796,7 +2797,7 @@ int virtqueue_set_dma_premapped(struct virtqueue *_vq) return -EINVAL; } - vq->premapped = true; + vq->premapped = premapped; END_USE(vq); diff --git a/include/linux/virtio.h b/include/linux/virtio.h index ca318a66a7e1..69677d02cee9 100644 --- a/include/linux/virtio.h +++ b/include/linux/virtio.h @@ -81,7 +81,7 @@ bool virtqueue_enable_cb(struct virtqueue *vq); unsigned virtqueue_enable_cb_prepare(struct virtqueue *vq); -int virtqueue_set_dma_premapped(struct virtqueue *_vq); +int virtqueue_set_dma_premapped(struct virtqueue *_vq, bool premapped); bool virtqueue_poll(struct virtqueue *vq, unsigned); From patchwork Fri Jun 14 06:39:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13697903 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-101.freemail.mail.aliyun.com (out30-101.freemail.mail.aliyun.com [115.124.30.101]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F27751487E7; Fri, 14 Jun 2024 06:39:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.101 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718347184; cv=none; b=ibgy1jdk6C3e8+Gr8kPZ65jhyRJr5iiHqVgXh1cjKgsQ9nhpWm711t/A7Ip3+5HCSCrtuUowF8KkDCnqmDcqn9ljAtZi6GG+UPZyc4EbStAmBVFDKpz7sMAvuTerfw9mROA23VfvtalrlJxwe7i7kaeu03uY106Vtdk6zeHpDHI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718347184; c=relaxed/simple; bh=dpvycL/PK0HeYJyn6r9/RWK557m02BKf3Eccd6x4xLw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=MMCMrMFNjzde4OSYbK5UJn9JWIltFcMkthOzkSn0yWaVogUB9y+yLsUV/Zj2nhgfeJ/ONwVIs77krQ5Gc7td8eoDJxlpxS9YSECr8oy6dxKpnkp6RbpL0AGE/CIWsKkaHmT0LZUwYEKriv5GleSJ3cLBUeNgZ1te8KWHS18aKrA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=lMtdhdsI; arc=none smtp.client-ip=115.124.30.101 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="lMtdhdsI" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1718347179; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=+bZgsSRKnPDty/2zTa+8CikoQZlg9RVPyzoPeztif5k=; b=lMtdhdsIOibC/LGsERg2aqVJljXSZFxHGRwITTr8pgZ60AkGzXbq6kh/veUOgU9x7V9dtNAVNe/eA4TugSeYSJ3a6jZ32CVhgU37cdPUeV4IN4WIGHTXlvtdx/QCWjvHajFmoNCBJ9w+Dk8iFUDjO299HDyNyRoQ1YitU3nEWVM= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R461e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033032014031;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=15;SR=0;TI=SMTPD_---0W8Q97vb_1718347177; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W8Q97vb_1718347177) by smtp.aliyun-inc.com; Fri, 14 Jun 2024 14:39:38 +0800 From: Xuan Zhuo To: netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , =?utf-8?q?Eugenio_P=C3=A9rez?= , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , virtualization@lists.linux.dev, bpf@vger.kernel.org Subject: [PATCH net-next v5 04/15] virtio_net: separate virtnet_rx_resize() Date: Fri, 14 Jun 2024 14:39:22 +0800 Message-Id: <20240614063933.108811-5-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240614063933.108811-1-xuanzhuo@linux.alibaba.com> References: <20240614063933.108811-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: e008fb4a0943 X-Patchwork-Delegate: kuba@kernel.org This patch separates two sub-functions from virtnet_rx_resize(): * virtnet_rx_pause * virtnet_rx_resume Then the subsequent reset rx for xsk can share these two functions. Signed-off-by: Xuan Zhuo Acked-by: Jason Wang --- drivers/net/virtio_net.c | 29 +++++++++++++++++++++-------- 1 file changed, 21 insertions(+), 8 deletions(-) diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index 838b450d9591..597d2c5fccc0 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -2609,28 +2609,41 @@ static netdev_tx_t start_xmit(struct sk_buff *skb, struct net_device *dev) return NETDEV_TX_OK; } -static int virtnet_rx_resize(struct virtnet_info *vi, - struct receive_queue *rq, u32 ring_num) +static void virtnet_rx_pause(struct virtnet_info *vi, struct receive_queue *rq) { bool running = netif_running(vi->dev); - int err, qindex; - - qindex = rq - vi->rq; if (running) { napi_disable(&rq->napi); cancel_work_sync(&rq->dim.work); } +} - err = virtqueue_resize(rq->vq, ring_num, virtnet_rq_unmap_free_buf); - if (err) - netdev_err(vi->dev, "resize rx fail: rx queue index: %d err: %d\n", qindex, err); +static void virtnet_rx_resume(struct virtnet_info *vi, struct receive_queue *rq) +{ + bool running = netif_running(vi->dev); if (!try_fill_recv(vi, rq, GFP_KERNEL)) schedule_delayed_work(&vi->refill, 0); if (running) virtnet_napi_enable(rq->vq, &rq->napi); +} + +static int virtnet_rx_resize(struct virtnet_info *vi, + struct receive_queue *rq, u32 ring_num) +{ + int err, qindex; + + qindex = rq - vi->rq; + + virtnet_rx_pause(vi, rq); + + err = virtqueue_resize(rq->vq, ring_num, virtnet_rq_unmap_free_buf); + if (err) + netdev_err(vi->dev, "resize rx fail: rx queue index: %d err: %d\n", qindex, err); + + virtnet_rx_resume(vi, rq); return err; } From patchwork Fri Jun 14 06:39:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13697912 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-100.freemail.mail.aliyun.com (out30-100.freemail.mail.aliyun.com [115.124.30.100]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C210E158D87; Fri, 14 Jun 2024 06:39:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.100 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718347190; cv=none; b=SKnrkNGBINQWZ5IoDk4AMeYOoH2WNyhqEhUG3JwIvSsPA+3chpIg1yBIiWnjog4RfKgiSZn7wSjJxlQagdVasPKlUj+h3fPVpgDFzB2RxAextbWYBgMej36KZbtUUICCKXfTnyTuBcxygNEAE9MaufMHcoE9qQo/JT59OmMdn5Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718347190; c=relaxed/simple; bh=rQS7mnZgNYZTs8wY1Dwfl6b9ZybCf3uIAEXsRUTv4F4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=j3HL703mkZR6ORcgWHGqrfHxvIbWdTdKonKk5wVqGCOoDtYP3yEKiP65hscK+dEztbV4L9SSbqAhRcAfvAnLmrUYAKP9VH7Xh7DYi/nJtwI2qRQVEWtrdO8XnKLKQDM5ZF4Hnp0AhYV219TSiXZUC/qbd9gNL71RhxwCzv7w+R4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=I5hJBMgD; arc=none smtp.client-ip=115.124.30.100 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="I5hJBMgD" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1718347180; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=a5sz/5qr62F8OtqoIw0iDhCuO+4o5p7+tUqebg6SBto=; b=I5hJBMgD+3RnimPLszVpxYGBH6tpWMqcSdWsxUDsfMWGOYdQ3OM1xnWzy6fshPma1PajVgkPIPYCMsTBKiTE99egW4Yh57sUgMCxgMMnJXpuzyRRQRLLCq85PB+TPFckoxSPQoH6BjtjDZNRX7P1L1/JplscqiAfw5RtdXgVF84= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R171e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033045046011;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=15;SR=0;TI=SMTPD_---0W8QF6RP_1718347178; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W8QF6RP_1718347178) by smtp.aliyun-inc.com; Fri, 14 Jun 2024 14:39:39 +0800 From: Xuan Zhuo To: netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , =?utf-8?q?Eugenio_P=C3=A9rez?= , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , virtualization@lists.linux.dev, bpf@vger.kernel.org Subject: [PATCH net-next v5 05/15] virtio_net: separate virtnet_tx_resize() Date: Fri, 14 Jun 2024 14:39:23 +0800 Message-Id: <20240614063933.108811-6-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240614063933.108811-1-xuanzhuo@linux.alibaba.com> References: <20240614063933.108811-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: e008fb4a0943 X-Patchwork-Delegate: kuba@kernel.org This patch separates two sub-functions from virtnet_tx_resize(): * virtnet_tx_pause * virtnet_tx_resume Then the subsequent virtnet_tx_reset() can share these two functions. Signed-off-by: Xuan Zhuo Acked-by: Jason Wang --- drivers/net/virtio_net.c | 35 +++++++++++++++++++++++++++++------ 1 file changed, 29 insertions(+), 6 deletions(-) diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index 597d2c5fccc0..8c51947abce9 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -2647,12 +2647,11 @@ static int virtnet_rx_resize(struct virtnet_info *vi, return err; } -static int virtnet_tx_resize(struct virtnet_info *vi, - struct send_queue *sq, u32 ring_num) +static void virtnet_tx_pause(struct virtnet_info *vi, struct send_queue *sq) { bool running = netif_running(vi->dev); struct netdev_queue *txq; - int err, qindex; + int qindex; qindex = sq - vi->sq; @@ -2673,10 +2672,17 @@ static int virtnet_tx_resize(struct virtnet_info *vi, netif_stop_subqueue(vi->dev, qindex); __netif_tx_unlock_bh(txq); +} - err = virtqueue_resize(sq->vq, ring_num, virtnet_sq_free_unused_buf); - if (err) - netdev_err(vi->dev, "resize tx fail: tx queue index: %d err: %d\n", qindex, err); +static void virtnet_tx_resume(struct virtnet_info *vi, struct send_queue *sq) +{ + bool running = netif_running(vi->dev); + struct netdev_queue *txq; + int qindex; + + qindex = sq - vi->sq; + + txq = netdev_get_tx_queue(vi->dev, qindex); __netif_tx_lock_bh(txq); sq->reset = false; @@ -2685,6 +2691,23 @@ static int virtnet_tx_resize(struct virtnet_info *vi, if (running) virtnet_napi_tx_enable(vi, sq->vq, &sq->napi); +} + +static int virtnet_tx_resize(struct virtnet_info *vi, struct send_queue *sq, + u32 ring_num) +{ + int qindex, err; + + qindex = sq - vi->sq; + + virtnet_tx_pause(vi, sq); + + err = virtqueue_resize(sq->vq, ring_num, virtnet_sq_free_unused_buf); + if (err) + netdev_err(vi->dev, "resize tx fail: tx queue index: %d err: %d\n", qindex, err); + + virtnet_tx_resume(vi, sq); + return err; } From patchwork Fri Jun 14 06:39:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13697905 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-119.freemail.mail.aliyun.com (out30-119.freemail.mail.aliyun.com [115.124.30.119]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5D4DE1487C9; Fri, 14 Jun 2024 06:39:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.119 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718347185; cv=none; b=hVrnvmQLUuI8Hg38rPIX1gCw4NbtA+8nbIOHNHZZfH+Ed/UkSy8xXOGEAkzmiHdvPMdxV77ghatUGxVJIgngx0NjO6SE/SIBBYJ1QB96pCFHnNXOTagDKSVe6zlCWWw6Y9eFabZVdQUuPTK/U4JZ7madcvVaNi9Ej8tqvZSntxQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718347185; c=relaxed/simple; bh=4seTsozVSv+ezpete5JyfgV166HQrZcU8nGvlj7gUss=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=HVotdssagK0mVdWZyFKI/ZOsa9AQce8mC5D8AaKKP0x+FTMBLogj35EqdRiM/UIu96l2S/xLo+p2bge4j+sNaHvIOsLGDvp0ZTALEipqNrkr06S1cYq7pWRu2SLMZ2UvCMTb0fmZHTQpr71PggvRLBG/SOghueoAj4jMmncoxso= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=O9X0ZdiR; arc=none smtp.client-ip=115.124.30.119 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="O9X0ZdiR" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1718347181; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=npQee9M15+Lo4pMMFfrdbZsJ16sd84cpsyzjvN2BTSw=; b=O9X0ZdiRqUN+Pu1Iy2SI557ZzOv8TlcPrCGXsqmKoa1otNxkHs9tJ17gEoVOcdEg3/isV3MlvuqaOovnk+13fn8s25BzYbKE9GUFOIpLvcMai/0lXN+2WY/ZrF/i7eZ4Cp9obM8LOTSKtkHBf5qiCvyv7ncfAZ5Hy7v3A16dBlc= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R181e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033032014031;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=15;SR=0;TI=SMTPD_---0W8QF6Rt_1718347179; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W8QF6Rt_1718347179) by smtp.aliyun-inc.com; Fri, 14 Jun 2024 14:39:39 +0800 From: Xuan Zhuo To: netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , =?utf-8?q?Eugenio_P=C3=A9rez?= , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , virtualization@lists.linux.dev, bpf@vger.kernel.org Subject: [PATCH net-next v5 06/15] virtio_net: separate receive_buf Date: Fri, 14 Jun 2024 14:39:24 +0800 Message-Id: <20240614063933.108811-7-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240614063933.108811-1-xuanzhuo@linux.alibaba.com> References: <20240614063933.108811-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: e008fb4a0943 X-Patchwork-Delegate: kuba@kernel.org This commit separates the function receive_buf(), then we wrap the logic of handling the skb to an independent function virtnet_receive_done(). The subsequent commit will reuse it. Signed-off-by: Xuan Zhuo Acked-by: Jason Wang --- drivers/net/virtio_net.c | 56 +++++++++++++++++++++++----------------- 1 file changed, 32 insertions(+), 24 deletions(-) diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index 8c51947abce9..161694957065 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -1935,32 +1935,11 @@ static void virtio_skb_set_hash(const struct virtio_net_hdr_v1_hash *hdr_hash, skb_set_hash(skb, __le32_to_cpu(hdr_hash->hash_value), rss_hash_type); } -static void receive_buf(struct virtnet_info *vi, struct receive_queue *rq, - void *buf, unsigned int len, void **ctx, - unsigned int *xdp_xmit, - struct virtnet_rq_stats *stats) +static void virtnet_receive_done(struct virtnet_info *vi, struct receive_queue *rq, + struct sk_buff *skb) { - struct net_device *dev = vi->dev; - struct sk_buff *skb; struct virtio_net_common_hdr *hdr; - - if (unlikely(len < vi->hdr_len + ETH_HLEN)) { - pr_debug("%s: short packet %i\n", dev->name, len); - DEV_STATS_INC(dev, rx_length_errors); - virtnet_rq_free_buf(vi, rq, buf); - return; - } - - if (vi->mergeable_rx_bufs) - skb = receive_mergeable(dev, vi, rq, buf, ctx, len, xdp_xmit, - stats); - else if (vi->big_packets) - skb = receive_big(dev, vi, rq, buf, len, stats); - else - skb = receive_small(dev, vi, rq, buf, ctx, len, xdp_xmit, stats); - - if (unlikely(!skb)) - return; + struct net_device *dev = vi->dev; hdr = skb_vnet_common_hdr(skb); if (dev->features & NETIF_F_RXHASH && vi->has_rss_hash_report) @@ -1990,6 +1969,35 @@ static void receive_buf(struct virtnet_info *vi, struct receive_queue *rq, dev_kfree_skb(skb); } +static void receive_buf(struct virtnet_info *vi, struct receive_queue *rq, + void *buf, unsigned int len, void **ctx, + unsigned int *xdp_xmit, + struct virtnet_rq_stats *stats) +{ + struct net_device *dev = vi->dev; + struct sk_buff *skb; + + if (unlikely(len < vi->hdr_len + ETH_HLEN)) { + pr_debug("%s: short packet %i\n", dev->name, len); + DEV_STATS_INC(dev, rx_length_errors); + virtnet_rq_free_buf(vi, rq, buf); + return; + } + + if (vi->mergeable_rx_bufs) + skb = receive_mergeable(dev, vi, rq, buf, ctx, len, xdp_xmit, + stats); + else if (vi->big_packets) + skb = receive_big(dev, vi, rq, buf, len, stats); + else + skb = receive_small(dev, vi, rq, buf, ctx, len, xdp_xmit, stats); + + if (unlikely(!skb)) + return; + + virtnet_receive_done(vi, rq, skb); +} + /* Unlike mergeable buffers, all buffers are allocated to the * same size, except for the headroom. For this reason we do * not need to use mergeable_len_to_ctx here - it is enough From patchwork Fri Jun 14 06:39:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13697914 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-133.freemail.mail.aliyun.com (out30-133.freemail.mail.aliyun.com [115.124.30.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C4D461487C0; Fri, 14 Jun 2024 06:39:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.133 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718347191; cv=none; b=eVJPlm4GpGGiaE34xFt7fH0K3XDrJ6hMhcXRsi63uSDJ9ptunkEzG2tqR1gtfC6e6CYU1iSDtSGukqOX+KNK+jgZn6AjQ0N5aHXouxcj30GJs5YYlRkLO9R29WSQA/dS339ZcDTuw792GFxn3bK2uuWXczGLzAM4J1QLJZ6Rygg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718347191; c=relaxed/simple; bh=wByw0bT2yfYbSS2RCYMCVKjRqbg2fzSsRoeS4hrYkC4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=EsaiNb/VyKrE+n/oUWcpUsrd+3pU9eIdW7AvBtahfvoI7Tgq3P16YNCe7CM2GwAGD/cqztsJDlH8pGybHO1TcZ2UFYNGd2fRhR3Cne/mpxvrN9j6MLB2c4jT55zoUatKjpgp8BESidOSsV/r5IwzyoKYlc9r2i/uZMdjWIbIzho= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=MB8q6sTP; arc=none smtp.client-ip=115.124.30.133 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="MB8q6sTP" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1718347181; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=MWecKdKmfziyZ50SMpKq4drSxB5pIfvjl3IquVboWWc=; b=MB8q6sTPYpb+O6/9F+b+optHSmnqbZ+k73T3DPrU5sqwre+SFgLKht0yRtBe7kL4UoWy1HaQcs69GzBN21Ji+BtcGCtpfvuWNid01d5U8em9f1vA1YsH00R4Ya06wEGLdtaT7j6WvE9bClnvJtUk7np3zjj8mKi2ez2Z5X0T6Yk= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R791e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033022160150;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=15;SR=0;TI=SMTPD_---0W8QF6S3_1718347180; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W8QF6S3_1718347180) by smtp.aliyun-inc.com; Fri, 14 Jun 2024 14:39:40 +0800 From: Xuan Zhuo To: netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , =?utf-8?q?Eugenio_P=C3=A9rez?= , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , virtualization@lists.linux.dev, bpf@vger.kernel.org Subject: [PATCH net-next v5 07/15] virtio_net: refactor the xmit type Date: Fri, 14 Jun 2024 14:39:25 +0800 Message-Id: <20240614063933.108811-8-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240614063933.108811-1-xuanzhuo@linux.alibaba.com> References: <20240614063933.108811-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: e008fb4a0943 X-Patchwork-Delegate: kuba@kernel.org Because the af-xdp and sq premapped mode will introduce two new xmit type, so I refactor the xmit type mechanism first. We use the last two bits of the pointer to distinguish the xmit type, so we can distinguish four xmit types. Now we have two xmit types: SKB and XDP. Signed-off-by: Xuan Zhuo --- drivers/net/virtio_net.c | 58 +++++++++++++++++++++++++++------------- 1 file changed, 40 insertions(+), 18 deletions(-) diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index 161694957065..e84a4624549b 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -47,8 +47,6 @@ module_param(napi_tx, bool, 0644); #define VIRTIO_XDP_TX BIT(0) #define VIRTIO_XDP_REDIR BIT(1) -#define VIRTIO_XDP_FLAG BIT(0) - /* RX packet size EWMA. The average packet size is used to determine the packet * buffer size when refilling RX rings. As the entire RX ring may be refilled * at once, the weight is chosen so that the EWMA will be insensitive to short- @@ -491,42 +489,62 @@ struct virtio_net_common_hdr { static void virtnet_sq_free_unused_buf(struct virtqueue *vq, void *buf); -static bool is_xdp_frame(void *ptr) +enum virtnet_xmit_type { + VIRTNET_XMIT_TYPE_SKB, + VIRTNET_XMIT_TYPE_XDP, +}; + +#define VIRTNET_XMIT_TYPE_MASK (VIRTNET_XMIT_TYPE_SKB | VIRTNET_XMIT_TYPE_XDP) + +static enum virtnet_xmit_type virtnet_xmit_ptr_strip(void **ptr) { - return (unsigned long)ptr & VIRTIO_XDP_FLAG; + unsigned long p = (unsigned long)*ptr; + + *ptr = (void *)(p & ~VIRTNET_XMIT_TYPE_MASK); + + return p & VIRTNET_XMIT_TYPE_MASK; } -static void *xdp_to_ptr(struct xdp_frame *ptr) +static void *virtnet_xmit_ptr_mix(void *ptr, enum virtnet_xmit_type type) { - return (void *)((unsigned long)ptr | VIRTIO_XDP_FLAG); + return (void *)((unsigned long)ptr | type); } -static struct xdp_frame *ptr_to_xdp(void *ptr) +static int virtnet_add_outbuf(struct send_queue *sq, int num, void *data, + enum virtnet_xmit_type type) { - return (struct xdp_frame *)((unsigned long)ptr & ~VIRTIO_XDP_FLAG); + return virtqueue_add_outbuf(sq->vq, sq->sg, num, + virtnet_xmit_ptr_mix(data, type), + GFP_ATOMIC); } static void __free_old_xmit(struct send_queue *sq, bool in_napi, struct virtnet_sq_free_stats *stats) { + struct xdp_frame *frame; + struct sk_buff *skb; unsigned int len; void *ptr; while ((ptr = virtqueue_get_buf(sq->vq, &len)) != NULL) { ++stats->packets; - if (!is_xdp_frame(ptr)) { - struct sk_buff *skb = ptr; + switch (virtnet_xmit_ptr_strip(&ptr)) { + case VIRTNET_XMIT_TYPE_SKB: + skb = ptr; pr_debug("Sent skb %p\n", skb); stats->bytes += skb->len; napi_consume_skb(skb, in_napi); - } else { - struct xdp_frame *frame = ptr_to_xdp(ptr); + break; + + case VIRTNET_XMIT_TYPE_XDP: + frame = ptr; stats->bytes += xdp_get_frame_len(frame); xdp_return_frame(frame); + break; } } } @@ -1064,8 +1082,7 @@ static int __virtnet_xdp_xmit_one(struct virtnet_info *vi, skb_frag_size(frag), skb_frag_off(frag)); } - err = virtqueue_add_outbuf(sq->vq, sq->sg, nr_frags + 1, - xdp_to_ptr(xdpf), GFP_ATOMIC); + err = virtnet_add_outbuf(sq, nr_frags + 1, xdpf, VIRTNET_XMIT_TYPE_XDP); if (unlikely(err)) return -ENOSPC; /* Caller handle free/refcnt */ @@ -2557,7 +2574,7 @@ static int xmit_skb(struct send_queue *sq, struct sk_buff *skb) return num_sg; num_sg++; } - return virtqueue_add_outbuf(sq->vq, sq->sg, num_sg, skb, GFP_ATOMIC); + return virtnet_add_outbuf(sq, num_sg, skb, VIRTNET_XMIT_TYPE_SKB); } static netdev_tx_t start_xmit(struct sk_buff *skb, struct net_device *dev) @@ -5263,10 +5280,15 @@ static void free_receive_page_frags(struct virtnet_info *vi) static void virtnet_sq_free_unused_buf(struct virtqueue *vq, void *buf) { - if (!is_xdp_frame(buf)) + switch (virtnet_xmit_ptr_strip(&buf)) { + case VIRTNET_XMIT_TYPE_SKB: dev_kfree_skb(buf); - else - xdp_return_frame(ptr_to_xdp(buf)); + break; + + case VIRTNET_XMIT_TYPE_XDP: + xdp_return_frame(buf); + break; + } } static void free_unused_bufs(struct virtnet_info *vi) From patchwork Fri Jun 14 06:39:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13697906 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-101.freemail.mail.aliyun.com (out30-101.freemail.mail.aliyun.com [115.124.30.101]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 21A51148319; Fri, 14 Jun 2024 06:39:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.101 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718347186; cv=none; b=Fipr5AnndxqUprnIGbAzpes9qqMfQSa/pW5sStcneMlYs7ikjDZYzORz8NgoyxnKnkRztSYURYswwMNx6KO7mQKhjkKVKumDdG3XVp9mOTS0wf7+tozL4aVdxE04YD1bMNbUsDVtbZkwjynp1t17fECjPkHhrZrEQm0Rwj8aCZ0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718347186; c=relaxed/simple; bh=nwexYCz1T+L2DdYqGjJCPzMa7rW6o5J+jtI8VMg43jw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=um1bvSGwxxcLnmwFzkuvSQjASu/Ztc9ScNLJbon7ZX2klYHOzBEM8wO7qvG1aCHVqu+ztS6bw7rbGCfXPK31aOY02JK4+zvJFjJC7+SaYMajveAc2viHzQmD+T4wZhAr29vqbOxN+maYq8mYya3hWA3y8RjiMG+YBfzdkGXPcRw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=skjVAGvk; arc=none smtp.client-ip=115.124.30.101 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="skjVAGvk" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1718347182; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=A4g4giwFwkBAmjzmCqfn9xxtnXuPwka8EB6NUbkE/50=; b=skjVAGvkpWudBXG00Nw5JwLIRCvaC9qJ9mAyhptFCwth4X8Pej4vQg3DZ2c1i31prnsYMcap0Cdo+niPHKhjWJMkZ0F6FG4L11zm4qehE+JfSck9X8g0pdfl3HyY4KPMSgQa7goxqmX/Hfu39FlsSierSasu2HXTPy5EAsOU+gg= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R441e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033022160150;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=15;SR=0;TI=SMTPD_---0W8Q9u9c_1718347180; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W8Q9u9c_1718347180) by smtp.aliyun-inc.com; Fri, 14 Jun 2024 14:39:41 +0800 From: Xuan Zhuo To: netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , =?utf-8?q?Eugenio_P=C3=A9rez?= , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , virtualization@lists.linux.dev, bpf@vger.kernel.org Subject: [PATCH net-next v5 08/15] virtio_net: sq support premapped mode Date: Fri, 14 Jun 2024 14:39:26 +0800 Message-Id: <20240614063933.108811-9-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240614063933.108811-1-xuanzhuo@linux.alibaba.com> References: <20240614063933.108811-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: e008fb4a0943 X-Patchwork-Delegate: kuba@kernel.org If the xsk is enabling, the xsk tx will share the send queue. But the xsk requires that the send queue use the premapped mode. So the send queue must support premapped mode when it is bound to af-xdp. * virtnet_sq_set_premapped(sq, true) is used to enable premapped mode. In this mode, the driver will record the dma info when skb or xdp frame is sent. Currently, the SQ premapped mode is operational only with af-xdp. In this mode, af-xdp, the kernel stack, and xdp tx/redirect will share the same SQ. Af-xdp independently manages its DMA. The kernel stack and xdp tx/redirect utilize this DMA metadata to manage the DMA info. If the indirect descriptor feature be supported, the volume of DMA details we need to maintain becomes quite substantial. Here, we have a cap on the amount of DMA info we manage. If the kernel stack and xdp tx/redirect attempt to use more descriptors, virtnet_add_outbuf() will return an -ENOMEM error. But the af-xdp can work continually. * virtnet_sq_set_premapped(sq, false) is used to disable premapped mode. Signed-off-by: Xuan Zhuo --- drivers/net/virtio_net.c | 228 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 224 insertions(+), 4 deletions(-) diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index e84a4624549b..88ab9ea1646f 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -25,6 +25,7 @@ #include #include #include +#include static int napi_weight = NAPI_POLL_WEIGHT; module_param(napi_weight, int, 0444); @@ -276,6 +277,26 @@ struct virtnet_rq_dma { u16 need_sync; }; +struct virtnet_sq_dma { + union { + struct llist_node node; + struct llist_head head; + void *data; + }; + dma_addr_t addr; + u32 len; + u8 num; +}; + +struct virtnet_sq_dma_info { + /* record for kfree */ + void *p; + + u32 free_num; + + struct llist_head free; +}; + /* Internal representation of a send virtqueue */ struct send_queue { /* Virtqueue associated with this send _queue */ @@ -295,6 +316,11 @@ struct send_queue { /* Record whether sq is in reset state. */ bool reset; + + /* SQ is premapped mode or not. */ + bool premapped; + + struct virtnet_sq_dma_info dmainfo; }; /* Internal representation of a receive virtqueue */ @@ -492,9 +518,11 @@ static void virtnet_sq_free_unused_buf(struct virtqueue *vq, void *buf); enum virtnet_xmit_type { VIRTNET_XMIT_TYPE_SKB, VIRTNET_XMIT_TYPE_XDP, + VIRTNET_XMIT_TYPE_DMA, }; -#define VIRTNET_XMIT_TYPE_MASK (VIRTNET_XMIT_TYPE_SKB | VIRTNET_XMIT_TYPE_XDP) +#define VIRTNET_XMIT_TYPE_MASK (VIRTNET_XMIT_TYPE_SKB | VIRTNET_XMIT_TYPE_XDP \ + | VIRTNET_XMIT_TYPE_DMA) static enum virtnet_xmit_type virtnet_xmit_ptr_strip(void **ptr) { @@ -510,12 +538,180 @@ static void *virtnet_xmit_ptr_mix(void *ptr, enum virtnet_xmit_type type) return (void *)((unsigned long)ptr | type); } +static void virtnet_sq_unmap(struct send_queue *sq, void **data) +{ + struct virtnet_sq_dma *head, *tail, *p; + int i; + + head = *data; + + p = head; + + for (i = 0; i < head->num; ++i) { + virtqueue_dma_unmap_page_attrs(sq->vq, p->addr, p->len, + DMA_TO_DEVICE, 0); + tail = p; + p = llist_entry(llist_next(&p->node), struct virtnet_sq_dma, node); + } + + *data = tail->data; + + __llist_add_batch(&head->node, &tail->node, &sq->dmainfo.free); + + sq->dmainfo.free_num += head->num; +} + +static void *virtnet_dma_chain_update(struct send_queue *sq, + struct virtnet_sq_dma *head, + struct virtnet_sq_dma *tail, + u8 num, void *data) +{ + sq->dmainfo.free_num -= num; + head->num = num; + + tail->data = data; + + return virtnet_xmit_ptr_mix(head, VIRTNET_XMIT_TYPE_DMA); +} + +static struct virtnet_sq_dma *virtnet_sq_map_sg(struct send_queue *sq, int num, void *data) +{ + struct virtnet_sq_dma *head = NULL, *p = NULL; + struct scatterlist *sg; + dma_addr_t addr; + int i, err; + + if (num > sq->dmainfo.free_num) + return NULL; + + for (i = 0; i < num; ++i) { + sg = &sq->sg[i]; + + addr = virtqueue_dma_map_page_attrs(sq->vq, sg_page(sg), + sg->offset, + sg->length, DMA_TO_DEVICE, + 0); + err = virtqueue_dma_mapping_error(sq->vq, addr); + if (err) + goto err; + + sg->dma_address = addr; + + p = llist_entry(llist_del_first(&sq->dmainfo.free), + struct virtnet_sq_dma, node); + + p->addr = sg->dma_address; + p->len = sg->length; + + if (head) + __llist_add(&p->node, &head->head); + else + head = p; + } + + return virtnet_dma_chain_update(sq, head, p, num, data); + +err: + if (i) { + virtnet_dma_chain_update(sq, head, p, i, data); + virtnet_sq_unmap(sq, (void **)&head); + } + + return NULL; +} + static int virtnet_add_outbuf(struct send_queue *sq, int num, void *data, enum virtnet_xmit_type type) { - return virtqueue_add_outbuf(sq->vq, sq->sg, num, - virtnet_xmit_ptr_mix(data, type), - GFP_ATOMIC); + int ret; + + data = virtnet_xmit_ptr_mix(data, type); + + if (sq->premapped) { + data = virtnet_sq_map_sg(sq, num, data); + if (!data) + return -ENOMEM; + } + + ret = virtqueue_add_outbuf(sq->vq, sq->sg, num, data, GFP_ATOMIC); + if (ret && sq->premapped) { + virtnet_xmit_ptr_strip(&data); + virtnet_sq_unmap(sq, &data); + } + + return ret; +} + +static int virtnet_sq_alloc_dma_meta(struct send_queue *sq) +{ + struct virtnet_sq_dma *d; + int num, i; + + num = virtqueue_get_vring_size(sq->vq); + + /* Currently, the SQ premapped mode is operational only with af-xdp. In + * this mode, af-xdp, the kernel stack, and xdp tx/redirect will share + * the same SQ. Af-xdp independently manages its DMA. The kernel stack + * and xdp tx/redirect utilize this DMA metadata to manage the DMA info. + * + * If the indirect descriptor feature be supported, the volume of DMA + * details we need to maintain becomes quite substantial. Here, we have + * a cap on the amount of DMA info we manage, effectively limiting it to + * twice the size of the ring buffer. + * + * If the kernel stack and xdp tx/redirect attempt to use more + * descriptors than allowed by this double ring buffer size, + * virtnet_add_outbuf() will return an -ENOMEM error. But the af-xdp can + * work continually. + */ + if (virtio_has_feature(sq->vq->vdev, VIRTIO_RING_F_INDIRECT_DESC)) + num = num * 2; + + sq->dmainfo.p = kvcalloc(num, sizeof(struct virtnet_sq_dma), GFP_KERNEL); + if (!sq->dmainfo.p) + return -ENOMEM; + + init_llist_head(&sq->dmainfo.free); + + sq->dmainfo.free_num = num; + + for (i = 0; i < num; ++i) { + d = sq->dmainfo.p + sizeof(struct virtnet_sq_dma) * i; + + __llist_add(&d->node, &sq->dmainfo.free); + } + + return 0; +} + +static void virtnet_sq_free_dma_meta(struct send_queue *sq) +{ + kvfree(sq->dmainfo.p); + + sq->dmainfo.p = NULL; + sq->dmainfo.free_num = 0; +} + +/* This function must be called immediately after creating the vq, or after vq + * reset, and before adding any buffers to it. + */ +static __maybe_unused int virtnet_sq_set_premapped(struct send_queue *sq, bool premapped) +{ + if (premapped) { + int r; + + r = virtnet_sq_alloc_dma_meta(sq); + + if (r) + return r; + } else { + virtnet_sq_free_dma_meta(sq); + } + + BUG_ON(virtqueue_set_dma_premapped(sq->vq, premapped)); + + sq->premapped = premapped; + return 0; } static void __free_old_xmit(struct send_queue *sq, bool in_napi, @@ -529,6 +725,7 @@ static void __free_old_xmit(struct send_queue *sq, bool in_napi, while ((ptr = virtqueue_get_buf(sq->vq, &len)) != NULL) { ++stats->packets; +retry: switch (virtnet_xmit_ptr_strip(&ptr)) { case VIRTNET_XMIT_TYPE_SKB: skb = ptr; @@ -545,6 +742,16 @@ static void __free_old_xmit(struct send_queue *sq, bool in_napi, stats->bytes += xdp_get_frame_len(frame); xdp_return_frame(frame); break; + + case VIRTNET_XMIT_TYPE_DMA: + virtnet_sq_unmap(sq, &ptr); + + /* For TYPE_DMA, the ptr pointed to the virtnet_sq_dma + * struct. After the virtnet_sq_unmap, the ptr points to + * the skb or xdp pointer | TYPE. So we call the strip + * func again. + */ + goto retry; } } } @@ -5232,6 +5439,8 @@ static void virtnet_free_queues(struct virtnet_info *vi) for (i = 0; i < vi->max_queue_pairs; i++) { __netif_napi_del(&vi->rq[i].napi); __netif_napi_del(&vi->sq[i].napi); + + virtnet_sq_free_dma_meta(&vi->sq[i]); } /* We called __netif_napi_del(), @@ -5280,6 +5489,13 @@ static void free_receive_page_frags(struct virtnet_info *vi) static void virtnet_sq_free_unused_buf(struct virtqueue *vq, void *buf) { + struct virtnet_info *vi = vq->vdev->priv; + struct send_queue *sq; + int i = vq2rxq(vq); + + sq = &vi->sq[i]; + +retry: switch (virtnet_xmit_ptr_strip(&buf)) { case VIRTNET_XMIT_TYPE_SKB: dev_kfree_skb(buf); @@ -5288,6 +5504,10 @@ static void virtnet_sq_free_unused_buf(struct virtqueue *vq, void *buf) case VIRTNET_XMIT_TYPE_XDP: xdp_return_frame(buf); break; + + case VIRTNET_XMIT_TYPE_DMA: + virtnet_sq_unmap(sq, &buf); + goto retry; } } From patchwork Fri Jun 14 06:39:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13697908 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-119.freemail.mail.aliyun.com (out30-119.freemail.mail.aliyun.com [115.124.30.119]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CBC7614882E; Fri, 14 Jun 2024 06:39:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.119 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718347187; cv=none; b=RmCz2yRq5pPvGf868d3rtHosTAGfLQXNlF5+nLpTyHkkPggfuQcp9J0zyu9lTtPnVxfQ8FFRrfNyDxR+uxYBU7v72b/03Bu+XO7tIMGQrsQoidPTcEUsy+ICDeakfef5z9FEMFnh/5L4z/rmjwfCiWBQSEtdlZlKU1aepPSyzBY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718347187; c=relaxed/simple; bh=Np/C09W9yZYjHwosSlj0aFHMRzQvkZx0IAaAl8Zi+aA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=FH21cMeDuHNQjpOiV3mUHADz8MBOzi4d42zuQ5ZwEMv8bscbbx1tjTFcLEH5eboAOFCLCFcUcKKxG18QIjFOgLO1Zs9qnKbVqXNZ59zLVGPZ3g1bdTv19d/zhMXbISWjkLhNNkXuRf9/bKHJXNPHbqBNJgd0ah4IMl+xyq9GzcQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=gq0Y2+my; arc=none smtp.client-ip=115.124.30.119 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="gq0Y2+my" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1718347182; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=KG/bd50OMKJaDIdNSTSVLUrSecHFGb0oLXPCZ9UUqtc=; b=gq0Y2+myYJy4qKppPCmgimozZJQMEbu6luvOP3iVSydu6HYBvf3hMurJw9nvCGp1nv+QvIEzA6jOAY8JpkkzUWvaG/sXKE25mRLAH65zvfHHH9F0FKWvfgT20EnkwD9wJyqtTYU03e2ga9oZsMXLV16NEwsDVTT3EI7Wh6FcZRQ= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R131e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033037067112;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=15;SR=0;TI=SMTPD_---0W8QKgUj_1718347181; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W8QKgUj_1718347181) by smtp.aliyun-inc.com; Fri, 14 Jun 2024 14:39:42 +0800 From: Xuan Zhuo To: netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , =?utf-8?q?Eugenio_P=C3=A9rez?= , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , virtualization@lists.linux.dev, bpf@vger.kernel.org Subject: [PATCH net-next v5 09/15] virtio_net: xsk: bind/unbind xsk Date: Fri, 14 Jun 2024 14:39:27 +0800 Message-Id: <20240614063933.108811-10-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240614063933.108811-1-xuanzhuo@linux.alibaba.com> References: <20240614063933.108811-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: e008fb4a0943 X-Patchwork-Delegate: kuba@kernel.org This patch implement the logic of bind/unbind xsk pool to sq and rq. Signed-off-by: Xuan Zhuo --- drivers/net/virtio_net.c | 201 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 200 insertions(+), 1 deletion(-) diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index 88ab9ea1646f..35fd8bca7fcf 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -26,6 +26,7 @@ #include #include #include +#include static int napi_weight = NAPI_POLL_WEIGHT; module_param(napi_weight, int, 0444); @@ -57,6 +58,8 @@ DECLARE_EWMA(pkt_len, 0, 64) #define VIRTNET_DRIVER_VERSION "1.0.0" +static struct virtio_net_hdr_mrg_rxbuf xsk_hdr; + static const unsigned long guest_offloads[] = { VIRTIO_NET_F_GUEST_TSO4, VIRTIO_NET_F_GUEST_TSO6, @@ -321,6 +324,12 @@ struct send_queue { bool premapped; struct virtnet_sq_dma_info dmainfo; + + struct { + struct xsk_buff_pool *pool; + + dma_addr_t hdr_dma_address; + } xsk; }; /* Internal representation of a receive virtqueue */ @@ -372,6 +381,13 @@ struct receive_queue { /* Record the last dma info to free after new pages is allocated. */ struct virtnet_rq_dma *last_dma; + + struct { + struct xsk_buff_pool *pool; + + /* xdp rxq used by xsk */ + struct xdp_rxq_info xdp_rxq; + } xsk; }; /* This structure can contain rss message with maximum settings for indirection table and keysize @@ -695,7 +711,7 @@ static void virtnet_sq_free_dma_meta(struct send_queue *sq) /* This function must be called immediately after creating the vq, or after vq * reset, and before adding any buffers to it. */ -static __maybe_unused int virtnet_sq_set_premapped(struct send_queue *sq, bool premapped) +static int virtnet_sq_set_premapped(struct send_queue *sq, bool premapped) { if (premapped) { int r; @@ -5177,6 +5193,187 @@ static int virtnet_restore_guest_offloads(struct virtnet_info *vi) return virtnet_set_guest_offloads(vi, offloads); } +static int virtnet_rq_bind_xsk_pool(struct virtnet_info *vi, struct receive_queue *rq, + struct xsk_buff_pool *pool) +{ + int err, qindex; + + qindex = rq - vi->rq; + + if (pool) { + err = xdp_rxq_info_reg(&rq->xsk.xdp_rxq, vi->dev, qindex, rq->napi.napi_id); + if (err < 0) + return err; + + err = xdp_rxq_info_reg_mem_model(&rq->xsk.xdp_rxq, + MEM_TYPE_XSK_BUFF_POOL, NULL); + if (err < 0) { + xdp_rxq_info_unreg(&rq->xsk.xdp_rxq); + return err; + } + + xsk_pool_set_rxq_info(pool, &rq->xsk.xdp_rxq); + } + + virtnet_rx_pause(vi, rq); + + err = virtqueue_reset(rq->vq, virtnet_rq_unmap_free_buf); + if (err) { + netdev_err(vi->dev, "reset rx fail: rx queue index: %d err: %d\n", qindex, err); + + pool = NULL; + } + + if (!pool) + xdp_rxq_info_unreg(&rq->xsk.xdp_rxq); + + rq->xsk.pool = pool; + + virtnet_rx_resume(vi, rq); + + return err; +} + +static int virtnet_sq_bind_xsk_pool(struct virtnet_info *vi, + struct send_queue *sq, + struct xsk_buff_pool *pool) +{ + int err, qindex; + + qindex = sq - vi->sq; + + virtnet_tx_pause(vi, sq); + + err = virtqueue_reset(sq->vq, virtnet_sq_free_unused_buf); + if (err) + netdev_err(vi->dev, "reset tx fail: tx queue index: %d err: %d\n", qindex, err); + else + err = virtnet_sq_set_premapped(sq, !!pool); + + if (err) + pool = NULL; + + sq->xsk.pool = pool; + + virtnet_tx_resume(vi, sq); + + return err; +} + +static int virtnet_xsk_pool_enable(struct net_device *dev, + struct xsk_buff_pool *pool, + u16 qid) +{ + struct virtnet_info *vi = netdev_priv(dev); + struct receive_queue *rq; + struct send_queue *sq; + struct device *dma_dev; + dma_addr_t hdr_dma; + int err; + + /* In big_packets mode, xdp cannot work, so there is no need to + * initialize xsk of rq. + * + * Support for small mode firstly. + */ + if (vi->big_packets) + return -ENOENT; + + if (qid >= vi->curr_queue_pairs) + return -EINVAL; + + sq = &vi->sq[qid]; + rq = &vi->rq[qid]; + + /* xsk tx zerocopy depend on the tx napi. + * + * All xsk packets are actually consumed and sent out from the xsk tx + * queue under the tx napi mechanism. + */ + if (!sq->napi.weight) + return -EPERM; + + /* For the xsk, the tx and rx should have the same device. But + * vq->dma_dev allows every vq has the respective dma dev. So I check + * the dma dev of vq and sq is the same dev. + */ + if (virtqueue_dma_dev(rq->vq) != virtqueue_dma_dev(sq->vq)) + return -EPERM; + + dma_dev = virtqueue_dma_dev(rq->vq); + if (!dma_dev) + return -EPERM; + + hdr_dma = dma_map_single(dma_dev, &xsk_hdr, vi->hdr_len, DMA_TO_DEVICE); + if (dma_mapping_error(dma_dev, hdr_dma)) + return -ENOMEM; + + err = xsk_pool_dma_map(pool, dma_dev, 0); + if (err) + goto err_xsk_map; + + err = virtnet_rq_bind_xsk_pool(vi, rq, pool); + if (err) + goto err_rq; + + err = virtnet_sq_bind_xsk_pool(vi, sq, pool); + if (err) + goto err_sq; + + /* Now, we do not support tx offset, so all the tx virtnet hdr is zero. + * So all the tx packets can share a single hdr. + */ + sq->xsk.hdr_dma_address = hdr_dma; + + return 0; + +err_sq: + virtnet_rq_bind_xsk_pool(vi, rq, NULL); +err_rq: + xsk_pool_dma_unmap(pool, 0); +err_xsk_map: + dma_unmap_single(dma_dev, hdr_dma, vi->hdr_len, DMA_TO_DEVICE); + return err; +} + +static int virtnet_xsk_pool_disable(struct net_device *dev, u16 qid) +{ + struct virtnet_info *vi = netdev_priv(dev); + struct xsk_buff_pool *pool; + struct device *dma_dev; + struct receive_queue *rq; + struct send_queue *sq; + int err1, err2; + + if (qid >= vi->curr_queue_pairs) + return -EINVAL; + + sq = &vi->sq[qid]; + rq = &vi->rq[qid]; + + pool = sq->xsk.pool; + + err1 = virtnet_sq_bind_xsk_pool(vi, sq, NULL); + err2 = virtnet_rq_bind_xsk_pool(vi, rq, NULL); + + xsk_pool_dma_unmap(pool, 0); + + dma_dev = virtqueue_dma_dev(rq->vq); + + dma_unmap_single(dma_dev, sq->xsk.hdr_dma_address, vi->hdr_len, DMA_TO_DEVICE); + + return err1 | err2; +} + +static int virtnet_xsk_pool_setup(struct net_device *dev, struct netdev_bpf *xdp) +{ + if (xdp->xsk.pool) + return virtnet_xsk_pool_enable(dev, xdp->xsk.pool, + xdp->xsk.queue_id); + else + return virtnet_xsk_pool_disable(dev, xdp->xsk.queue_id); +} + static int virtnet_xdp_set(struct net_device *dev, struct bpf_prog *prog, struct netlink_ext_ack *extack) { @@ -5302,6 +5499,8 @@ static int virtnet_xdp(struct net_device *dev, struct netdev_bpf *xdp) switch (xdp->command) { case XDP_SETUP_PROG: return virtnet_xdp_set(dev, xdp->prog, xdp->extack); + case XDP_SETUP_XSK_POOL: + return virtnet_xsk_pool_setup(dev, xdp); default: return -EINVAL; } From patchwork Fri Jun 14 06:39:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13697910 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-100.freemail.mail.aliyun.com (out30-100.freemail.mail.aliyun.com [115.124.30.100]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DFB52148858; Fri, 14 Jun 2024 06:39:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.100 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718347187; cv=none; b=q91nCZMT8INHJuKIGS8CWqzY7WwS5EODtdw/97RHEFhslSMf4ubJP9cg9GOY0aRxXBPK3CaqpS8pOKMvwWpya0I3poCLvhKNPWOpO5koJf1zc1qNxRuYOvezvXHA00SZhbg3gHI4GB0VL/xeYNo980gUmqV1IX30DVcyaLB1BRE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718347187; c=relaxed/simple; bh=MVqiUbhM36USFb1zK5vYprnbNEY0Ueo+8g2JG0pei8A=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=J+zVSI1dhyahHgnXLhbnKynR5cax+nWDuDQI+Nl8uhQwpetM36+M+TaPfECCrM/0Y5tm59WQQJr7KMcrLtTKIba6PC47Us0YpXnKSIpybG4DK8g4BVtcMOZWSVIB/uehOz4WGNii5ixiu381Ett25+ht6a7wG7uscnlKoY70wM4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=m7DvQlHE; arc=none smtp.client-ip=115.124.30.100 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="m7DvQlHE" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1718347183; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=r4/jhH2IhiUj+1EGMSxVzby5BQcjGc1Wbn9DmT0Kewc=; b=m7DvQlHEYzwlhA2yvtJF9pddUaqyJ94wciD8DFj8q+7LwIGJJuNYF4haopy9UYXsZQSR6RnC2jmqH0dymc3MnsuFAtcKvZAYFCsnNHXPSG2QHLU2Z5M9TYpqy4+eNQsOK720HE/ecapGY1cQlDA98/VLnf586NsLach2tbVcUzw= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R121e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033037067113;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=15;SR=0;TI=SMTPD_---0W8QEkB4_1718347182; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W8QEkB4_1718347182) by smtp.aliyun-inc.com; Fri, 14 Jun 2024 14:39:43 +0800 From: Xuan Zhuo To: netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , =?utf-8?q?Eugenio_P=C3=A9rez?= , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , virtualization@lists.linux.dev, bpf@vger.kernel.org Subject: [PATCH net-next v5 10/15] virtio_net: xsk: prevent disable tx napi Date: Fri, 14 Jun 2024 14:39:28 +0800 Message-Id: <20240614063933.108811-11-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240614063933.108811-1-xuanzhuo@linux.alibaba.com> References: <20240614063933.108811-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: e008fb4a0943 X-Patchwork-Delegate: kuba@kernel.org Since xsk's TX queue is consumed by TX NAPI, if sq is bound to xsk, then we must stop tx napi from being disabled. Signed-off-by: Xuan Zhuo Acked-by: Jason Wang --- drivers/net/virtio_net.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index 35fd8bca7fcf..2767338dc060 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -4738,7 +4738,7 @@ static int virtnet_set_coalesce(struct net_device *dev, struct netlink_ext_ack *extack) { struct virtnet_info *vi = netdev_priv(dev); - int ret, queue_number, napi_weight; + int ret, queue_number, napi_weight, i; bool update_napi = false; /* Can't change NAPI weight if the link is up */ @@ -4767,6 +4767,14 @@ static int virtnet_set_coalesce(struct net_device *dev, return ret; if (update_napi) { + /* xsk xmit depends on the tx napi. So if xsk is active, + * prevent modifications to tx napi. + */ + for (i = queue_number; i < vi->max_queue_pairs; i++) { + if (vi->sq[i].xsk.pool) + return -EBUSY; + } + for (; queue_number < vi->max_queue_pairs; queue_number++) vi->sq[queue_number].napi.weight = napi_weight; } From patchwork Fri Jun 14 06:39:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13697917 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-99.freemail.mail.aliyun.com (out30-99.freemail.mail.aliyun.com [115.124.30.99]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C23CB1836FC; Fri, 14 Jun 2024 06:39:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.99 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718347194; cv=none; b=RaXDY4jhTHmCsilWidlIM7DDMIRavBzIe/8b67BvCN8TG44v47FGat8fGU0qWj2t4Pqav4OiBenfPPosJd7ygItJs1pTGmiP9gu3+EzVt1R7prWDVQMxsKREMuGQOG84bvT2LOmhXbUhXU7LD9FXUaYzvXfJ7DFqUYVtbsMUwiA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718347194; c=relaxed/simple; bh=lORUynQLlBgk0LIg09eEQb6gdoTTuvpHT8mCANsDWaM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=u232DpY6dB5ip7AD97yA+Ntx10azoZFNcy5ndnHcbqAt7rWbYBnFtyFlE8y3N6BYxZuy6JKlTtxmPQY46hE15AZogU4cbN+XQcik6d+SeLTMaCn187e1YstuEoI5CafNMZJnh1j9FNdPHMfXpyOKdI99SpIUyCH+4fw1GJFTjgk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=CppM8lQw; arc=none smtp.client-ip=115.124.30.99 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="CppM8lQw" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1718347184; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=vxhupZRiQo3CGiqsow8O30uFtigQb+2m40C54t88A7Y=; b=CppM8lQwFSI4FzMe/FMo27a90PAydTCtnbAxXBhEBfkaasNZcEv4f5HXZhsFsfzxEV5pm5+6kA+a1KOHxduO6vlIFrxhOmAPMEzlO3hkcou0G4ZSWHNi90t0aBbKNV38Tui/deDifNwNspSUQ0zuE2csJtwynSja5sS41Y2cImA= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R121e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033068173054;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=15;SR=0;TI=SMTPD_---0W8QF6TX_1718347183; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W8QF6TX_1718347183) by smtp.aliyun-inc.com; Fri, 14 Jun 2024 14:39:43 +0800 From: Xuan Zhuo To: netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , =?utf-8?q?Eugenio_P=C3=A9rez?= , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , virtualization@lists.linux.dev, bpf@vger.kernel.org Subject: [PATCH net-next v5 11/15] virtio_net: xsk: tx: support xmit xsk buffer Date: Fri, 14 Jun 2024 14:39:29 +0800 Message-Id: <20240614063933.108811-12-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240614063933.108811-1-xuanzhuo@linux.alibaba.com> References: <20240614063933.108811-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: e008fb4a0943 X-Patchwork-Delegate: kuba@kernel.org The driver's tx napi is very important for XSK. It is responsible for obtaining data from the XSK queue and sending it out. At the beginning, we need to trigger tx napi. Signed-off-by: Xuan Zhuo --- drivers/net/virtio_net.c | 121 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 119 insertions(+), 2 deletions(-) diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index 2767338dc060..7e811f392768 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -535,10 +535,13 @@ enum virtnet_xmit_type { VIRTNET_XMIT_TYPE_SKB, VIRTNET_XMIT_TYPE_XDP, VIRTNET_XMIT_TYPE_DMA, + VIRTNET_XMIT_TYPE_XSK, }; #define VIRTNET_XMIT_TYPE_MASK (VIRTNET_XMIT_TYPE_SKB | VIRTNET_XMIT_TYPE_XDP \ - | VIRTNET_XMIT_TYPE_DMA) + | VIRTNET_XMIT_TYPE_DMA | VIRTNET_XMIT_TYPE_XSK) + +#define VIRTIO_XSK_FLAG_OFFSET 4 static enum virtnet_xmit_type virtnet_xmit_ptr_strip(void **ptr) { @@ -768,6 +771,10 @@ static void __free_old_xmit(struct send_queue *sq, bool in_napi, * func again. */ goto retry; + + case VIRTNET_XMIT_TYPE_XSK: + /* Make gcc happy. DONE in subsequent commit */ + break; } } } @@ -1265,6 +1272,102 @@ static void check_sq_full_and_disable(struct virtnet_info *vi, } } +static void *virtnet_xsk_to_ptr(u32 len) +{ + unsigned long p; + + p = len << VIRTIO_XSK_FLAG_OFFSET; + + return virtnet_xmit_ptr_mix((void *)p, VIRTNET_XMIT_TYPE_XSK); +} + +static void sg_fill_dma(struct scatterlist *sg, dma_addr_t addr, u32 len) +{ + sg->dma_address = addr; + sg->length = len; +} + +static int virtnet_xsk_xmit_one(struct send_queue *sq, + struct xsk_buff_pool *pool, + struct xdp_desc *desc) +{ + struct virtnet_info *vi; + dma_addr_t addr; + + vi = sq->vq->vdev->priv; + + addr = xsk_buff_raw_get_dma(pool, desc->addr); + xsk_buff_raw_dma_sync_for_device(pool, addr, desc->len); + + sg_init_table(sq->sg, 2); + + sg_fill_dma(sq->sg, sq->xsk.hdr_dma_address, vi->hdr_len); + sg_fill_dma(sq->sg + 1, addr, desc->len); + + return virtqueue_add_outbuf(sq->vq, sq->sg, 2, + virtnet_xsk_to_ptr(desc->len), GFP_ATOMIC); +} + +static int virtnet_xsk_xmit_batch(struct send_queue *sq, + struct xsk_buff_pool *pool, + unsigned int budget, + u64 *kicks) +{ + struct xdp_desc *descs = pool->tx_descs; + bool kick = false; + u32 nb_pkts, i; + int err; + + budget = min_t(u32, budget, sq->vq->num_free); + + nb_pkts = xsk_tx_peek_release_desc_batch(pool, budget); + if (!nb_pkts) + return 0; + + for (i = 0; i < nb_pkts; i++) { + err = virtnet_xsk_xmit_one(sq, pool, &descs[i]); + if (unlikely(err)) { + xsk_tx_completed(sq->xsk.pool, nb_pkts - i); + break; + } + + kick = true; + } + + if (kick && virtqueue_kick_prepare(sq->vq) && virtqueue_notify(sq->vq)) + (*kicks)++; + + return i; +} + +static bool virtnet_xsk_xmit(struct send_queue *sq, struct xsk_buff_pool *pool, + int budget) +{ + struct virtnet_info *vi = sq->vq->vdev->priv; + struct virtnet_sq_free_stats stats = {}; + u64 kicks = 0; + int sent; + + __free_old_xmit(sq, true, &stats); + + sent = virtnet_xsk_xmit_batch(sq, pool, budget, &kicks); + + if (!is_xdp_raw_buffer_queue(vi, sq - vi->sq)) + check_sq_full_and_disable(vi, vi->dev, sq); + + u64_stats_update_begin(&sq->stats.syncp); + u64_stats_add(&sq->stats.packets, stats.packets); + u64_stats_add(&sq->stats.bytes, stats.bytes); + u64_stats_add(&sq->stats.kicks, kicks); + u64_stats_add(&sq->stats.xdp_tx, sent); + u64_stats_update_end(&sq->stats.syncp); + + if (xsk_uses_need_wakeup(pool)) + xsk_set_tx_need_wakeup(pool); + + return sent == budget; +} + static int __virtnet_xdp_xmit_one(struct virtnet_info *vi, struct send_queue *sq, struct xdp_frame *xdpf) @@ -2707,6 +2810,7 @@ static int virtnet_poll_tx(struct napi_struct *napi, int budget) struct virtnet_info *vi = sq->vq->vdev->priv; unsigned int index = vq2txq(sq->vq); struct netdev_queue *txq; + bool xsk_busy = false; int opaque; bool done; @@ -2719,7 +2823,11 @@ static int virtnet_poll_tx(struct napi_struct *napi, int budget) txq = netdev_get_tx_queue(vi->dev, index); __netif_tx_lock(txq, raw_smp_processor_id()); virtqueue_disable_cb(sq->vq); - free_old_xmit(sq, true); + + if (sq->xsk.pool) + xsk_busy = virtnet_xsk_xmit(sq, sq->xsk.pool, budget); + else + free_old_xmit(sq, true); if (sq->vq->num_free >= 2 + MAX_SKB_FRAGS) { if (netif_tx_queue_stopped(txq)) { @@ -2730,6 +2838,11 @@ static int virtnet_poll_tx(struct napi_struct *napi, int budget) netif_tx_wake_queue(txq); } + if (xsk_busy) { + __netif_tx_unlock(txq); + return budget; + } + opaque = virtqueue_enable_cb_prepare(sq->vq); done = napi_complete_done(napi, 0); @@ -5715,6 +5828,10 @@ static void virtnet_sq_free_unused_buf(struct virtqueue *vq, void *buf) case VIRTNET_XMIT_TYPE_DMA: virtnet_sq_unmap(sq, &buf); goto retry; + + case VIRTNET_XMIT_TYPE_XSK: + /* Make gcc happy. DONE in subsequent commit */ + break; } } From patchwork Fri Jun 14 06:39:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13697911 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-124.freemail.mail.aliyun.com (out30-124.freemail.mail.aliyun.com [115.124.30.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 70D3E149C4A; Fri, 14 Jun 2024 06:39:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718347189; cv=none; b=tnMjevjvUl7jVtpMmvT+LAjEm2oiS5ZdJOOQBBJHuoQEW1PH8tU0T9JL+n4J8qrcwzDjTE/bLMetG3H50FGLbY71UAshmec2z1y5LWvFCoURBDBNVZU3X286wkv2tPzfBz7z6hO89GDWgyHyYXjPNfSWz2OAvqoL9ZEBM0xpeXo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718347189; c=relaxed/simple; bh=JNOnnmOFu1NUWWjC8WtNGzYk18S62nPmUhl6YzbpQds=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=kOPijKG5H+HDin1JdBVE1oLQQIm8xo7WPhNJe+P1WKezXutdAxaPX0ftFc5cSPrvdruD2uoZQL4XGK4chT6IvfFeYpRCkD3Hq0ZGoPYeRpTi63WLmzt7CmC+pI6dSFAbKo4Oat0wXvz8b55Dse5VhqzkaT0aOvVT0jDew+3v6g0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=U5MshxJo; arc=none smtp.client-ip=115.124.30.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="U5MshxJo" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1718347185; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=AsxCsmPwJLPCPz8R6ARZ0TKBqaj/tWLFeD6Lswatkas=; b=U5MshxJotsTtaBkTcpVgNnGfM22mTO3cjRvrbxvS0Dv/NpUplIUBZzfSzSWtsJd9u0gV3HLfpnjmj7ygkRKZOHO8/YHMyu5CGUFdm0wi8q1mkpfgQSISooTTgr5FoYIgJFZ8ZQUCm50bh3ah1Qiv1uChQ6u6sUKyiT/L80Ik1cI= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R121e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033045075189;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=15;SR=0;TI=SMTPD_---0W8QKgVb_1718347184; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W8QKgVb_1718347184) by smtp.aliyun-inc.com; Fri, 14 Jun 2024 14:39:44 +0800 From: Xuan Zhuo To: netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , =?utf-8?q?Eugenio_P=C3=A9rez?= , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , virtualization@lists.linux.dev, bpf@vger.kernel.org Subject: [PATCH net-next v5 12/15] virtio_net: xsk: tx: support wakeup Date: Fri, 14 Jun 2024 14:39:30 +0800 Message-Id: <20240614063933.108811-13-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240614063933.108811-1-xuanzhuo@linux.alibaba.com> References: <20240614063933.108811-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: e008fb4a0943 X-Patchwork-Delegate: kuba@kernel.org xsk wakeup is used to trigger the logic for xsk xmit by xsk framework or user. Virtio-net does not support to actively generate an interruption, so it tries to trigger tx NAPI on the local cpu. Signed-off-by: Xuan Zhuo Acked-by: Jason Wang --- drivers/net/virtio_net.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index 7e811f392768..9bfccef18e27 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -1368,6 +1368,29 @@ static bool virtnet_xsk_xmit(struct send_queue *sq, struct xsk_buff_pool *pool, return sent == budget; } +static int virtnet_xsk_wakeup(struct net_device *dev, u32 qid, u32 flag) +{ + struct virtnet_info *vi = netdev_priv(dev); + struct send_queue *sq; + + if (!netif_running(dev)) + return -ENETDOWN; + + if (qid >= vi->curr_queue_pairs) + return -EINVAL; + + sq = &vi->sq[qid]; + + if (napi_if_scheduled_mark_missed(&sq->napi)) + return 0; + + local_bh_disable(); + virtqueue_napi_schedule(&sq->napi, sq->vq); + local_bh_enable(); + + return 0; +} + static int __virtnet_xdp_xmit_one(struct virtnet_info *vi, struct send_queue *sq, struct xdp_frame *xdpf) @@ -5706,6 +5729,7 @@ static const struct net_device_ops virtnet_netdev = { .ndo_vlan_rx_kill_vid = virtnet_vlan_rx_kill_vid, .ndo_bpf = virtnet_xdp, .ndo_xdp_xmit = virtnet_xdp_xmit, + .ndo_xsk_wakeup = virtnet_xsk_wakeup, .ndo_features_check = passthru_features_check, .ndo_get_phys_port_name = virtnet_get_phys_port_name, .ndo_set_features = virtnet_set_features, From patchwork Fri Jun 14 06:39:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13697913 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-100.freemail.mail.aliyun.com (out30-100.freemail.mail.aliyun.com [115.124.30.100]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2D6DE1836C7; Fri, 14 Jun 2024 06:39:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.100 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718347190; cv=none; b=n14+ArwYFVdJQIkj1H3u2TSyilHpqhHLc3DKqCuRnVWCeX6R9cn6U4uf45YAnDikIw5XfUnkPIw5KfeA/h8uv/WDM/XpiczZ6RL65twLaq8PM+7jNT/I4RMyiR77HuKglvzgNWIQRhQzzpWvlmV22b+4LrNENqMsnSwaYloEyyE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718347190; c=relaxed/simple; bh=mHbPA2yoaUoBHrdV0L+VpYOk3kHXG/SX7tMi2lycnyE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ZppV5261iSGXS6j+6eeF5onRE7lf2xKIOPTsufh3L6TxN/EehnRtwYWSPjIBXObVYo/p6LNrY+CZ+xaQhkJWtxKUQogJyBuz3bjTtVGfSaaBLnEUHOR3r2W46hZyGBOTARq1bt9E1yzNs/3oGFdEu+PigBLyAHLraPJ/8QOsR/I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=t7J2NZMt; arc=none smtp.client-ip=115.124.30.100 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="t7J2NZMt" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1718347186; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=PKxa5NOAvL+Dy50gDQCc3bEWPaH/ki6UgcOAl3k/kyg=; b=t7J2NZMt3QnFm9RMm2L/Yz9xMllqPEGvE783AtUyRtqfrrlLxYHLD/bZs00xX1NvBxe/qA3EnsUcuU6FER/cXS3hUFpgcfNnQgRg/FntBRQx0wT31SIb7kg2saLf9TXH5I+Pz9J6kyj4y3TksYmlGGroeiMuoxIymebAbGuUIsQ= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R851e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033037067113;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=15;SR=0;TI=SMTPD_---0W8Q97xn_1718347185; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W8Q97xn_1718347185) by smtp.aliyun-inc.com; Fri, 14 Jun 2024 14:39:45 +0800 From: Xuan Zhuo To: netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , =?utf-8?q?Eugenio_P=C3=A9rez?= , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , virtualization@lists.linux.dev, bpf@vger.kernel.org Subject: [PATCH net-next v5 13/15] virtio_net: xsk: tx: handle the transmitted xsk buffer Date: Fri, 14 Jun 2024 14:39:31 +0800 Message-Id: <20240614063933.108811-14-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240614063933.108811-1-xuanzhuo@linux.alibaba.com> References: <20240614063933.108811-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: e008fb4a0943 X-Patchwork-Delegate: kuba@kernel.org virtnet_free_old_xmit distinguishes three type ptr(skb, xdp frame, xsk buffer) by the last bits of the pointer. Signed-off-by: Xuan Zhuo --- drivers/net/virtio_net.c | 56 +++++++++++++++++++++++++++++++++------- 1 file changed, 46 insertions(+), 10 deletions(-) diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index 9bfccef18e27..d6be569bd849 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -87,6 +87,7 @@ struct virtnet_stat_desc { struct virtnet_sq_free_stats { u64 packets; u64 bytes; + u64 xsk; }; struct virtnet_sq_stats { @@ -530,6 +531,7 @@ struct virtio_net_common_hdr { }; static void virtnet_sq_free_unused_buf(struct virtqueue *vq, void *buf); +static void virtnet_xsk_completed(struct send_queue *sq, int num); enum virtnet_xmit_type { VIRTNET_XMIT_TYPE_SKB, @@ -733,6 +735,11 @@ static int virtnet_sq_set_premapped(struct send_queue *sq, bool premapped) return 0; } +static u32 virtnet_ptr_to_xsk(void *ptr) +{ + return ((unsigned long)ptr) >> VIRTIO_XSK_FLAG_OFFSET; +} + static void __free_old_xmit(struct send_queue *sq, bool in_napi, struct virtnet_sq_free_stats *stats) { @@ -773,12 +780,22 @@ static void __free_old_xmit(struct send_queue *sq, bool in_napi, goto retry; case VIRTNET_XMIT_TYPE_XSK: - /* Make gcc happy. DONE in subsequent commit */ + stats->bytes += virtnet_ptr_to_xsk(ptr); + stats->xsk++; break; } } } +static void virtnet_free_old_xmit(struct send_queue *sq, bool in_napi, + struct virtnet_sq_free_stats *stats) +{ + __free_old_xmit(sq, in_napi, stats); + + if (stats->xsk) + virtnet_xsk_completed(sq, stats->xsk); +} + /* Converting between virtqueue no. and kernel tx/rx queue no. * 0:rx0 1:tx0 2:rx1 3:tx1 ... 2N:rxN 2N+1:txN 2N+2:cvq */ @@ -1207,7 +1224,7 @@ static void free_old_xmit(struct send_queue *sq, bool in_napi) { struct virtnet_sq_free_stats stats = {0}; - __free_old_xmit(sq, in_napi, &stats); + virtnet_free_old_xmit(sq, in_napi, &stats); /* Avoid overhead when no packets have been processed * happens when called speculatively from start_xmit. @@ -1348,8 +1365,12 @@ static bool virtnet_xsk_xmit(struct send_queue *sq, struct xsk_buff_pool *pool, u64 kicks = 0; int sent; + /* Avoid to wakeup napi meanless, so call __free_old_xmit. */ __free_old_xmit(sq, true, &stats); + if (stats.xsk) + xsk_tx_completed(sq->xsk.pool, stats.xsk); + sent = virtnet_xsk_xmit_batch(sq, pool, budget, &kicks); if (!is_xdp_raw_buffer_queue(vi, sq - vi->sq)) @@ -1368,6 +1389,16 @@ static bool virtnet_xsk_xmit(struct send_queue *sq, struct xsk_buff_pool *pool, return sent == budget; } +static void xsk_wakeup(struct send_queue *sq) +{ + if (napi_if_scheduled_mark_missed(&sq->napi)) + return; + + local_bh_disable(); + virtqueue_napi_schedule(&sq->napi, sq->vq); + local_bh_enable(); +} + static int virtnet_xsk_wakeup(struct net_device *dev, u32 qid, u32 flag) { struct virtnet_info *vi = netdev_priv(dev); @@ -1381,14 +1412,19 @@ static int virtnet_xsk_wakeup(struct net_device *dev, u32 qid, u32 flag) sq = &vi->sq[qid]; - if (napi_if_scheduled_mark_missed(&sq->napi)) - return 0; + xsk_wakeup(sq); + return 0; +} - local_bh_disable(); - virtqueue_napi_schedule(&sq->napi, sq->vq); - local_bh_enable(); +static void virtnet_xsk_completed(struct send_queue *sq, int num) +{ + xsk_tx_completed(sq->xsk.pool, num); - return 0; + /* If this is called by rx poll, start_xmit and xdp xmit we should + * wakeup the tx napi to consume the xsk tx queue, because the tx + * interrupt may not be triggered. + */ + xsk_wakeup(sq); } static int __virtnet_xdp_xmit_one(struct virtnet_info *vi, @@ -1504,7 +1540,7 @@ static int virtnet_xdp_xmit(struct net_device *dev, } /* Free up any pending old buffers before queueing new ones. */ - __free_old_xmit(sq, false, &stats); + virtnet_free_old_xmit(sq, false, &stats); for (i = 0; i < n; i++) { struct xdp_frame *xdpf = frames[i]; @@ -5854,7 +5890,7 @@ static void virtnet_sq_free_unused_buf(struct virtqueue *vq, void *buf) goto retry; case VIRTNET_XMIT_TYPE_XSK: - /* Make gcc happy. DONE in subsequent commit */ + xsk_tx_completed(sq->xsk.pool, 1); break; } } From patchwork Fri Jun 14 06:39:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13697915 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-131.freemail.mail.aliyun.com (out30-131.freemail.mail.aliyun.com [115.124.30.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4372A1836FE; Fri, 14 Jun 2024 06:39:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.131 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718347191; cv=none; b=elgPZPYDLSKfVDzQ2MnskhfszdToYX/qR7yR/+yZZFTuC3lW+faLNWAe+3eE1pHNPmwVV/LBhM8bocqcKr2ldxTWKsZZXVEVt5JgozKd7u0SgOM6hlrh1PYdvMQoxyzctYKxqFaEqwgV53QDMpFNf3cjEV8Mv4htqHcgNEVOt48= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718347191; c=relaxed/simple; bh=9I01Lwl2TpiEZN5qEofkAVWb/D3UjnwYft8EofcPXv4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=rNdFAb2RDmvq3Q9yPR1D1GLKiVS2OfTFxBDxRBmCuLGGf8ICqxeXx1u5ECIUXIUJHL/GWiGp6+f3vptHNgpL4dmGX5wvfFUL9PHccGqyQP6gFJ8zb33bXzAxiTBRRaaO2NIoqkpRZ61BOKMIaDQkBv9GjgR/V0PUeGq1yrZEEwE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=HgyH5+d3; arc=none smtp.client-ip=115.124.30.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="HgyH5+d3" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1718347187; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=9Umv27qjMXQboC0xgAvDAY4yYsoXM/5r3TmBkvb6Qbg=; b=HgyH5+d3LSeUCaYnQx3mj2u+/utgti6CbdsJveJYj3wdWLF/qbUpAYn4RYRslJclCr7Hz8juokbWk9AZZYp+cRsU6OVSpnta0ptmYDDpueNOvQDKOdza9n/PlZh8YIcGRTwl8+hnFDhJUz5wdGJz+DcJHVCJ6HLSeTT8gAC9+yY= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R201e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033037067110;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=15;SR=0;TI=SMTPD_---0W8QKgWR_1718347185; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W8QKgWR_1718347185) by smtp.aliyun-inc.com; Fri, 14 Jun 2024 14:39:46 +0800 From: Xuan Zhuo To: netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , =?utf-8?q?Eugenio_P=C3=A9rez?= , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , virtualization@lists.linux.dev, bpf@vger.kernel.org Subject: [PATCH net-next v5 14/15] virtio_net: xsk: rx: support fill with xsk buffer Date: Fri, 14 Jun 2024 14:39:32 +0800 Message-Id: <20240614063933.108811-15-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240614063933.108811-1-xuanzhuo@linux.alibaba.com> References: <20240614063933.108811-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: e008fb4a0943 X-Patchwork-Delegate: kuba@kernel.org Implement the logic of filling rq with XSK buffers. Signed-off-by: Xuan Zhuo --- drivers/net/virtio_net.c | 64 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 62 insertions(+), 2 deletions(-) diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index d6be569bd849..4e5645d8bb7d 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -388,6 +388,8 @@ struct receive_queue { /* xdp rxq used by xsk */ struct xdp_rxq_info xdp_rxq; + + struct xdp_buff **xsk_buffs; } xsk; }; @@ -532,6 +534,8 @@ struct virtio_net_common_hdr { static void virtnet_sq_free_unused_buf(struct virtqueue *vq, void *buf); static void virtnet_xsk_completed(struct send_queue *sq, int num); +static int virtnet_add_recvbuf_xsk(struct virtnet_info *vi, struct receive_queue *rq, + struct xsk_buff_pool *pool, gfp_t gfp); enum virtnet_xmit_type { VIRTNET_XMIT_TYPE_SKB, @@ -1304,6 +1308,47 @@ static void sg_fill_dma(struct scatterlist *sg, dma_addr_t addr, u32 len) sg->length = len; } +static int virtnet_add_recvbuf_xsk(struct virtnet_info *vi, struct receive_queue *rq, + struct xsk_buff_pool *pool, gfp_t gfp) +{ + struct xdp_buff **xsk_buffs; + dma_addr_t addr; + u32 len, i; + int err = 0; + int num; + + xsk_buffs = rq->xsk.xsk_buffs; + + num = xsk_buff_alloc_batch(pool, xsk_buffs, rq->vq->num_free); + if (!num) + return -ENOMEM; + + len = xsk_pool_get_rx_frame_size(pool) + vi->hdr_len; + + for (i = 0; i < num; ++i) { + /* use the part of XDP_PACKET_HEADROOM as the virtnet hdr space */ + addr = xsk_buff_xdp_get_dma(xsk_buffs[i]) - vi->hdr_len; + + sg_init_table(rq->sg, 1); + sg_fill_dma(rq->sg, addr, len); + + err = virtqueue_add_inbuf(rq->vq, rq->sg, 1, xsk_buffs[i], gfp); + if (err) + goto err; + } + + return num; + +err: + if (i) + err = i; + + for (; i < num; ++i) + xsk_buff_free(xsk_buffs[i]); + + return err; +} + static int virtnet_xsk_xmit_one(struct send_queue *sq, struct xsk_buff_pool *pool, struct xdp_desc *desc) @@ -2560,6 +2605,11 @@ static bool try_fill_recv(struct virtnet_info *vi, struct receive_queue *rq, int err; bool oom; + if (rq->xsk.pool) { + err = virtnet_add_recvbuf_xsk(vi, rq, rq->xsk.pool, gfp); + goto kick; + } + do { if (vi->mergeable_rx_bufs) err = add_recvbuf_mergeable(vi, rq, gfp); @@ -2568,10 +2618,11 @@ static bool try_fill_recv(struct virtnet_info *vi, struct receive_queue *rq, else err = add_recvbuf_small(vi, rq, gfp); - oom = err == -ENOMEM; if (err) break; } while (rq->vq->num_free); + +kick: if (virtqueue_kick_prepare(rq->vq) && virtqueue_notify(rq->vq)) { unsigned long flags; @@ -2580,6 +2631,7 @@ static bool try_fill_recv(struct virtnet_info *vi, struct receive_queue *rq, u64_stats_update_end_irqrestore(&rq->stats.syncp, flags); } + oom = err == -ENOMEM; return !oom; } @@ -5449,7 +5501,7 @@ static int virtnet_xsk_pool_enable(struct net_device *dev, struct send_queue *sq; struct device *dma_dev; dma_addr_t hdr_dma; - int err; + int err, size; /* In big_packets mode, xdp cannot work, so there is no need to * initialize xsk of rq. @@ -5484,6 +5536,12 @@ static int virtnet_xsk_pool_enable(struct net_device *dev, if (!dma_dev) return -EPERM; + size = virtqueue_get_vring_size(rq->vq); + + rq->xsk.xsk_buffs = kvcalloc(size, sizeof(*rq->xsk.xsk_buffs), GFP_KERNEL); + if (!rq->xsk.xsk_buffs) + return -ENOMEM; + hdr_dma = dma_map_single(dma_dev, &xsk_hdr, vi->hdr_len, DMA_TO_DEVICE); if (dma_mapping_error(dma_dev, hdr_dma)) return -ENOMEM; @@ -5542,6 +5600,8 @@ static int virtnet_xsk_pool_disable(struct net_device *dev, u16 qid) dma_unmap_single(dma_dev, sq->xsk.hdr_dma_address, vi->hdr_len, DMA_TO_DEVICE); + kvfree(rq->xsk.xsk_buffs); + return err1 | err2; } From patchwork Fri Jun 14 06:39:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13697916 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-100.freemail.mail.aliyun.com (out30-100.freemail.mail.aliyun.com [115.124.30.100]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 38816184117; Fri, 14 Jun 2024 06:39:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.100 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718347192; cv=none; b=ozvlDr8CHTpecnSjPCyV1rOLAVSOl94RTiYc1q7TibyRK5QTsoAcTZEbwTM0KxSWS63P0YutwoBi+pOuGTTV32oV5g/X9HtzAeg/VIILdW8yWxyJ/ed4AchsYM6t+aXLg+St/00x3Q6NIKGpMlqsrwGMf7qB3z5LEQCqeolGrYE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718347192; c=relaxed/simple; bh=4hzviNY8F32a7Sfcg+5g/D/fPH7GcUSteBD1xcyb9u8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=OWpWojLqHmW8ivciQwKMBHsuvyUua83li9jJDjxBWYlHJexJc5GKDIXmkkaMwBRUEKAtg1YZxwY1sIgYZLYOuwLyFZIB4lA+TVt3Wqz7F2j+2THa41PquVuDkpPw12qPF/R9daSH/tDCdx5h4WYivJmdrf/W2bYhilaV5XleA+g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=iLmsb4TP; arc=none smtp.client-ip=115.124.30.100 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="iLmsb4TP" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1718347188; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=wprsRKvJKPiHy2/z6PRklFNL9mVFn/gr0qILRFB0hmw=; b=iLmsb4TPRCcHxqWES/9pvPM4FUxYGZ2y5Nsd0E636VAGJpV7guRNYBnQSBZDxdeuKjDq3deUvMWwrGM90eEu1ZeN7KsM4jGbg4Ys458mo73mVU0gher6ZjsFQa2WvROPc3sc961Mu1Dbo+a60EIsw/WKg16jI3IQQew9k0xNGA8= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R551e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033068173054;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=15;SR=0;TI=SMTPD_---0W8Q97yj_1718347186; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W8Q97yj_1718347186) by smtp.aliyun-inc.com; Fri, 14 Jun 2024 14:39:47 +0800 From: Xuan Zhuo To: netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , =?utf-8?q?Eugenio_P=C3=A9rez?= , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , virtualization@lists.linux.dev, bpf@vger.kernel.org Subject: [PATCH net-next v5 15/15] virtio_net: xsk: rx: support recv small mode Date: Fri, 14 Jun 2024 14:39:33 +0800 Message-Id: <20240614063933.108811-16-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240614063933.108811-1-xuanzhuo@linux.alibaba.com> References: <20240614063933.108811-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: e008fb4a0943 X-Patchwork-Delegate: kuba@kernel.org The virtnet_xdp_handler() is re-used. But 1. We need to copy data to create skb for XDP_PASS. 2. We need to call xsk_buff_free() to release the buffer. 3. The handle for xdp_buff is difference. If we pushed this logic into existing receive handle(merge and small), we would have to maintain code scattered inside merge and small (and big). So I think it is a good choice for us to put the xsk code into an independent function. Signed-off-by: Xuan Zhuo --- drivers/net/virtio_net.c | 142 +++++++++++++++++++++++++++++++++++++-- 1 file changed, 138 insertions(+), 4 deletions(-) diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index 4e5645d8bb7d..72c4d2f0c0ea 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -534,8 +534,10 @@ struct virtio_net_common_hdr { static void virtnet_sq_free_unused_buf(struct virtqueue *vq, void *buf); static void virtnet_xsk_completed(struct send_queue *sq, int num); -static int virtnet_add_recvbuf_xsk(struct virtnet_info *vi, struct receive_queue *rq, - struct xsk_buff_pool *pool, gfp_t gfp); +static int virtnet_xdp_handler(struct bpf_prog *xdp_prog, struct xdp_buff *xdp, + struct net_device *dev, + unsigned int *xdp_xmit, + struct virtnet_rq_stats *stats); enum virtnet_xmit_type { VIRTNET_XMIT_TYPE_SKB, @@ -1218,6 +1220,11 @@ static void virtnet_rq_unmap_free_buf(struct virtqueue *vq, void *buf) rq = &vi->rq[i]; + if (rq->xsk.pool) { + xsk_buff_free((struct xdp_buff *)buf); + return; + } + if (!vi->big_packets || vi->mergeable_rx_bufs) virtnet_rq_unmap(rq, buf, 0); @@ -1308,6 +1315,120 @@ static void sg_fill_dma(struct scatterlist *sg, dma_addr_t addr, u32 len) sg->length = len; } +static struct xdp_buff *buf_to_xdp(struct virtnet_info *vi, + struct receive_queue *rq, void *buf, u32 len) +{ + struct xdp_buff *xdp; + u32 bufsize; + + xdp = (struct xdp_buff *)buf; + + bufsize = xsk_pool_get_rx_frame_size(rq->xsk.pool) + vi->hdr_len; + + if (unlikely(len > bufsize)) { + pr_debug("%s: rx error: len %u exceeds truesize %u\n", + vi->dev->name, len, bufsize); + DEV_STATS_INC(vi->dev, rx_length_errors); + xsk_buff_free(xdp); + return NULL; + } + + xsk_buff_set_size(xdp, len); + xsk_buff_dma_sync_for_cpu(xdp); + + return xdp; +} + +static struct sk_buff *xdp_construct_skb(struct receive_queue *rq, + struct xdp_buff *xdp) +{ + unsigned int metasize = xdp->data - xdp->data_meta; + struct sk_buff *skb; + unsigned int size; + + size = xdp->data_end - xdp->data_hard_start; + skb = napi_alloc_skb(&rq->napi, size); + if (unlikely(!skb)) { + xsk_buff_free(xdp); + return NULL; + } + + skb_reserve(skb, xdp->data_meta - xdp->data_hard_start); + + size = xdp->data_end - xdp->data_meta; + memcpy(__skb_put(skb, size), xdp->data_meta, size); + + if (metasize) { + __skb_pull(skb, metasize); + skb_metadata_set(skb, metasize); + } + + xsk_buff_free(xdp); + + return skb; +} + +static struct sk_buff *virtnet_receive_xsk_small(struct net_device *dev, struct virtnet_info *vi, + struct receive_queue *rq, struct xdp_buff *xdp, + unsigned int *xdp_xmit, + struct virtnet_rq_stats *stats) +{ + struct bpf_prog *prog; + u32 ret; + + ret = XDP_PASS; + rcu_read_lock(); + prog = rcu_dereference(rq->xdp_prog); + if (prog) + ret = virtnet_xdp_handler(prog, xdp, dev, xdp_xmit, stats); + rcu_read_unlock(); + + switch (ret) { + case XDP_PASS: + return xdp_construct_skb(rq, xdp); + + case XDP_TX: + case XDP_REDIRECT: + return NULL; + + default: + /* drop packet */ + xsk_buff_free(xdp); + u64_stats_inc(&stats->drops); + return NULL; + } +} + +static struct sk_buff *virtnet_receive_xsk_buf(struct virtnet_info *vi, struct receive_queue *rq, + void *buf, u32 len, + unsigned int *xdp_xmit, + struct virtnet_rq_stats *stats) +{ + struct net_device *dev = vi->dev; + struct sk_buff *skb = NULL; + struct xdp_buff *xdp; + + len -= vi->hdr_len; + + u64_stats_add(&stats->bytes, len); + + xdp = buf_to_xdp(vi, rq, buf, len); + if (!xdp) + return NULL; + + if (unlikely(len < ETH_HLEN)) { + pr_debug("%s: short packet %i\n", dev->name, len); + DEV_STATS_INC(dev, rx_length_errors); + xsk_buff_free(xdp); + return NULL; + } + + if (!vi->mergeable_rx_bufs) + skb = virtnet_receive_xsk_small(dev, vi, rq, xdp, xdp_xmit, stats); + + return skb; +} + static int virtnet_add_recvbuf_xsk(struct virtnet_info *vi, struct receive_queue *rq, struct xsk_buff_pool *pool, gfp_t gfp) { @@ -2713,9 +2834,22 @@ static int virtnet_receive(struct receive_queue *rq, int budget, void *buf; int i; - if (!vi->big_packets || vi->mergeable_rx_bufs) { - void *ctx; + if (rq->xsk.pool) { + struct sk_buff *skb; + + while (packets < budget) { + buf = virtqueue_get_buf(rq->vq, &len); + if (!buf) + break; + skb = virtnet_receive_xsk_buf(vi, rq, buf, len, xdp_xmit, &stats); + if (skb) + virtnet_receive_done(vi, rq, skb); + + packets++; + } + } else if (!vi->big_packets || vi->mergeable_rx_bufs) { + void *ctx; while (packets < budget && (buf = virtnet_rq_get_buf(rq, &len, &ctx))) { receive_buf(vi, rq, buf, len, ctx, xdp_xmit, &stats);