[vhost,v5,00/10] virtio: drivers maintain dma info for premapped vq

Message

Xuan Zhuo March 25, 2024, 8:54 a.m. UTC

As discussed:

http://lore.kernel.org/all/[e-mail]

If the virtio is premapped mode, the driver should manage the dma info by self.
So the virtio core should not store the dma info. We can release the memory used
to store the dma info.

For virtio-net xmit queue, if the virtio-net maintains the dma info,
the virtio-net must allocate too much memory(19 * queue_size for per-queue), so
we do not plan to make the virtio-net to maintain the dma info by default. The
virtio-net xmit queue only maintain the dma info when premapped mode is enable
(such as AF_XDP is enable).

So this patch set try to do:

1. make the virtio core to do not store the dma info when driver can do that
    - But if the desc_extra has not dma info, we face a new question,
      it is hard to get the dma info of the desc with indirect flag.
      For split mode, that is easy from desc, but for the packed mode,
      it is hard to get the dma info from the desc. And hardening
      the dma unmap is safe, we should store the dma info of indirect
      descs when the virtio core does not store the bufer dma info.

      The follow patches to this:
         * virtio_ring: packed: structure the indirect desc table
         * virtio_ring: split: structure the indirect desc table

    - On the other side, in the umap handle, we mix the indirect descs with
      other descs. That make things too complex. I found if we we distinguish
      the descs with VRING_DESC_F_INDIRECT before unmap, thing will be clearer.

      The follow patches do this.
         * virtio_ring: packed: remove double check of the unmap ops
         * virtio_ring: split: structure the indirect desc table

2. make the virtio core to enable premapped mode by find_vqs() params
    - Because the find_vqs() will try to allocate memory for the dma info.
      If we set the premapped mode after find_vqs() and release the
      dma info, that is odd.


Please review.

Thanks

v5:
    1. use the existing structure to replace vring_split_desc_indir

v4:
    1. virtio-net xmit queue does not enable premapped mode by default

v3:
    1. fix the conflict with the vp_modern_create_avq().

v2:
    1. change the dma item of virtio-net, every item have MAX_SKB_FRAGS + 2 addr + len pairs.
    2. introduce virtnet_sq_free_stats for __free_old_xmit

v1:
    1. rename transport_vq_config to vq_transport_config
    2. virtio-net set dma meta number to (ring-size + 1)(MAX_SKB_FRGAS +2)
    3. introduce virtqueue_dma_map_sg_attrs
    4. separate vring_create_virtqueue to an independent commit


Xuan Zhuo (10):
  virtio_ring: introduce vring_need_unmap_buffer
  virtio_ring: packed: remove double check of the unmap ops
  virtio_ring: packed: structure the indirect desc table
  virtio_ring: split: remove double check of the unmap ops
  virtio_ring: split: structure the indirect desc table
  virtio_ring: no store dma info when unmap is not needed
  virtio: find_vqs: add new parameter premapped
  virtio_ring: export premapped to driver by struct virtqueue
  virtio_net: set premapped mode by find_vqs()
  virtio_ring: virtqueue_set_dma_premapped support disable

 drivers/net/virtio_net.c      |  57 +++--
 drivers/virtio/virtio_ring.c  | 430 +++++++++++++++++++++-------------
 include/linux/virtio.h        |   3 +-
 include/linux/virtio_config.h |  17 +-
 4 files changed, 301 insertions(+), 206 deletions(-)

--
2.32.0.3.g01195cf9f

Comments

Jason Wang March 26, 2024, 6:35 a.m. UTC | #1

On Mon, Mar 25, 2024 at 4:54 PM Xuan Zhuo <[e-mail]> wrote:
>
> As discussed:
>
> http://lore.kernel.org/all/[e-mail]
>
> If the virtio is premapped mode, the driver should manage the dma info by self.
> So the virtio core should not store the dma info. We can release the memory used
> to store the dma info.
>
> For virtio-net xmit queue, if the virtio-net maintains the dma info,
> the virtio-net must allocate too much memory(19 * queue_size for per-queue), so
> we do not plan to make the virtio-net to maintain the dma info by default. The
> virtio-net xmit queue only maintain the dma info when premapped mode is enable
> (such as AF_XDP is enable).
>
> So this patch set try to do:
>
> 1. make the virtio core to do not store the dma info when driver can do that
>     - But if the desc_extra has not dma info, we face a new question,
>       it is hard to get the dma info of the desc with indirect flag.
>       For split mode, that is easy from desc, but for the packed mode,
>       it is hard to get the dma info from the desc. And hardening
>       the dma unmap is safe, we should store the dma info of indirect
>       descs when the virtio core does not store the bufer dma info.
>
>       The follow patches to this:
>          * virtio_ring: packed: structure the indirect desc table
>          * virtio_ring: split: structure the indirect desc table
>
>     - On the other side, in the umap handle, we mix the indirect descs with
>       other descs. That make things too complex. I found if we we distinguish
>       the descs with VRING_DESC_F_INDIRECT before unmap, thing will be clearer.
>
>       The follow patches do this.
>          * virtio_ring: packed: remove double check of the unmap ops
>          * virtio_ring: split: structure the indirect desc table
>
> 2. make the virtio core to enable premapped mode by find_vqs() params
>     - Because the find_vqs() will try to allocate memory for the dma info.
>       If we set the premapped mode after find_vqs() and release the
>       dma info, that is odd.
>
>
> Please review.
>
> Thanks

This doesn't apply cleany on vhost.git linux-next branch.

Which tree is this based on?

Thanks

Xuan Zhuo March 27, 2024, 7:14 a.m. UTC | #2

On Tue, 26 Mar 2024 14:35:21 +0800, Jason Wang <[e-mail]> wrote:
> On Mon, Mar 25, 2024 at 4:54 PM Xuan Zhuo <[e-mail]> wrote:
> >
> > As discussed:
> >
> > http://lore.kernel.org/all/[e-mail]
> >
> > If the virtio is premapped mode, the driver should manage the dma info by self.
> > So the virtio core should not store the dma info. We can release the memory used
> > to store the dma info.
> >
> > For virtio-net xmit queue, if the virtio-net maintains the dma info,
> > the virtio-net must allocate too much memory(19 * queue_size for per-queue), so
> > we do not plan to make the virtio-net to maintain the dma info by default. The
> > virtio-net xmit queue only maintain the dma info when premapped mode is enable
> > (such as AF_XDP is enable).
> >
> > So this patch set try to do:
> >
> > 1. make the virtio core to do not store the dma info when driver can do that
> >     - But if the desc_extra has not dma info, we face a new question,
> >       it is hard to get the dma info of the desc with indirect flag.
> >       For split mode, that is easy from desc, but for the packed mode,
> >       it is hard to get the dma info from the desc. And hardening
> >       the dma unmap is safe, we should store the dma info of indirect
> >       descs when the virtio core does not store the bufer dma info.
> >
> >       The follow patches to this:
> >          * virtio_ring: packed: structure the indirect desc table
> >          * virtio_ring: split: structure the indirect desc table
> >
> >     - On the other side, in the umap handle, we mix the indirect descs with
> >       other descs. That make things too complex. I found if we we distinguish
> >       the descs with VRING_DESC_F_INDIRECT before unmap, thing will be clearer.
> >
> >       The follow patches do this.
> >          * virtio_ring: packed: remove double check of the unmap ops
> >          * virtio_ring: split: structure the indirect desc table
> >
> > 2. make the virtio core to enable premapped mode by find_vqs() params
> >     - Because the find_vqs() will try to allocate memory for the dma info.
> >       If we set the premapped mode after find_vqs() and release the
> >       dma info, that is odd.
> >
> >
> > Please review.
> >
> > Thanks
>
> This doesn't apply cleany on vhost.git linux-next branch.
>
> Which tree is this based on?


Sorry. That is on the top of "[PATCH vhost v5 0/6] refactor the params of
find_vqs()".

Lore-URL: http://lore.kernel.org/all/[e-mail]

Thanks.

>
> Thanks
>

Jason Wang March 27, 2024, 7:50 a.m. UTC | #3

On Wed, Mar 27, 2024 at 3:16 PM Xuan Zhuo <[e-mail]> wrote:
>
> On Tue, 26 Mar 2024 14:35:21 +0800, Jason Wang <[e-mail]> wrote:
> > On Mon, Mar 25, 2024 at 4:54 PM Xuan Zhuo <[e-mail]> wrote:
> > >
> > > As discussed:
> > >
> > > http://lore.kernel.org/all/[e-mail]
> > >
> > > If the virtio is premapped mode, the driver should manage the dma info by self.
> > > So the virtio core should not store the dma info. We can release the memory used
> > > to store the dma info.
> > >
> > > For virtio-net xmit queue, if the virtio-net maintains the dma info,
> > > the virtio-net must allocate too much memory(19 * queue_size for per-queue), so
> > > we do not plan to make the virtio-net to maintain the dma info by default. The
> > > virtio-net xmit queue only maintain the dma info when premapped mode is enable
> > > (such as AF_XDP is enable).
> > >
> > > So this patch set try to do:
> > >
> > > 1. make the virtio core to do not store the dma info when driver can do that
> > >     - But if the desc_extra has not dma info, we face a new question,
> > >       it is hard to get the dma info of the desc with indirect flag.
> > >       For split mode, that is easy from desc, but for the packed mode,
> > >       it is hard to get the dma info from the desc. And hardening
> > >       the dma unmap is safe, we should store the dma info of indirect
> > >       descs when the virtio core does not store the bufer dma info.
> > >
> > >       The follow patches to this:
> > >          * virtio_ring: packed: structure the indirect desc table
> > >          * virtio_ring: split: structure the indirect desc table
> > >
> > >     - On the other side, in the umap handle, we mix the indirect descs with
> > >       other descs. That make things too complex. I found if we we distinguish
> > >       the descs with VRING_DESC_F_INDIRECT before unmap, thing will be clearer.
> > >
> > >       The follow patches do this.
> > >          * virtio_ring: packed: remove double check of the unmap ops
> > >          * virtio_ring: split: structure the indirect desc table
> > >
> > > 2. make the virtio core to enable premapped mode by find_vqs() params
> > >     - Because the find_vqs() will try to allocate memory for the dma info.
> > >       If we set the premapped mode after find_vqs() and release the
> > >       dma info, that is odd.
> > >
> > >
> > > Please review.
> > >
> > > Thanks
> >
> > This doesn't apply cleany on vhost.git linux-next branch.
> >
> > Which tree is this based on?
>
>
> Sorry. That is on the top of "[PATCH vhost v5 0/6] refactor the params of
> find_vqs()".
>
> Lore-URL: http://lore.kernel.org/all/[e-mail]
>
> Thanks.

I've tried that but it doesn't work:

% git am ~/Downloads/\[PATCH\ vhost\ v5\ 01_10\]\ virtio_ring_\
introduce\ vring_need_unmap_buffer.eml
Applying: virtio_ring: introduce vring_need_unmap_buffer
error: patch failed: drivers/virtio/virtio_ring.c:2080
error: drivers/virtio/virtio_ring.c: patch does not apply
Patch failed at 0001 virtio_ring: introduce vring_need_unmap_buffer
hint: Use 'git am --show-current-patch=diff' to see the failed patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

I'm using vhost.git linux-next branch, HEAD is

commit 56e71885b0349241c07631a7b979b61e81afab6a
Author: Maxime Coquelin <[e-mail]>
Date:   Tue Jan 9 12:10:24 2024 +0100

    vduse: Temporarily fail if control queue feature requested

Thanks

>
> >
> > Thanks
> >
>

Xuan Zhuo March 27, 2024, 8:04 a.m. UTC | #4

On Wed, 27 Mar 2024 15:50:17 +0800, Jason Wang <[e-mail]> wrote:
> On Wed, Mar 27, 2024 at 3:16 PM Xuan Zhuo <[e-mail]> wrote:
> >
> > On Tue, 26 Mar 2024 14:35:21 +0800, Jason Wang <[e-mail]> wrote:
> > > On Mon, Mar 25, 2024 at 4:54 PM Xuan Zhuo <[e-mail]> wrote:
> > > >
> > > > As discussed:
> > > >
> > > > http://lore.kernel.org/all/[e-mail]
> > > >
> > > > If the virtio is premapped mode, the driver should manage the dma info by self.
> > > > So the virtio core should not store the dma info. We can release the memory used
> > > > to store the dma info.
> > > >
> > > > For virtio-net xmit queue, if the virtio-net maintains the dma info,
> > > > the virtio-net must allocate too much memory(19 * queue_size for per-queue), so
> > > > we do not plan to make the virtio-net to maintain the dma info by default. The
> > > > virtio-net xmit queue only maintain the dma info when premapped mode is enable
> > > > (such as AF_XDP is enable).
> > > >
> > > > So this patch set try to do:
> > > >
> > > > 1. make the virtio core to do not store the dma info when driver can do that
> > > >     - But if the desc_extra has not dma info, we face a new question,
> > > >       it is hard to get the dma info of the desc with indirect flag.
> > > >       For split mode, that is easy from desc, but for the packed mode,
> > > >       it is hard to get the dma info from the desc. And hardening
> > > >       the dma unmap is safe, we should store the dma info of indirect
> > > >       descs when the virtio core does not store the bufer dma info.
> > > >
> > > >       The follow patches to this:
> > > >          * virtio_ring: packed: structure the indirect desc table
> > > >          * virtio_ring: split: structure the indirect desc table
> > > >
> > > >     - On the other side, in the umap handle, we mix the indirect descs with
> > > >       other descs. That make things too complex. I found if we we distinguish
> > > >       the descs with VRING_DESC_F_INDIRECT before unmap, thing will be clearer.
> > > >
> > > >       The follow patches do this.
> > > >          * virtio_ring: packed: remove double check of the unmap ops
> > > >          * virtio_ring: split: structure the indirect desc table
> > > >
> > > > 2. make the virtio core to enable premapped mode by find_vqs() params
> > > >     - Because the find_vqs() will try to allocate memory for the dma info.
> > > >       If we set the premapped mode after find_vqs() and release the
> > > >       dma info, that is odd.
> > > >
> > > >
> > > > Please review.
> > > >
> > > > Thanks
> > >
> > > This doesn't apply cleany on vhost.git linux-next branch.
> > >
> > > Which tree is this based on?
> >
> >
> > Sorry. That is on the top of "[PATCH vhost v5 0/6] refactor the params of
> > find_vqs()".
> >
> > Lore-URL: http://lore.kernel.org/all/[e-mail]
> >
> > Thanks.
>
> I've tried that but it doesn't work:
>
> % git am ~/Downloads/\[PATCH\ vhost\ v5\ 01_10\]\ virtio_ring_\
> introduce\ vring_need_unmap_buffer.eml
> Applying: virtio_ring: introduce vring_need_unmap_buffer
> error: patch failed: drivers/virtio/virtio_ring.c:2080
> error: drivers/virtio/virtio_ring.c: patch does not apply
> Patch failed at 0001 virtio_ring: introduce vring_need_unmap_buffer
> hint: Use 'git am --show-current-patch=diff' to see the failed patch
> When you have resolved this problem, run "git am --continue".
> If you prefer to skip this patch, run "git am --skip" instead.
> To restore the original branch and stop patching, run "git am --abort".
>
> I'm using vhost.git linux-next branch, HEAD is
>
> commit 56e71885b0349241c07631a7b979b61e81afab6a
> Author: Maxime Coquelin <[e-mail]>
> Date:   Tue Jan 9 12:10:24 2024 +0100
>
>     vduse: Temporarily fail if control queue feature requested


NOT ON the vhost directly.

That is on the top of "refactor the params of find_vqs"

"refactor the params of find_vqs" said:

	"""
	This pathset is splited from the

	     http://lore.kernel.org/all/[e-mail]

	That may needs some cycles to discuss. But that notifies too many people.
	"""

But now that is broken due to the change of the that patch set.

I will post the new version of these two patch set soon.

Thanks.


>
> Thanks
>
> >
> > >
> > > Thanks
> > >
> >
>

Jason Wang March 27, 2024, 8:47 a.m. UTC | #5

On Wed, Mar 27, 2024 at 4:07 PM Xuan Zhuo <[e-mail]> wrote:
>
> On Wed, 27 Mar 2024 15:50:17 +0800, Jason Wang <[e-mail]> wrote:
> > On Wed, Mar 27, 2024 at 3:16 PM Xuan Zhuo <[e-mail]> wrote:
> > >
> > > On Tue, 26 Mar 2024 14:35:21 +0800, Jason Wang <[e-mail]> wrote:
> > > > On Mon, Mar 25, 2024 at 4:54 PM Xuan Zhuo <[e-mail]> wrote:
> > > > >
> > > > > As discussed:
> > > > >
> > > > > http://lore.kernel.org/all/[e-mail]
> > > > >
> > > > > If the virtio is premapped mode, the driver should manage the dma info by self.
> > > > > So the virtio core should not store the dma info. We can release the memory used
> > > > > to store the dma info.
> > > > >
> > > > > For virtio-net xmit queue, if the virtio-net maintains the dma info,
> > > > > the virtio-net must allocate too much memory(19 * queue_size for per-queue), so
> > > > > we do not plan to make the virtio-net to maintain the dma info by default. The
> > > > > virtio-net xmit queue only maintain the dma info when premapped mode is enable
> > > > > (such as AF_XDP is enable).
> > > > >
> > > > > So this patch set try to do:
> > > > >
> > > > > 1. make the virtio core to do not store the dma info when driver can do that
> > > > >     - But if the desc_extra has not dma info, we face a new question,
> > > > >       it is hard to get the dma info of the desc with indirect flag.
> > > > >       For split mode, that is easy from desc, but for the packed mode,
> > > > >       it is hard to get the dma info from the desc. And hardening
> > > > >       the dma unmap is safe, we should store the dma info of indirect
> > > > >       descs when the virtio core does not store the bufer dma info.
> > > > >
> > > > >       The follow patches to this:
> > > > >          * virtio_ring: packed: structure the indirect desc table
> > > > >          * virtio_ring: split: structure the indirect desc table
> > > > >
> > > > >     - On the other side, in the umap handle, we mix the indirect descs with
> > > > >       other descs. That make things too complex. I found if we we distinguish
> > > > >       the descs with VRING_DESC_F_INDIRECT before unmap, thing will be clearer.
> > > > >
> > > > >       The follow patches do this.
> > > > >          * virtio_ring: packed: remove double check of the unmap ops
> > > > >          * virtio_ring: split: structure the indirect desc table
> > > > >
> > > > > 2. make the virtio core to enable premapped mode by find_vqs() params
> > > > >     - Because the find_vqs() will try to allocate memory for the dma info.
> > > > >       If we set the premapped mode after find_vqs() and release the
> > > > >       dma info, that is odd.
> > > > >
> > > > >
> > > > > Please review.
> > > > >
> > > > > Thanks
> > > >
> > > > This doesn't apply cleany on vhost.git linux-next branch.
> > > >
> > > > Which tree is this based on?
> > >
> > >
> > > Sorry. That is on the top of "[PATCH vhost v5 0/6] refactor the params of
> > > find_vqs()".
> > >
> > > Lore-URL: http://lore.kernel.org/all/[e-mail]
> > >
> > > Thanks.
> >
> > I've tried that but it doesn't work:
> >
> > % git am ~/Downloads/\[PATCH\ vhost\ v5\ 01_10\]\ virtio_ring_\
> > introduce\ vring_need_unmap_buffer.eml
> > Applying: virtio_ring: introduce vring_need_unmap_buffer
> > error: patch failed: drivers/virtio/virtio_ring.c:2080
> > error: drivers/virtio/virtio_ring.c: patch does not apply
> > Patch failed at 0001 virtio_ring: introduce vring_need_unmap_buffer
> > hint: Use 'git am --show-current-patch=diff' to see the failed patch
> > When you have resolved this problem, run "git am --continue".
> > If you prefer to skip this patch, run "git am --skip" instead.
> > To restore the original branch and stop patching, run "git am --abort".
> >
> > I'm using vhost.git linux-next branch, HEAD is
> >
> > commit 56e71885b0349241c07631a7b979b61e81afab6a
> > Author: Maxime Coquelin <[e-mail]>
> > Date:   Tue Jan 9 12:10:24 2024 +0100
> >
> >     vduse: Temporarily fail if control queue feature requested
>
>
> NOT ON the vhost directly.
>
> That is on the top of "refactor the params of find_vqs"

I meant I just did it on top of this series.

>
> "refactor the params of find_vqs" said:
>
>         """
>         This pathset is splited from the
>
>              http://lore.kernel.org/all/[e-mail]
>
>         That may needs some cycles to discuss. But that notifies too many people.
>         """
>
> But now that is broken due to the change of the that patch set.
>
> I will post the new version of these two patch set soon.

Ok.

Thanks

>
> Thanks.
>
>
> >
> > Thanks
> >
> > >
> > > >
> > > > Thanks
> > > >
> > >
> >
>