| Message ID | 2ac35e2c-26a8-6f6d-2236-c4692600db9e@sberdevices.ru (mailing list archive) |
|---|---|
| Headers | show |
| Series | vsock: updates for SO_RCVLOWAT handling | expand |
Hi Arseniy, sorry but I didn't have time to review this series. I will definitely do it next Monday! Have a nice weekend, Stefano On Wed, Aug 03, 2022 at 01:48:06PM +0000, Arseniy Krasnov wrote: >Hello, > >This patchset includes some updates for SO_RCVLOWAT: > >1) af_vsock: > During my experiments with zerocopy receive, i found, that in some > cases, poll() implementation violates POSIX: when socket has non- > default SO_RCVLOWAT(e.g. not 1), poll() will always set POLLIN and > POLLRDNORM bits in 'revents' even number of bytes available to read > on socket is smaller than SO_RCVLOWAT value. In this case,user sees > POLLIN flag and then tries to read data(for example using 'read()' > call), but read call will be blocked, because SO_RCVLOWAT logic is > supported in dequeue loop in af_vsock.c. But the same time, POSIX > requires that: > > "POLLIN Data other than high-priority data may be read without > blocking. > POLLRDNORM Normal data may be read without blocking." > > See https://www.open-std.org/jtc1/sc22/open/n4217.pdf, page 293. > > So, we have, that poll() syscall returns POLLIN, but read call will > be blocked. > > Also in man page socket(7) i found that: > > "Since Linux 2.6.28, select(2), poll(2), and epoll(7) indicate a > socket as readable only if at least SO_RCVLOWAT bytes are available." > > I checked TCP callback for poll()(net/ipv4/tcp.c, tcp_poll()), it > uses SO_RCVLOWAT value to set POLLIN bit, also i've tested TCP with > this case for TCP socket, it works as POSIX required. > > I've added some fixes to af_vsock.c and virtio_transport_common.c, > test is also implemented. > >2) virtio/vsock: > It adds some optimization to wake ups, when new data arrived. Now, > SO_RCVLOWAT is considered before wake up sleepers who wait new data. > There is no sense, to kick waiter, when number of available bytes > in socket's queue < SO_RCVLOWAT, because if we wake up reader in > this case, it will wait for SO_RCVLOWAT data anyway during dequeue, > or in poll() case, POLLIN/POLLRDNORM bits won't be set, so such > exit from poll() will be "spurious". This logic is also used in TCP > sockets. > >3) vmci/vsock: > Same as 2), but i'm not sure about this changes. Will be very good, > to get comments from someone who knows this code. > >4) Hyper-V: > As Dexuan Cui mentioned, for Hyper-V transport it is difficult to > support SO_RCVLOWAT, so he suggested to disable this feature for > Hyper-V. > >Thank You > >Arseniy Krasnov(9): > vsock: SO_RCVLOWAT transport set callback > hv_sock: disable SO_RCVLOWAT support > virtio/vsock: use 'target' in notify_poll_in callback > vmci/vsock: use 'target' in notify_poll_in callback > vsock: pass sock_rcvlowat to notify_poll_in as target > vsock: add API call for data ready > virtio/vsock: check SO_RCVLOWAT before wake up reader > vmci/vsock: check SO_RCVLOWAT before wake up reader > vsock_test: POLLIN + SO_RCVLOWAT test > > include/net/af_vsock.h | 2 + > net/vmw_vsock/af_vsock.c | 38 +++++++++- > net/vmw_vsock/hyperv_transport.c | 7 ++ > net/vmw_vsock/virtio_transport_common.c | 7 +- > net/vmw_vsock/vmci_transport_notify.c | 10 +-- > net/vmw_vsock/vmci_transport_notify_qstate.c | 12 +-- > tools/testing/vsock/vsock_test.c | 107 +++++++++++++++++++++++++++ > 7 files changed, 166 insertions(+), 17 deletions(-) > > Changelog: > > v1 -> v2: > 1) Patches for VMCI transport(same as for virtio-vsock). > 2) Patches for Hyper-V transport(disabling SO_RCVLOWAT setting). > 3) Waiting logic in test was updated(sleep() -> poll()). > > v2 -> v3: > 1) Patches were reordered. > 2) Commit message updated in 0005. > 3) Check 'transport' pointer in 0001 for NULL. > 4) Check 'value' in 0001 for > buffer_size. > >-- >2.25.1
On 05.08.2022 19:05, Stefano Garzarella wrote: > Hi Arseniy, > sorry but I didn't have time to review this series. I will definitely do it next Monday! > > Have a nice weekend, > Stefano Hello, no problem Thank You > > On Wed, Aug 03, 2022 at 01:48:06PM +0000, Arseniy Krasnov wrote: >> Hello, >> >> This patchset includes some updates for SO_RCVLOWAT: >> >> 1) af_vsock: >> During my experiments with zerocopy receive, i found, that in some >> cases, poll() implementation violates POSIX: when socket has non- >> default SO_RCVLOWAT(e.g. not 1), poll() will always set POLLIN and >> POLLRDNORM bits in 'revents' even number of bytes available to read >> on socket is smaller than SO_RCVLOWAT value. In this case,user sees >> POLLIN flag and then tries to read data(for example using 'read()' >> call), but read call will be blocked, because SO_RCVLOWAT logic is >> supported in dequeue loop in af_vsock.c. But the same time, POSIX >> requires that: >> >> "POLLIN Data other than high-priority data may be read without >> blocking. >> POLLRDNORM Normal data may be read without blocking." >> >> See https://www.open-std.org/jtc1/sc22/open/n4217.pdf, page 293. >> >> So, we have, that poll() syscall returns POLLIN, but read call will >> be blocked. >> >> Also in man page socket(7) i found that: >> >> "Since Linux 2.6.28, select(2), poll(2), and epoll(7) indicate a >> socket as readable only if at least SO_RCVLOWAT bytes are available." >> >> I checked TCP callback for poll()(net/ipv4/tcp.c, tcp_poll()), it >> uses SO_RCVLOWAT value to set POLLIN bit, also i've tested TCP with >> this case for TCP socket, it works as POSIX required. >> >> I've added some fixes to af_vsock.c and virtio_transport_common.c, >> test is also implemented. >> >> 2) virtio/vsock: >> It adds some optimization to wake ups, when new data arrived. Now, >> SO_RCVLOWAT is considered before wake up sleepers who wait new data. >> There is no sense, to kick waiter, when number of available bytes >> in socket's queue < SO_RCVLOWAT, because if we wake up reader in >> this case, it will wait for SO_RCVLOWAT data anyway during dequeue, >> or in poll() case, POLLIN/POLLRDNORM bits won't be set, so such >> exit from poll() will be "spurious". This logic is also used in TCP >> sockets. >> >> 3) vmci/vsock: >> Same as 2), but i'm not sure about this changes. Will be very good, >> to get comments from someone who knows this code. >> >> 4) Hyper-V: >> As Dexuan Cui mentioned, for Hyper-V transport it is difficult to >> support SO_RCVLOWAT, so he suggested to disable this feature for >> Hyper-V. >> >> Thank You >> >> Arseniy Krasnov(9): >> vsock: SO_RCVLOWAT transport set callback >> hv_sock: disable SO_RCVLOWAT support >> virtio/vsock: use 'target' in notify_poll_in callback >> vmci/vsock: use 'target' in notify_poll_in callback >> vsock: pass sock_rcvlowat to notify_poll_in as target >> vsock: add API call for data ready >> virtio/vsock: check SO_RCVLOWAT before wake up reader >> vmci/vsock: check SO_RCVLOWAT before wake up reader >> vsock_test: POLLIN + SO_RCVLOWAT test >> >> include/net/af_vsock.h | 2 + >> net/vmw_vsock/af_vsock.c | 38 +++++++++- >> net/vmw_vsock/hyperv_transport.c | 7 ++ >> net/vmw_vsock/virtio_transport_common.c | 7 +- >> net/vmw_vsock/vmci_transport_notify.c | 10 +-- >> net/vmw_vsock/vmci_transport_notify_qstate.c | 12 +-- >> tools/testing/vsock/vsock_test.c | 107 +++++++++++++++++++++++++++ >> 7 files changed, 166 insertions(+), 17 deletions(-) >> >> Changelog: >> >> v1 -> v2: >> 1) Patches for VMCI transport(same as for virtio-vsock). >> 2) Patches for Hyper-V transport(disabling SO_RCVLOWAT setting). >> 3) Waiting logic in test was updated(sleep() -> poll()). >> >> v2 -> v3: >> 1) Patches were reordered. >> 2) Commit message updated in 0005. >> 3) Check 'transport' pointer in 0001 for NULL. >> 4) Check 'value' in 0001 for > buffer_size. >> >> -- >> 2.25.1 >
Hi Arseniy, On Wed, Aug 03, 2022 at 01:48:06PM +0000, Arseniy Krasnov wrote: >Hello, > >This patchset includes some updates for SO_RCVLOWAT: I have reviewed all the patches, run tests and everything seems okay :-) I left some minor comments and asked Bryan and Vishnu to take a better look at VMCI patches. In general I ask you to revisit the patch descriptions a bit (for example adding a space after punctuation). The next version I think can be without RFC. Remember to send it with the net-next tag. Note: net-next is closed for now since we are in the merge window. It should re-open in a week (you can check here: http://vger.kernel.org/~davem/net-next.html). I'll be on vacation the next 2 weeks (Aug 15 - 28), but I'll try to check out your patches! Thanks, Stefano
On 08.08.2022 14:22, Stefano Garzarella wrote: > Hi Arseniy, > > On Wed, Aug 03, 2022 at 01:48:06PM +0000, Arseniy Krasnov wrote: >> Hello, >> >> This patchset includes some updates for SO_RCVLOWAT: > > I have reviewed all the patches, run tests and everything seems okay :-) Thank You > > I left some minor comments and asked Bryan and Vishnu to take a better look at VMCI patches. Ok, let's wait their reply before v4 > > In general I ask you to revisit the patch descriptions a bit (for example adding a space after punctuation). The next version I think can be without RFC. ack > > Remember to send it with the net-next tag. > Note: net-next is closed for now since we are in the merge window. > It should re-open in a week (you can check here: http://vger.kernel.org/~davem/net-next.html). > > I'll be on vacation the next 2 weeks (Aug 15 - 28), but I'll try to check out your patches! No problem, thanks > > Thanks, > Stefano >
Hello, This patchset includes some updates for SO_RCVLOWAT: 1) af_vsock: During my experiments with zerocopy receive, i found, that in some cases, poll() implementation violates POSIX: when socket has non- default SO_RCVLOWAT(e.g. not 1), poll() will always set POLLIN and POLLRDNORM bits in 'revents' even number of bytes available to read on socket is smaller than SO_RCVLOWAT value. In this case,user sees POLLIN flag and then tries to read data(for example using 'read()' call), but read call will be blocked, because SO_RCVLOWAT logic is supported in dequeue loop in af_vsock.c. But the same time, POSIX requires that: "POLLIN Data other than high-priority data may be read without blocking. POLLRDNORM Normal data may be read without blocking." See https://www.open-std.org/jtc1/sc22/open/n4217.pdf, page 293. So, we have, that poll() syscall returns POLLIN, but read call will be blocked. Also in man page socket(7) i found that: "Since Linux 2.6.28, select(2), poll(2), and epoll(7) indicate a socket as readable only if at least SO_RCVLOWAT bytes are available." I checked TCP callback for poll()(net/ipv4/tcp.c, tcp_poll()), it uses SO_RCVLOWAT value to set POLLIN bit, also i've tested TCP with this case for TCP socket, it works as POSIX required. I've added some fixes to af_vsock.c and virtio_transport_common.c, test is also implemented. 2) virtio/vsock: It adds some optimization to wake ups, when new data arrived. Now, SO_RCVLOWAT is considered before wake up sleepers who wait new data. There is no sense, to kick waiter, when number of available bytes in socket's queue < SO_RCVLOWAT, because if we wake up reader in this case, it will wait for SO_RCVLOWAT data anyway during dequeue, or in poll() case, POLLIN/POLLRDNORM bits won't be set, so such exit from poll() will be "spurious". This logic is also used in TCP sockets. 3) vmci/vsock: Same as 2), but i'm not sure about this changes. Will be very good, to get comments from someone who knows this code. 4) Hyper-V: As Dexuan Cui mentioned, for Hyper-V transport it is difficult to support SO_RCVLOWAT, so he suggested to disable this feature for Hyper-V. Thank You Arseniy Krasnov(9): vsock: SO_RCVLOWAT transport set callback hv_sock: disable SO_RCVLOWAT support virtio/vsock: use 'target' in notify_poll_in callback vmci/vsock: use 'target' in notify_poll_in callback vsock: pass sock_rcvlowat to notify_poll_in as target vsock: add API call for data ready virtio/vsock: check SO_RCVLOWAT before wake up reader vmci/vsock: check SO_RCVLOWAT before wake up reader vsock_test: POLLIN + SO_RCVLOWAT test include/net/af_vsock.h | 2 + net/vmw_vsock/af_vsock.c | 38 +++++++++- net/vmw_vsock/hyperv_transport.c | 7 ++ net/vmw_vsock/virtio_transport_common.c | 7 +- net/vmw_vsock/vmci_transport_notify.c | 10 +-- net/vmw_vsock/vmci_transport_notify_qstate.c | 12 +-- tools/testing/vsock/vsock_test.c | 107 +++++++++++++++++++++++++++ 7 files changed, 166 insertions(+), 17 deletions(-) Changelog: v1 -> v2: 1) Patches for VMCI transport(same as for virtio-vsock). 2) Patches for Hyper-V transport(disabling SO_RCVLOWAT setting). 3) Waiting logic in test was updated(sleep() -> poll()). v2 -> v3: 1) Patches were reordered. 2) Commit message updated in 0005. 3) Check 'transport' pointer in 0001 for NULL. 4) Check 'value' in 0001 for > buffer_size.