[net-next,1/4] netlink: truncate overlength attribute list in nla_nest_end()

Commit Message

Edwin Peer Jan. 23, 2021, 4:53 a.m. UTC

If a nested list of attributes is too long, then the length will
exceed the 16-bit nla_len of the parent nlattr. In such cases,
determine how many whole attributes can fit and truncate the
message to this length. This properly maintains the nesting
hierarchy, keeping the entire message valid, while fitting more
subelements inside the nest range than may result if the length
is wrapped modulo 64KB.

Marking truncated attributes, such that user space can determine
the precise attribute truncated, by means of an additional bit in
the nla_type was considered and rejected. The NLA_F_NESTED and
NLA_F_NET_BYTEORDER flags are supposed to be mutually exclusive.
So, in theory, the latter bit could have been redefined for nested
attributes in order to indicate truncation, but user space tools
(most notably iproute2) cannot be relied on to honor NLA_TYPE_MASK,
resulting in alteration of the perceived nla_type and subsequent
catastrophic failure.

Failing the entire message with a hard error must also be rejected,
as this would break existing user space functionality. The trigger
issue is evident for IFLA_VFINFO_LIST and a hard error here would
cause iproute2 to fail to render an entire interface list even if
only a single interface warranted a truncated VF list. Instead, set
NLM_F_NEST_TRUNCATED in the netlink header to inform user space
about the incomplete data. In this particular case, however, user
space can better ascertain which instance is truncated by consulting
the associated IFLA_NUM_VF to determine how many VFs were expected.

Fixes: bfa83a9e03cf ("[NETLINK]: Type-safe netlink messages/attributes interface")
Signed-off-by: Edwin Peer <[e-mail]>
---
 include/net/netlink.h        | 11 +++++++++--
 include/uapi/linux/netlink.h |  1 +
 lib/nlattr.c                 | 27 +++++++++++++++++++++++++++
 3 files changed, 37 insertions(+), 2 deletions(-)

Comments

David Ahern Jan. 23, 2021, 7:14 p.m. UTC | #1

On 1/22/21 9:53 PM, Edwin Peer wrote:
> If a nested list of attributes is too long, then the length will
> exceed the 16-bit nla_len of the parent nlattr. In such cases,
> determine how many whole attributes can fit and truncate the
> message to this length. This properly maintains the nesting
> hierarchy, keeping the entire message valid, while fitting more
> subelements inside the nest range than may result if the length
> is wrapped modulo 64KB.
> 
> Marking truncated attributes, such that user space can determine
> the precise attribute truncated, by means of an additional bit in
> the nla_type was considered and rejected. The NLA_F_NESTED and
> NLA_F_NET_BYTEORDER flags are supposed to be mutually exclusive.
> So, in theory, the latter bit could have been redefined for nested
> attributes in order to indicate truncation, but user space tools
> (most notably iproute2) cannot be relied on to honor NLA_TYPE_MASK,
> resulting in alteration of the perceived nla_type and subsequent
> catastrophic failure.
> 

Did you look at using NETLINK_CB / netlink_skb_parms to keep a running
length of nested attributes to avoid the need to trim?

Edwin Peer Jan. 23, 2021, 8:42 p.m. UTC | #2

On Sat, Jan 23, 2021 at 11:14 AM David Ahern <[e-mail]> wrote:

> > Marking truncated attributes, such that user space can determine
> > the precise attribute truncated, by means of an additional bit in
> > the nla_type was considered and rejected. The NLA_F_NESTED and
> > NLA_F_NET_BYTEORDER flags are supposed to be mutually exclusive.
> > So, in theory, the latter bit could have been redefined for nested
> > attributes in order to indicate truncation, but user space tools
> > (most notably iproute2) cannot be relied on to honor NLA_TYPE_MASK,
> > resulting in alteration of the perceived nla_type and subsequent
> > catastrophic failure.
>
> Did you look at using NETLINK_CB / netlink_skb_parms to keep a running
> length of nested attributes to avoid the need to trim?

I did not, but thinking about it now, I don't think that's necessarily
the way to go. We shouldn't be concerned about the cost of iterating
over the list and trimming the skb for what should be a rare exception
path. Ideally, we want to make sure at compile time (by having correct
code) that we don't ever exceed this limit at run time. Perhaps we
should investigate static analysis approaches to prove nla_len can't
be exceeded?

Tracking the outer nest length during nla_put() would provide for
convenient error indication at the precise location where things go
wrong, but that's a fair amount of housekeeping that isn't free of
complexity and run time cost either. Instead of rarely (if ever)
undoing work, we'll always do extra work that we hardly ever need.

Then, if nla_put() can detect nesting errors, there's the issue of
what to do in the case of errors. Case in point, the IFLA_VFINFO_LIST
scenario would now require explicit error handling in the generator
logic, because we can't fail hard at that point. We would need to be
sure we propagate all possible nesting errors up to a common location
(probably where the nest ends, which is where we're dealing with the
problem in this solution), set the truncated flag and carry on (for
the same net effect the trim in nla_nest_end() has). If there are
other cases we don't know about today, they might turn from soft fails
into hard errors, breaking existing user space. Truncating the list is
the only non-obtrusive solution to any existing brokenness that is
guaranteed to not make things worse, but we can't know where we need
to do that apriori and would need to explicitly handle each case as
they come up.

Hard errors on nest overflow can only reliably work for new code. That
is, assuming it is tested to the extremes when it goes in, not after
user space comes to rely on the broken behavior.

Regards,
Edwin Peer

Edwin Peer Jan. 23, 2021, 9:03 p.m. UTC | #3

On Sat, Jan 23, 2021 at 12:42 PM Edwin Peer <[e-mail]> wrote:

> Then, if nla_put() can detect nesting errors, there's the issue of
> what to do in the case of errors. Case in point, the IFLA_VFINFO_LIST
> scenario would now require explicit error handling in the generator
> logic, because we can't fail hard at that point. We would need to be
> sure we propagate all possible nesting errors up to a common location
> (probably where the nest ends, which is where we're dealing with the
> problem in this solution), set the truncated flag and carry on (for
> the same net effect the trim in nla_nest_end() has).

Also, the unwind here turns out to be just as complicated, requiring a
traversal from the start and a trim anyway, because the attribute that
triggers the overflow may be deep inside the hierarchy. We can't
simply truncate at this point. We should remove whole elements at the
uppermost level, or risk having partially filled attribute trees
rather than missing attributes at the level of the exceeded nest -
which may be worse.

Regards,
Edwin Peer

Jakub Kicinski Jan. 26, 2021, 1:43 a.m. UTC | #4

On Fri, 22 Jan 2021 20:53:18 -0800 Edwin Peer wrote:
> If a nested list of attributes is too long, then the length will
> exceed the 16-bit nla_len of the parent nlattr. In such cases,
> determine how many whole attributes can fit and truncate the
> message to this length. This properly maintains the nesting
> hierarchy, keeping the entire message valid, while fitting more
> subelements inside the nest range than may result if the length
> is wrapped modulo 64KB.
> 
> Marking truncated attributes, such that user space can determine
> the precise attribute truncated, by means of an additional bit in
> the nla_type was considered and rejected. The NLA_F_NESTED and
> NLA_F_NET_BYTEORDER flags are supposed to be mutually exclusive.
> So, in theory, the latter bit could have been redefined for nested
> attributes in order to indicate truncation, but user space tools
> (most notably iproute2) cannot be relied on to honor NLA_TYPE_MASK,
> resulting in alteration of the perceived nla_type and subsequent
> catastrophic failure.
> 
> Failing the entire message with a hard error must also be rejected,
> as this would break existing user space functionality. The trigger
> issue is evident for IFLA_VFINFO_LIST and a hard error here would
> cause iproute2 to fail to render an entire interface list even if
> only a single interface warranted a truncated VF list. Instead, set
> NLM_F_NEST_TRUNCATED in the netlink header to inform user space
> about the incomplete data. In this particular case, however, user
> space can better ascertain which instance is truncated by consulting
> the associated IFLA_NUM_VF to determine how many VFs were expected.
> 
> Fixes: bfa83a9e03cf ("[NETLINK]: Type-safe netlink messages/attributes interface")
> Signed-off-by: Edwin Peer <[e-mail]>


> diff --git a/include/net/netlink.h b/include/net/netlink.h
> index 1ceec518ab49..fc8c57dafb05 100644
> --- a/include/net/netlink.h
> +++ b/include/net/netlink.h
> @@ -1785,19 +1785,26 @@ static inline struct nlattr *nla_nest_start(struct sk_buff *skb, int attrtype)
>  	return nla_nest_start_noflag(skb, attrtype | NLA_F_NESTED);
>  }
>  
> +int __nla_nest_trunc_msg(struct sk_buff *skb, const struct nlattr *start);
> +
>  /**
>   * nla_nest_end - Finalize nesting of attributes
>   * @skb: socket buffer the attributes are stored in
>   * @start: container attribute
>   *
>   * Corrects the container attribute header to include the all
> - * appeneded attributes.
> + * appeneded attributes. The list of attributes will be truncated
> + * if too long to fit within the parent attribute's maximum reach.
>   *
>   * Returns the total data length of the skb.
>   */
>  static inline int nla_nest_end(struct sk_buff *skb, struct nlattr *start)

What are the semantics for multiple nests? All attrs down will always
have the trunc flag set? Because I'd have expected the skb_tail to be
trimmed in __nla_nest_trunc_msg()..

In fact if there is another nest around the "full" one, and the "full"
one is close to 0xffff wouldn't that end up cascading all the way down
to the outermost attr yielding an empty top level attr? I feel like we
cater to a specific use case here where that doesn't happen.

After initially being concerned about using up another flag, I warmed
up to the concept of NLM_F_NEST_TRUNCATED, but I think this is all
better driven by the writer. The writer knows the size and count of the
attrs because it sizes the skb. So can the writer not drive the
truncation? Add a "how much space do we have left" check in
rtnl_fill_vf() ?

This would also avoid the assumption that the contents of the nla are
other nlas.

>  {
> -	start->nla_len = skb_tail_pointer(skb) - (unsigned char *)start;
> +	int len = skb_tail_pointer(skb) - (unsigned char *)start;
> +
> +	if (len > 0xffff)
> +		len = __nla_nest_trunc_msg(skb, start);
> +	start->nla_len = len;
>  	return skb->len;

This function really needs to start returning an error on overflow.
Perhaps even with a WARN(). 

Could you please check how many callers we'd need to change to have
nla_nest_end() have the same semantics as nla_put() (i.e. 0 or -errno)?
On a quick scan I see that most cases only care about errors already.

>  }
>  
> diff --git a/include/uapi/linux/netlink.h b/include/uapi/linux/netlink.h
> index 3d94269bbfa8..44a250825c30 100644
> --- a/include/uapi/linux/netlink.h
> +++ b/include/uapi/linux/netlink.h
> @@ -57,6 +57,7 @@ struct nlmsghdr {
>  #define NLM_F_ECHO		0x08	/* Echo this request 		*/
>  #define NLM_F_DUMP_INTR		0x10	/* Dump was inconsistent due to sequence change */
>  #define NLM_F_DUMP_FILTERED	0x20	/* Dump was filtered as requested */
> +#define NLM_F_NEST_TRUNCATED	0x40	/* Message contains truncated nested attribute */
>  
>  /* Modifiers to GET request */
>  #define NLM_F_ROOT	0x100	/* specify tree	root	*/
> diff --git a/lib/nlattr.c b/lib/nlattr.c
> index 5b6116e81f9f..2a267c0d3e16 100644
> --- a/lib/nlattr.c
> +++ b/lib/nlattr.c
> @@ -1119,4 +1119,31 @@ int nla_append(struct sk_buff *skb, int attrlen, const void *data)
>  	return 0;
>  }
>  EXPORT_SYMBOL(nla_append);
> +
> +/**
> + * __nla_nest_trunc_msg - Truncate list of nested netlink attributes to max len
> + * @skb: socket buffer with tail pointer positioned after end of nested list
> + * @start: container attribute designating the beginning of the list
> + *
> + * Trims the skb to fit only the attributes which are within the range of the
> + * containing nest attribute. This is a helper for nla_nest_end, to prevent
> + * adding unduly to the length of what is an inline function. It is not
> + * intended to be called from anywhere else.
> + *
> + * Returns the truncated length of the enclosing nest attribute in accordance
> + * with the number of whole attributes that can fit.
> + */
> +int __nla_nest_trunc_msg(struct sk_buff *skb, const struct nlattr *start)
> +{
> +	struct nlattr *attr = nla_data(start);
> +	int rem = 0xffff - NLA_HDRLEN;
> +
> +	while (nla_ok(attr, rem))
> +		attr = nla_next(attr, &rem);
> +	nlmsg_trim(skb, attr);
> +	nlmsg_hdr(skb)->nlmsg_flags |= NLM_F_NEST_TRUNCATED;
> +	return (unsigned char *)attr - (unsigned char *)start;
> +}
> +EXPORT_SYMBOL(__nla_nest_trunc_msg);
> +
>  #endif

David Ahern Jan. 26, 2021, 4:50 a.m. UTC | #5

On 1/23/21 1:42 PM, Edwin Peer wrote:
> On Sat, Jan 23, 2021 at 11:14 AM David Ahern <[e-mail]> wrote:
> 
>>> Marking truncated attributes, such that user space can determine
>>> the precise attribute truncated, by means of an additional bit in
>>> the nla_type was considered and rejected. The NLA_F_NESTED and
>>> NLA_F_NET_BYTEORDER flags are supposed to be mutually exclusive.
>>> So, in theory, the latter bit could have been redefined for nested
>>> attributes in order to indicate truncation, but user space tools
>>> (most notably iproute2) cannot be relied on to honor NLA_TYPE_MASK,
>>> resulting in alteration of the perceived nla_type and subsequent
>>> catastrophic failure.
>>
>> Did you look at using NETLINK_CB / netlink_skb_parms to keep a running
>> length of nested attributes to avoid the need to trim?
> 
> I did not, but thinking about it now, I don't think that's necessarily
> the way to go. We shouldn't be concerned about the cost of iterating
> over the list and trimming the skb for what should be a rare exception
> path. Ideally, we want to make sure at compile time (by having correct
> code) that we don't ever exceed this limit at run time. Perhaps we
> should investigate static analysis approaches to prove nla_len can't
> be exceeded?

It's not a rare exception path if VF data is dumped on every ip link
dump request. I think you would be surprised how often random s/w does a
link dump.

As for the static analysis, the number of VFs is dynamic so impossible
to detect at compile time. Limiting the number of VFs to what can fit
would be a different kind of regression.

> 
> Tracking the outer nest length during nla_put() would provide for
> convenient error indication at the precise location where things go
> wrong, but that's a fair amount of housekeeping that isn't free of
> complexity and run time cost either. Instead of rarely (if ever)
> undoing work, we'll always do extra work that we hardly ever need.
> 
> Then, if nla_put() can detect nesting errors, there's the issue of
> what to do in the case of errors. Case in point, the IFLA_VFINFO_LIST
> scenario would now require explicit error handling in the generator
> logic, because we can't fail hard at that point. We would need to be
> sure we propagate all possible nesting errors up to a common location
> (probably where the nest ends, which is where we're dealing with the
> problem in this solution), set the truncated flag and carry on (for
> the same net effect the trim in nla_nest_end() has). If there are
> other cases we don't know about today, they might turn from soft fails
> into hard errors, breaking existing user space. Truncating the list is
> the only non-obtrusive solution to any existing brokenness that is
> guaranteed to not make things worse, but we can't know where we need
> to do that apriori and would need to explicitly handle each case as
> they come up.

Yes, tracking the errors is hard given the flow of netlink helpers, but
ultimately all of the space checks come down to:

	if (unlikely(skb_tailroom(skb) < nla_total_size(attrlen)))
		return -EMSGSIZE;

so surely there is a solution. There are only a few entry points to
track space available (open coded checks deserve their fate) which can
be used to track overflow, nesting and multiple layers of nesting to
then avoid the memcpy in __nla_put, __nla_put_64bit, and __nla_put_nohdr

> 
> Hard errors on nest overflow can only reliably work for new code. That
> is, assuming it is tested to the extremes when it goes in, not after
> user space comes to rely on the broken behavior.
> 
> Regards,
> Edwin Peer
>

David Ahern Jan. 26, 2021, 4:56 a.m. UTC | #6

On 1/23/21 2:03 PM, Edwin Peer wrote:
> On Sat, Jan 23, 2021 at 12:42 PM Edwin Peer <[e-mail]> wrote:
> 
>> Then, if nla_put() can detect nesting errors, there's the issue of
>> what to do in the case of errors. Case in point, the IFLA_VFINFO_LIST
>> scenario would now require explicit error handling in the generator
>> logic, because we can't fail hard at that point. We would need to be
>> sure we propagate all possible nesting errors up to a common location
>> (probably where the nest ends, which is where we're dealing with the
>> problem in this solution), set the truncated flag and carry on (for
>> the same net effect the trim in nla_nest_end() has).
> 
> Also, the unwind here turns out to be just as complicated, requiring a
> traversal from the start and a trim anyway, because the attribute that
> triggers the overflow may be deep inside the hierarchy. We can't
> simply truncate at this point. We should remove whole elements at the
> uppermost level, or risk having partially filled attribute trees
> rather than missing attributes at the level of the exceeded nest -
> which may be worse.
> 

I'm not a fan of the skb trim idea. I think it would be better to figure
out how to stop adding to the skb when an attr length is going to exceed
64kB. Not failing hard with an error (ip link sh needs to succeed), but
truncating the specific attribute of a message with a flag so userspace
knows it is short.

Edwin Peer Jan. 26, 2021, 5:51 p.m. UTC | #7

On Mon, Jan 25, 2021 at 8:56 PM David Ahern <[e-mail]> wrote:

> I'm not a fan of the skb trim idea. I think it would be better to figure
> out how to stop adding to the skb when an attr length is going to exceed
> 64kB. Not failing hard with an error (ip link sh needs to succeed), but
> truncating the specific attribute of a message with a flag so userspace
> knows it is short.

Absent the ability to do something useful in terms of actually
avoiding the overflow [1], I'm abandoning this approach entirely. I
have a different idea that I will propose in due course.

[1] https://marc.info/?l=linux-netdev&m=161163943811663

Regards,
Edwin Peer

Gal Pressman June 5, 2023, 7:28 a.m. UTC | #8

On 26/01/2021 19:51, Edwin Peer wrote:
> On Mon, Jan 25, 2021 at 8:56 PM David Ahern <[e-mail]> wrote:
> 
>> I'm not a fan of the skb trim idea. I think it would be better to figure
>> out how to stop adding to the skb when an attr length is going to exceed
>> 64kB. Not failing hard with an error (ip link sh needs to succeed), but
>> truncating the specific attribute of a message with a flag so userspace
>> knows it is short.
> 
> Absent the ability to do something useful in terms of actually
> avoiding the overflow [1], I'm abandoning this approach entirely. I
> have a different idea that I will propose in due course.
> 
> [1] https://marc.info/?l=linux-netdev&m=161163943811663
> 
> Regards,
> Edwin Peer

Hello Edwin,

I'm also interested in getting this issue resolved, have you had any
progress since this series? Are you still working on it?

Jakub Kicinski June 5, 2023, 6:58 p.m. UTC | #9

[Updating Edwin's email.]

On Mon, 5 Jun 2023 10:28:06 +0300 Gal Pressman wrote:
> On 26/01/2021 19:51, Edwin Peer wrote:
> > On Mon, Jan 25, 2021 at 8:56 PM David Ahern <[e-mail]> wrote:
> >   
> >> I'm not a fan of the skb trim idea. I think it would be better to figure
> >> out how to stop adding to the skb when an attr length is going to exceed
> >> 64kB. Not failing hard with an error (ip link sh needs to succeed), but
> >> truncating the specific attribute of a message with a flag so userspace
> >> knows it is short.  
> > 
> > Absent the ability to do something useful in terms of actually
> > avoiding the overflow [1], I'm abandoning this approach entirely. I
> > have a different idea that I will propose in due course.
> > 
> > [1] https://marc.info/?l=linux-netdev&m=161163943811663
> > 
> > Regards,
> > Edwin Peer  
> 
> Hello Edwin,
> 
> I'm also interested in getting this issue resolved, have you had any
> progress since this series? Are you still working on it?

Edwin Peer June 5, 2023, 7:27 p.m. UTC | #10

On Mon, Jun 5, 2023 at 11:58 AM Jakub Kicinski <[e-mail]> wrote:
>
> [Updating Edwin's email.]
>
> On Mon, 5 Jun 2023 10:28:06 +0300 Gal Pressman wrote:
> > On 26/01/2021 19:51, Edwin Peer wrote:
> > > On Mon, Jan 25, 2021 at 8:56 PM David Ahern <[e-mail]> wrote:
> > >
> > >> I'm not a fan of the skb trim idea. I think it would be better to figure
> > >> out how to stop adding to the skb when an attr length is going to exceed
> > >> 64kB. Not failing hard with an error (ip link sh needs to succeed), but
> > >> truncating the specific attribute of a message with a flag so userspace
> > >> knows it is short.
> > >
> > > Absent the ability to do something useful in terms of actually
> > > avoiding the overflow [1], I'm abandoning this approach entirely. I
> > > have a different idea that I will propose in due course.
> > >
> > > [1] https://marc.info/?l=linux-netdev&m=161163943811663
> > >
> > > Regards,
> > > Edwin Peer
> >
> > Hello Edwin,
> >
> > I'm also interested in getting this issue resolved, have you had any
> > progress since this series? Are you still working on it?

Hi Kuba,

Thanks for the CC, I left Broadcom quite some time ago and am no
longer subscribed to netdev as a result (been living in firmware land
doing work in Rust).

I have no immediate plans to pick this up, at least not in the short
to medium term. My work in progress was on the laptop I returned and I
cannot immediately recall what solution I had in mind here.

Regards,
Edwin Peer

Gal Pressman June 6, 2023, 8:01 a.m. UTC | #11

On 05/06/2023 22:27, Edwin Peer wrote:
> On Mon, Jun 5, 2023 at 11:58 AM Jakub Kicinski <[e-mail]> wrote:
>>
>> [Updating Edwin's email.]
>>
>> On Mon, 5 Jun 2023 10:28:06 +0300 Gal Pressman wrote:
>>> On 26/01/2021 19:51, Edwin Peer wrote:
>>>> On Mon, Jan 25, 2021 at 8:56 PM David Ahern <[e-mail]> wrote:
>>>>
>>>>> I'm not a fan of the skb trim idea. I think it would be better to figure
>>>>> out how to stop adding to the skb when an attr length is going to exceed
>>>>> 64kB. Not failing hard with an error (ip link sh needs to succeed), but
>>>>> truncating the specific attribute of a message with a flag so userspace
>>>>> knows it is short.
>>>>
>>>> Absent the ability to do something useful in terms of actually
>>>> avoiding the overflow [1], I'm abandoning this approach entirely. I
>>>> have a different idea that I will propose in due course.
>>>>
>>>> [1] https://marc.info/?l=linux-netdev&m=161163943811663
>>>>
>>>> Regards,
>>>> Edwin Peer
>>>
>>> Hello Edwin,
>>>
>>> I'm also interested in getting this issue resolved, have you had any
>>> progress since this series? Are you still working on it?
> 
> Hi Kuba,
> 
> Thanks for the CC, I left Broadcom quite some time ago and am no
> longer subscribed to netdev as a result (been living in firmware land
> doing work in Rust).
> 
> I have no immediate plans to pick this up, at least not in the short
> to medium term. My work in progress was on the laptop I returned and I
> cannot immediately recall what solution I had in mind here.
> 
> Regards,
> Edwin Peer

Jakub, sorry if this has been discussed already in the past, but can you
please clarify what is an accepted (or more importantly, not accepted)
solution for this issue? I'm not familiar with the history and don't
want to repeat previous mistakes.

So far I've seen discussions about increasing the recv buffer size, and
this patchset which changes the GETLINK ABI, both of which were nacked.

Having 'ip link show' broken is very unfortunate :\, how should one
approach this issue in 2023?

Jakub Kicinski June 6, 2023, 4:17 p.m. UTC | #12

On Tue, 6 Jun 2023 11:01:14 +0300 Gal Pressman wrote:
> On 05/06/2023 22:27, Edwin Peer wrote:
> > Thanks for the CC, I left Broadcom quite some time ago and am no
> > longer subscribed to netdev as a result (been living in firmware land
> > doing work in Rust).
> > 
> > I have no immediate plans to pick this up, at least not in the short
> > to medium term. My work in progress was on the laptop I returned and I
> > cannot immediately recall what solution I had in mind here.
> 
> Jakub, sorry if this has been discussed already in the past, but can you
> please clarify what is an accepted (or more importantly, not accepted)
> solution for this issue? I'm not familiar with the history and don't
> want to repeat previous mistakes.

The problem is basically that attributes can only be 64kB and 
the legacy SR-IOV API wraps all the link info in an attribute.

> So far I've seen discussions about increasing the recv buffer size, and
> this patchset which changes the GETLINK ABI, both of which were nacked.

Filtering out some of the info, like the stats, is okay, but that just
increases the limit. A limit still exists.

> Having 'ip link show' broken is very unfortunate :\, how should one
> approach this issue in 2023?

Sure is, which is why we should be moving away from the legacy SR-IOV
APIs.

Gal Pressman June 7, 2023, 1:31 p.m. UTC | #13

On 06/06/2023 19:17, Jakub Kicinski wrote:
> On Tue, 6 Jun 2023 11:01:14 +0300 Gal Pressman wrote:
>> On 05/06/2023 22:27, Edwin Peer wrote:
>>> Thanks for the CC, I left Broadcom quite some time ago and am no
>>> longer subscribed to netdev as a result (been living in firmware land
>>> doing work in Rust).
>>>
>>> I have no immediate plans to pick this up, at least not in the short
>>> to medium term. My work in progress was on the laptop I returned and I
>>> cannot immediately recall what solution I had in mind here.
>>
>> Jakub, sorry if this has been discussed already in the past, but can you
>> please clarify what is an accepted (or more importantly, not accepted)
>> solution for this issue? I'm not familiar with the history and don't
>> want to repeat previous mistakes.
> 
> The problem is basically that attributes can only be 64kB and 
> the legacy SR-IOV API wraps all the link info in an attribute.

Isn't that a second order issue? The skb itself is limited to 32kB AFAICT.

>> So far I've seen discussions about increasing the recv buffer size, and
>> this patchset which changes the GETLINK ABI, both of which were nacked.
> 
> Filtering out some of the info, like the stats, is okay, but that just
> increases the limit. A limit still exists.

Any objections to at least take the second patch here?
It doesn't introduce any ABI changes, but will allow 'ip link show' to
work properly (although 'ip -s link show' will remain broken).

>> Having 'ip link show' broken is very unfortunate :\, how should one
>> approach this issue in 2023?
> 
> Sure is, which is why we should be moving away from the legacy SR-IOV
> APIs.

Agreed!
I do not suggest to extend/improve this API, just make sure it's not broken.

Jakub Kicinski June 7, 2023, 4:33 p.m. UTC | #14

On Wed, 7 Jun 2023 16:31:48 +0300 Gal Pressman wrote:
> On 06/06/2023 19:17, Jakub Kicinski wrote:
> > On Tue, 6 Jun 2023 11:01:14 +0300 Gal Pressman wrote:  
> >> Jakub, sorry if this has been discussed already in the past, but can you
> >> please clarify what is an accepted (or more importantly, not accepted)
> >> solution for this issue? I'm not familiar with the history and don't
> >> want to repeat previous mistakes.  
> > 
> > The problem is basically that attributes can only be 64kB and 
> > the legacy SR-IOV API wraps all the link info in an attribute.  
> 
> Isn't that a second order issue? The skb itself is limited to 32kB AFAICT.

Hm, you're right. But allocation larger than 32kB are costly.
We can't make every link dump allocate 64kB, it will cause
regressions on systems under memory pressure (== real world).

You'd need to come up with some careful scheme of using larger
buffers.

> >> So far I've seen discussions about increasing the recv buffer size, and
> >> this patchset which changes the GETLINK ABI, both of which were nacked.  
> > 
> > Filtering out some of the info, like the stats, is okay, but that just
> > increases the limit. A limit still exists.  
> 
> Any objections to at least take the second patch here?
> It doesn't introduce any ABI changes, but will allow 'ip link show' to
> work properly (although 'ip -s link show' will remain broken).

Yup, retest / repost?

> >> Having 'ip link show' broken is very unfortunate :\, how should one
> >> approach this issue in 2023?  
> > 
> > Sure is, which is why we should be moving away from the legacy SR-IOV
> > APIs.  
> 
> Agreed!
> I do not suggest to extend/improve this API, just make sure it's not broken.
>

Stephen Hemminger June 7, 2023, 4:52 p.m. UTC | #15

On Wed, 7 Jun 2023 09:33:24 -0700
Jakub Kicinski <[e-mail]> wrote:

> > > 
> > > The problem is basically that attributes can only be 64kB and 
> > > the legacy SR-IOV API wraps all the link info in an attribute.    
> > 
> > Isn't that a second order issue? The skb itself is limited to 32kB AFAICT.  
> 
> Hm, you're right. But allocation larger than 32kB are costly.
> We can't make every link dump allocate 64kB, it will cause
> regressions on systems under memory pressure (== real world).
> 
> You'd need to come up with some careful scheme of using larger
> buffers.

Why does it all have to be a single message?
Things like 3 million routes are dumped fine, as multiple messages.

Jakub Kicinski June 7, 2023, 5:29 p.m. UTC | #16

On Wed, 7 Jun 2023 09:52:54 -0700 Stephen Hemminger wrote:
> > Hm, you're right. But allocation larger than 32kB are costly.
> > We can't make every link dump allocate 64kB, it will cause
> > regressions on systems under memory pressure (== real world).
> > 
> > You'd need to come up with some careful scheme of using larger
> > buffers.  
> 
> Why does it all have to be a single message?
> Things like 3 million routes are dumped fine, as multiple messages.

The old API we can't change. The new API is switchdev / devlink,
and it's already in place. We'd have to add a third API, like 
the old one but with different msg format which for obvious 
reasons I'd prefer not to do.

Patch

diff --git a/include/net/netlink.h b/include/net/netlink.h
index 1ceec518ab49..fc8c57dafb05 100644
--- a/include/net/netlink.h
+++ b/include/net/netlink.h
@@ -1785,19 +1785,26 @@  static inline struct nlattr *nla_nest_start(struct sk_buff *skb, int attrtype)
 	return nla_nest_start_noflag(skb, attrtype | NLA_F_NESTED);
 }
 
+int __nla_nest_trunc_msg(struct sk_buff *skb, const struct nlattr *start);
+
 /**
  * nla_nest_end - Finalize nesting of attributes
  * @skb: socket buffer the attributes are stored in
  * @start: container attribute
  *
  * Corrects the container attribute header to include the all
- * appeneded attributes.
+ * appeneded attributes. The list of attributes will be truncated
+ * if too long to fit within the parent attribute's maximum reach.
  *
  * Returns the total data length of the skb.
  */
 static inline int nla_nest_end(struct sk_buff *skb, struct nlattr *start)
 {
-	start->nla_len = skb_tail_pointer(skb) - (unsigned char *)start;
+	int len = skb_tail_pointer(skb) - (unsigned char *)start;
+
+	if (len > 0xffff)
+		len = __nla_nest_trunc_msg(skb, start);
+	start->nla_len = len;
 	return skb->len;
 }
 
diff --git a/include/uapi/linux/netlink.h b/include/uapi/linux/netlink.h
index 3d94269bbfa8..44a250825c30 100644
--- a/include/uapi/linux/netlink.h
+++ b/include/uapi/linux/netlink.h
@@ -57,6 +57,7 @@  struct nlmsghdr {
 #define NLM_F_ECHO		0x08	/* Echo this request 		*/
 #define NLM_F_DUMP_INTR		0x10	/* Dump was inconsistent due to sequence change */
 #define NLM_F_DUMP_FILTERED	0x20	/* Dump was filtered as requested */
+#define NLM_F_NEST_TRUNCATED	0x40	/* Message contains truncated nested attribute */
 
 /* Modifiers to GET request */
 #define NLM_F_ROOT	0x100	/* specify tree	root	*/
diff --git a/lib/nlattr.c b/lib/nlattr.c
index 5b6116e81f9f..2a267c0d3e16 100644
--- a/lib/nlattr.c
+++ b/lib/nlattr.c
@@ -1119,4 +1119,31 @@  int nla_append(struct sk_buff *skb, int attrlen, const void *data)
 	return 0;
 }
 EXPORT_SYMBOL(nla_append);
+
+/**
+ * __nla_nest_trunc_msg - Truncate list of nested netlink attributes to max len
+ * @skb: socket buffer with tail pointer positioned after end of nested list
+ * @start: container attribute designating the beginning of the list
+ *
+ * Trims the skb to fit only the attributes which are within the range of the
+ * containing nest attribute. This is a helper for nla_nest_end, to prevent
+ * adding unduly to the length of what is an inline function. It is not
+ * intended to be called from anywhere else.
+ *
+ * Returns the truncated length of the enclosing nest attribute in accordance
+ * with the number of whole attributes that can fit.
+ */
+int __nla_nest_trunc_msg(struct sk_buff *skb, const struct nlattr *start)
+{
+	struct nlattr *attr = nla_data(start);
+	int rem = 0xffff - NLA_HDRLEN;
+
+	while (nla_ok(attr, rem))
+		attr = nla_next(attr, &rem);
+	nlmsg_trim(skb, attr);
+	nlmsg_hdr(skb)->nlmsg_flags |= NLM_F_NEST_TRUNCATED;
+	return (unsigned char *)attr - (unsigned char *)start;
+}
+EXPORT_SYMBOL(__nla_nest_trunc_msg);
+
 #endif