| Message ID | 20220617121932.249381-1-victortoso@redhat.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | qapi: add generator for Golang interface | expand |
Victor Toso <victortoso@redhat.com> writes: > Hi, > > This is the second iteration of RFC v1: > https://lists.gnu.org/archive/html/qemu-devel/2022-04/msg00226.html > > > # What this is about? > > To generate a simple Golang interface that could communicate with QEMU > over QMP. The Go code that is generated is meant to be used as the bare > bones to exchange QMP messages. > > The goal is to have this as a Go module in QEMU gitlab namespace, > similar to what have been done to pyhon-qemu-qmp > https://gitlab.com/qemu-project/python-qemu-qmp Aspects of review: (1) Impact on common code, if any I care, because any messes made there are likely to affect me down the road. (2) The generated Go code Is it (close to) what we want long term? If not, is it good enough short term, and how could we make necessary improvements? I'd prefer to leave this to folks who actually know their Go. (3) General Python sanity We need eyes, but not necessarily mine. Any takers? [...] > scripts/qapi/golang.py | 765 +++++++++++++++++++++++++++++++++++++++++ > scripts/qapi/main.py | 2 + > 2 files changed, 767 insertions(+) > create mode 100644 scripts/qapi/golang.py This adds a new generator and calls it from generate(), i.e. review aspect (1) is empty. "Empty" is a quick & easy way to get my ACK! No tests? No documentation?
Hi Markus, On Mon, Jun 27, 2022 at 09:15:53AM +0200, Markus Armbruster wrote: > Victor Toso <victortoso@redhat.com> writes: > > > Hi, > > > > This is the second iteration of RFC v1: > > https://lists.gnu.org/archive/html/qemu-devel/2022-04/msg00226.html > > > > > > # What this is about? > > > > To generate a simple Golang interface that could communicate with QEMU > > over QMP. The Go code that is generated is meant to be used as the bare > > bones to exchange QMP messages. > > > > The goal is to have this as a Go module in QEMU gitlab namespace, > > similar to what have been done to pyhon-qemu-qmp > > https://gitlab.com/qemu-project/python-qemu-qmp > > Aspects of review: > > (1) Impact on common code, if any > > I care, because any messes made there are likely to affect me down > the road. For the first version of the Go generated interface, my goal is to have something that works and can be considered alpha to other Go projects. With this first version, I don't want to bring huge changes to the python library or to the QAPI spec and its usage in QEMU. Unless someone finds that a necessity. So I hope (1) is simple :) > (2) The generated Go code > > Is it (close to) what we want long term? If not, is it good enough > short term, and how could we make necessary improvements? > > I'd prefer to leave this to folks who actually know their Go. > (3) General Python sanity > > We need eyes, but not necessarily mine. Any takers? > > [...] > > > scripts/qapi/golang.py | 765 +++++++++++++++++++++++++++++++++++++++++ > > scripts/qapi/main.py | 2 + > > 2 files changed, 767 insertions(+) > > create mode 100644 scripts/qapi/golang.py > > This adds a new generator and calls it from generate(), i.e. > review aspect (1) is empty. "Empty" is a quick & easy way to > get my ACK! > > No tests? I've added tests but on the qapi-go module, those are the files with _test.go prefix on them. Example for commands: https://gitlab.com/victortoso/qapi-go/-/blob/main/pkg/qapi/commands_test.go Should the generator itself have tests or offloading that to the qapi-go seems reasonable? > No documentation? Yes, this iteration removed the documentation of the generated types. I'm a bit sad about that. I want to consume the documentation in the QAPI files to provide the latest info from types/fields but we can't 'copy' it, the only solution is 'refer' to it with hyperlink, which I haven't done yet. Cheers, Victor
Victor Toso <victortoso@redhat.com> writes: > Hi Markus, > > On Mon, Jun 27, 2022 at 09:15:53AM +0200, Markus Armbruster wrote: >> Victor Toso <victortoso@redhat.com> writes: >> >> > Hi, >> > >> > This is the second iteration of RFC v1: >> > https://lists.gnu.org/archive/html/qemu-devel/2022-04/msg00226.html >> > >> > >> > # What this is about? >> > >> > To generate a simple Golang interface that could communicate with QEMU >> > over QMP. The Go code that is generated is meant to be used as the bare >> > bones to exchange QMP messages. >> > >> > The goal is to have this as a Go module in QEMU gitlab namespace, >> > similar to what have been done to pyhon-qemu-qmp >> > https://gitlab.com/qemu-project/python-qemu-qmp >> >> Aspects of review: >> >> (1) Impact on common code, if any >> >> I care, because any messes made there are likely to affect me down >> the road. > > For the first version of the Go generated interface, my goal is > to have something that works and can be considered alpha to other > Go projects. > > With this first version, I don't want to bring huge changes to > the python library or to the QAPI spec and its usage in QEMU. > Unless someone finds that a necessity. > > So I hope (1) is simple :) > >> (2) The generated Go code >> >> Is it (close to) what we want long term? If not, is it good enough >> short term, and how could we make necessary improvements? >> >> I'd prefer to leave this to folks who actually know their Go. >> (3) General Python sanity >> >> We need eyes, but not necessarily mine. Any takers? >> >> [...] >> >> > scripts/qapi/golang.py | 765 +++++++++++++++++++++++++++++++++++++++++ >> > scripts/qapi/main.py | 2 + >> > 2 files changed, 767 insertions(+) >> > create mode 100644 scripts/qapi/golang.py >> >> This adds a new generator and calls it from generate(), i.e. >> review aspect (1) is empty. "Empty" is a quick & easy way to >> get my ACK! >> >> No tests? > > I've added tests but on the qapi-go module, those are the files > with _test.go prefix on them. Example for commands: > > https://gitlab.com/victortoso/qapi-go/-/blob/main/pkg/qapi/commands_test.go > > Should the generator itself have tests or offloading that to the > qapi-go seems reasonable? Offloading may be reasonable, but how am I to run the tests then? Documentation should tell me. We have roughly three kinds of tests so far: 1. Front end tests in tests/qapi-schema 2. Unit tests in tests/unit/ To find them: $ git-grep '#include ".*qapi-.*\.h"' tests/unit/ 3. Many tests talking QMP in tests/qtest/ Since you leave the front end alone, you don't need the first kind. The other two kinds are less clear. >> No documentation? > > Yes, this iteration removed the documentation of the generated > types. I'm a bit sad about that. I want to consume the > documentation in the QAPI files to provide the latest info from > types/fields but we can't 'copy' it, the only solution is 'refer' > to it with hyperlink, which I haven't done yet. Two kinds of documentation: generated documentation for the generated Go code, and documentation about the generator. I was thinking of the latter. Specifically, docs/devel/qapi-code-gen.rst section "Code generation". Opinions on its usefulness differ. I like it.
I've commented in detail to the single patches, just a couple of additional points. On Fri, Jun 17, 2022 at 02:19:24PM +0200, Victor Toso wrote: > * 7) Flat structs by removing embed types. Discussion with Andrea > Thread: https://lists.gnu.org/archive/html/qemu-devel/2022-05/msg01590.html > > No one required it but I decided to give it a try. Major issue that > I see with this approach is to have generated a few 'Base' structs > that are now useless. Overall, less nested structs seems better to > me. Opnions? > > Example: > | /* This is now useless, should be removed? */ > | type InetSocketAddressBase struct { > | Host string `json:"host"` > | Port string `json:"port"` > | } Can we somehow keep track, in the generator, of types that are only used as building blocks for other types, and prevent them from showing up in the generated code? Finally, looking at the repository containing the generated code I see that the generated type are sorted by kind, e.g. all unions are in a file, all events in another one and so on. I believe the structure should match more closely that of the QAPI schema, so e.g. block-related types should all go in one file, net-related types in another one and so on. Looking forward to the next iteration :)
Hi, On Tue, Jul 05, 2022 at 08:46:34AM -0700, Andrea Bolognani wrote: > I've commented in detail to the single patches, just a couple of > additional points. > > On Fri, Jun 17, 2022 at 02:19:24PM +0200, Victor Toso wrote: > > * 7) Flat structs by removing embed types. Discussion with Andrea > > Thread: https://lists.gnu.org/archive/html/qemu-devel/2022-05/msg01590.html > > > > No one required it but I decided to give it a try. Major issue that > > I see with this approach is to have generated a few 'Base' structs > > that are now useless. Overall, less nested structs seems better to > > me. Opnions? > > > > Example: > > | /* This is now useless, should be removed? */ > > | type InetSocketAddressBase struct { > > | Host string `json:"host"` > > | Port string `json:"port"` > > | } > > Can we somehow keep track, in the generator, of types that are > only used as building blocks for other types, and prevent them > from showing up in the generated code? I'm not 100% sure it is good to remove them from generated code because technically it is a valid qapi type. If all @base types are embed types and they don't show in other way or form, sure we can remove them from generated code... I'm not sure if it is possible to guarantee this. But yes, if possible, I'd like to remove what seems useless type definitions. > Finally, looking at the repository containing the generated > code I see that the generated type are sorted by kind, e.g. all > unions are in a file, all events in another one and so on. I > believe the structure should match more closely that of the > QAPI schema, so e.g. block-related types should all go in one > file, net-related types in another one and so on. That's something I don't mind adding but some hardcoded mapping is needed. If you look into git history of qapi/ folder, .json files can come and go, types be moved around, etc. So, we need to proper map types in a way that the generated code would be kept stable even if qapi files would have been rearranged. What I proposed was only the simplest solution. Also, the generator takes a qapi-schema.json as input. We are more focused in qemu/qapi/qapi-schema.json generated coded but would not hurt to think we could even use it for qemu-guest-agent from qemu/qga/qapi-schema.json -- this to say that the hardcoded mapping needs to take into account non qemu qapi schemas too. > Looking forward to the next iteration :) Me too, thanks again! Cheers, Victor
Hi, On Mon, Jun 27, 2022 at 05:29:26PM +0200, Markus Armbruster wrote: > Victor Toso <victortoso@redhat.com> writes: > > > Hi Markus, > > > > On Mon, Jun 27, 2022 at 09:15:53AM +0200, Markus Armbruster wrote: > >> Victor Toso <victortoso@redhat.com> writes: > >> > >> > Hi, > >> > > >> > This is the second iteration of RFC v1: > >> > https://lists.gnu.org/archive/html/qemu-devel/2022-04/msg00226.html > >> > > >> > > >> > # What this is about? > >> > > >> > To generate a simple Golang interface that could communicate with QEMU > >> > over QMP. The Go code that is generated is meant to be used as the bare > >> > bones to exchange QMP messages. > >> > > >> > The goal is to have this as a Go module in QEMU gitlab namespace, > >> > similar to what have been done to pyhon-qemu-qmp > >> > https://gitlab.com/qemu-project/python-qemu-qmp > >> > >> Aspects of review: > >> > >> (1) Impact on common code, if any > >> > >> I care, because any messes made there are likely to affect me down > >> the road. > > > > For the first version of the Go generated interface, my goal is > > to have something that works and can be considered alpha to other > > Go projects. > > > > With this first version, I don't want to bring huge changes to > > the python library or to the QAPI spec and its usage in QEMU. > > Unless someone finds that a necessity. > > > > So I hope (1) is simple :) > > > >> (2) The generated Go code > >> > >> Is it (close to) what we want long term? If not, is it good enough > >> short term, and how could we make necessary improvements? > >> > >> I'd prefer to leave this to folks who actually know their Go. > >> (3) General Python sanity > >> > >> We need eyes, but not necessarily mine. Any takers? > >> > >> [...] > >> > >> > scripts/qapi/golang.py | 765 +++++++++++++++++++++++++++++++++++++++++ > >> > scripts/qapi/main.py | 2 + > >> > 2 files changed, 767 insertions(+) > >> > create mode 100644 scripts/qapi/golang.py > >> > >> This adds a new generator and calls it from generate(), i.e. > >> review aspect (1) is empty. "Empty" is a quick & easy way to > >> get my ACK! > >> > >> No tests? > > > > I've added tests but on the qapi-go module, those are the files > > with _test.go prefix on them. Example for commands: > > > > https://gitlab.com/victortoso/qapi-go/-/blob/main/pkg/qapi/commands_test.go > > > > Should the generator itself have tests or offloading that to the > > qapi-go seems reasonable? > > Offloading may be reasonable, but how am I to run the tests then? > Documentation should tell me. > > We have roughly three kinds of tests so far: > > 1. Front end tests in tests/qapi-schema > > 2. Unit tests in tests/unit/ > > To find them: > > $ git-grep '#include ".*qapi-.*\.h"' tests/unit/ > > 3. Many tests talking QMP in tests/qtest/ I'm thinking on the tests in QEMU side. Perhaps adding something with Avocado that generates the qapi-go and communicates with a running QEMU with that generated Go module? One thing that I try to keep in mind is to not add Go dependencies in QEMU and this Golang work is not internal to QEMU itself. > Since you leave the front end alone, you don't need the first > kind. > > The other two kinds are less clear. I'm open for suggestions. I thought that, if we have a qapi-go Go module in Gitlab's qemu-project namespace, we could leverage most of the tests to the consumer of the actual generated code but I agree that it is necessary to have something in qemu too. > >> No documentation? > > > > Yes, this iteration removed the documentation of the generated > > types. I'm a bit sad about that. I want to consume the > > documentation in the QAPI files to provide the latest info from > > types/fields but we can't 'copy' it, the only solution is 'refer' > > to it with hyperlink, which I haven't done yet. > > Two kinds of documentation: generated documentation for the generated Go > code, and documentation about the generator. I was thinking of the > latter. Specifically, docs/devel/qapi-code-gen.rst section "Code > generation". Opinions on its usefulness differ. I like it. Me too. I'll add documentation for the next iteration, thanks for pointing it out. Cheers, Victor
Victor Toso <victortoso@redhat.com> writes: > Hi, > > On Tue, Jul 05, 2022 at 08:46:34AM -0700, Andrea Bolognani wrote: >> I've commented in detail to the single patches, just a couple of >> additional points. >> >> On Fri, Jun 17, 2022 at 02:19:24PM +0200, Victor Toso wrote: >> > * 7) Flat structs by removing embed types. Discussion with Andrea >> > Thread: https://lists.gnu.org/archive/html/qemu-devel/2022-05/msg01590.html >> > >> > No one required it but I decided to give it a try. Major issue that >> > I see with this approach is to have generated a few 'Base' structs >> > that are now useless. Overall, less nested structs seems better to >> > me. Opnions? >> > >> > Example: >> > | /* This is now useless, should be removed? */ >> > | type InetSocketAddressBase struct { >> > | Host string `json:"host"` >> > | Port string `json:"port"` >> > | } >> >> Can we somehow keep track, in the generator, of types that are >> only used as building blocks for other types, and prevent them >> from showing up in the generated code? > > I'm not 100% sure it is good to remove them from generated code > because technically it is a valid qapi type. If all @base types > are embed types and they don't show in other way or form, sure we > can remove them from generated code... I'm not sure if it is > possible to guarantee this. > > But yes, if possible, I'd like to remove what seems useless type > definitions. The existing C generators have to generate all the types, because the generated code is for QEMU's own use, where we need all the types. The existing introspection generator generates only the types visible in QAPI/QMP introspection. The former generate for internal use (where we want all the types), and the latter for external use (where only the types visible in the external interface are actually useful). >> Finally, looking at the repository containing the generated >> code I see that the generated type are sorted by kind, e.g. all >> unions are in a file, all events in another one and so on. I >> believe the structure should match more closely that of the >> QAPI schema, so e.g. block-related types should all go in one >> file, net-related types in another one and so on. > > That's something I don't mind adding but some hardcoded mapping > is needed. If you look into git history of qapi/ folder, .json > files can come and go, types be moved around, etc. So, we need to > proper map types in a way that the generated code would be kept > stable even if qapi files would have been rearranged. What I > proposed was only the simplest solution. > > Also, the generator takes a qapi-schema.json as input. We are > more focused in qemu/qapi/qapi-schema.json generated coded but > would not hurt to think we could even use it for qemu-guest-agent > from qemu/qga/qapi-schema.json -- this to say that the hardcoded > mapping needs to take into account non qemu qapi schemas too. In the beginning, the QAPI schema was monolithic. qga/qapi-schema.json still is. When keeping everything in a single qapi-schema.json became unwieldy, we split it into "modules" tied together with a simple include directive. Generated code remained monolithic. When monolithic generated code became too annoying (touch schema, recompile everything), we made it match the module structure: code for FOO.json goes into *-FOO.c and *-FOO.h, where the *-FOO.h #include the generated headers for the .json modules FOO.json includes. Schema code motion hasn't been much of a problem. Moving from FOO.json to one of the modules it includes is transparent. Non-transparent moves are relatively rare as long as the split into modules actually makes sense. >> Looking forward to the next iteration :) > > Me too, thanks again! > > Cheers, > Victor
Hi, On Mon, Aug 29, 2022 at 01:53:51PM +0200, Markus Armbruster wrote: > Victor Toso <victortoso@redhat.com> writes: > > > Hi, > > > > On Tue, Jul 05, 2022 at 08:46:34AM -0700, Andrea Bolognani wrote: > >> I've commented in detail to the single patches, just a couple of > >> additional points. > >> > >> On Fri, Jun 17, 2022 at 02:19:24PM +0200, Victor Toso wrote: > >> > * 7) Flat structs by removing embed types. Discussion with Andrea > >> > Thread: https://lists.gnu.org/archive/html/qemu-devel/2022-05/msg01590.html > >> > > >> > No one required it but I decided to give it a try. Major issue that > >> > I see with this approach is to have generated a few 'Base' structs > >> > that are now useless. Overall, less nested structs seems better to > >> > me. Opnions? > >> > > >> > Example: > >> > | /* This is now useless, should be removed? */ > >> > | type InetSocketAddressBase struct { > >> > | Host string `json:"host"` > >> > | Port string `json:"port"` > >> > | } > >> > >> Can we somehow keep track, in the generator, of types that are > >> only used as building blocks for other types, and prevent them > >> from showing up in the generated code? > > > > I'm not 100% sure it is good to remove them from generated code > > because technically it is a valid qapi type. If all @base types > > are embed types and they don't show in other way or form, sure we > > can remove them from generated code... I'm not sure if it is > > possible to guarantee this. > > > > But yes, if possible, I'd like to remove what seems useless type > > definitions. > > The existing C generators have to generate all the types, because the > generated code is for QEMU's own use, where we need all the types. > > The existing introspection generator generates only the types visible in > QAPI/QMP introspection. > > The former generate for internal use (where we want all the types), and > the latter for external use (where only the types visible in the > external interface are actually useful). My doubt are on types that might be okay to be hidden because they are embed in other types, like InetSocketAddressBase. Note that what I mean with the struct being embed is that the actual fields of InetSocketAddressBase being added to the type which uses it, like InetSocketAddress. | type InetSocketAddressBase struct { | Host string `json:"host"` | Port string `json:"port"` | } | | type InetSocketAddress struct { | // Base fields | Host string `json:"host"` | Port string `json:"port"` | | Numeric *bool `json:"numeric,omitempty"` | To *uint16 `json:"to,omitempty"` | Ipv4 *bool `json:"ipv4,omitempty"` | Ipv6 *bool `json:"ipv6,omitempty"` | KeepAlive *bool `json:"keep-alive,omitempty"` | Mptcp *bool `json:"mptcp,omitempty"` | } Andrea's suggestions is to have the generator to track if a given type is always embed in which case we can skip generating it in the Go module. I think that could work indeed. In the hypothetical case that hiddens structs like InetSocketAddressBase becomes a parameter on command in the future, the generator would know and start generating this type from that point onwards. > >> Finally, looking at the repository containing the generated > >> code I see that the generated type are sorted by kind, e.g. all > >> unions are in a file, all events in another one and so on. I > >> believe the structure should match more closely that of the > >> QAPI schema, so e.g. block-related types should all go in one > >> file, net-related types in another one and so on. > > > > That's something I don't mind adding but some hardcoded mapping > > is needed. If you look into git history of qapi/ folder, .json > > files can come and go, types be moved around, etc. So, we need to > > proper map types in a way that the generated code would be kept > > stable even if qapi files would have been rearranged. What I > > proposed was only the simplest solution. > > > > Also, the generator takes a qapi-schema.json as input. We are > > more focused in qemu/qapi/qapi-schema.json generated coded but > > would not hurt to think we could even use it for qemu-guest-agent > > from qemu/qga/qapi-schema.json -- this to say that the hardcoded > > mapping needs to take into account non qemu qapi schemas too. > > In the beginning, the QAPI schema was monolithic. > qga/qapi-schema.json still is. > > When keeping everything in a single qapi-schema.json became > unwieldy, we split it into "modules" tied together with a > simple include directive. Generated code remained monolithic. > When monolithic generated code became too annoying (touch > schema, recompile everything), we made it match the module > structure: code for FOO.json goes into *-FOO.c and *-FOO.h, > where the *-FOO.h #include the generated headers for the .json > modules FOO.json includes. > > Schema code motion hasn't been much of a problem. Moving from > FOO.json to one of the modules it includes is transparent. > Non-transparent moves are relatively rare as long as the split > into modules actually makes sense. To be honest, splitting it into different files based on their module names should not be a problem if we keep them in a single Go module as do intend to for this generated go code. So I'll go ahead and split it. Cheers, Victor
Bringing this thread back from the dead, since I had an in-person discussion on a key question below at KVM Forum this year and want to record it here. On Fri, Jun 17, 2022 at 02:19:24PM +0200, Victor Toso wrote: > Hi, > > This is the second iteration of RFC v1: > https://lists.gnu.org/archive/html/qemu-devel/2022-04/msg00226.html > > > # What this is about? > > To generate a simple Golang interface that could communicate with QEMU > over QMP. The Go code that is generated is meant to be used as the bare > bones to exchange QMP messages. > > The goal is to have this as a Go module in QEMU gitlab namespace, > similar to what have been done to pyhon-qemu-qmp > https://gitlab.com/qemu-project/python-qemu-qmp > > > # Issues raised in RFC v1 > > The leading '*' for issues I addressed in this iteration > > * 1) Documentation was removed to avoid License issues, by Daniel > Thread: https://lists.nongnu.org/archive/html/qemu-devel/2022-05/msg01889.html > > It is important for the generated Go module to be compatible with > Licenses used by projects that would be using this. Copying the > documentation of the QAPI spec might conflict with GPLv2+. > > I have not proposed another license in this iteration, but I'm > planning to go with MIT No Attribution, aka MIT-0 [0]. Does it make > sense to bind the generated code's license to MIT-0 already at > generator level? > > [0] https://github.com/aws/mit-0/blob/master/MIT-0 To recap the situation * The license of the code generator itself does not determine the license of the output generated code * The license of the inputs to the code generator, may or may not, determine the license of the output generated code depending on use context The primary input to the code generator is the QAPI schema, which is part of QEMU and thus licensed GPL-2.0-or-later. The QAPI schema includes both the API definitions AND the API documentation text. We can make the case that as the QEMU public interface, consuming the API definitions in the QAPI schema for the purpose of generating code is "fair use", and thus the output generated code does NOT need to match the GPL-2.0-or-later license of the QAPI schema. We can choose the code license, and a maximally permissive license looks appropriate. We want to have API documentation for the Golang bindings and the obvious way to achieve this is to copy the API docs from the QAPI schema into the Golang code. It is NOT reasonable to class such *direct copying* of docs "fair use". IOW, copied docs will be under GPL-2.0-or-later. Thus if we pick MIT-0 for the Golang code, and copy across the QAPI docs, the resulting Golang QAPI project code would be under a compound license term "MIT-0 AND GPL-2.0-or-later". The concern was that this will limit the ability of downstream applications to consume the Golang bindings, if they don't want their combined work to contain GPL-2.0-or-later. Ignoring whether this fear of GPL-2.0-or-later is sane or not, in retrospect I believe that this concern in fact has no legal basis. The license of a compiled application binary is the union of all the object files linked into it. Notice I said "Object file" there, **NOT** "Source file". This is the crucial distinction that makes the presense of GPL-2.0-or-later docs a non-issue from a licensing POV. When the compiler takes the "MIT-0 and GPL-2.0-or-later" license .go source file, and builds an object file, it will be discarding all the API documentation comments. IOW, all the parts that were under GPL-2.0-or-later are discarded. The only parts of the source file that get "compiled" are the Go language constructs which were MIT-0 licensed [1]. IOW, we have a "MIT-0 and GPL-2.0-or-later" .go source file, and an "MIT-0" object file. Thus while there may be a human perception problem with the Golang bindings being "MIT-0 and GPL-2.0-or-later", there are no legal licensing limitations, as the combined work for a library or application linking the bindings will only contain the MIT-0 part. The GPL-2.0-or-later docs won't influence the license of the combined work. Note, this interpretation applies only to languages which are compiled, not interpreted. If we are generating python code for example, the there is no "source file" vs "object file" distinction for licensing. The combined work in a python app is under the union of all the source file licenses. TL;DR: I think you can re-add the documentation comments to the Golang code generator, declare the resulting code as being "MIT-0 AND GPL-2.0-or-later". In the README.md file we need todo two important things: * Document our interpretation of the "combined work" license situation for applications consuming the project. * Declare that *ALL* manually written code contributions are exclusively MIT-0. With regards, Daniel [1] NB, there are languages where some code comments can have semantic impacts on the compiled out. I don't believe that to be the case for any human targetted API docs that we would be copying over from the QAPI schema though in the Golang case.
Daniel P. Berrangé <berrange@redhat.com> writes: > Bringing this thread back from the dead, since I had an in-person > discussion on a key question below at KVM Forum this year and want > to record it here. [...] > To recap the situation > > * The license of the code generator itself does not determine the > license of the output generated code For instance, GNU Bison is GPLv3+, but the parts the generator emits (the parser skeleton) come with a special exception. Such exceptions need to be granted by the copyright holder. As long as the code generating Go is not a derived work, the copyright holder situation should be simple enough to make this practical. > * The license of the inputs to the code generator, may or may > not, determine the license of the output generated code depending > on use context > > The primary input to the code generator is the QAPI schema, which is part > of QEMU and thus licensed GPL-2.0-or-later. > > The QAPI schema includes both the API definitions AND the API documentation > text. > > We can make the case that as the QEMU public interface, consuming the > API definitions in the QAPI schema for the purpose of generating code > is "fair use", and thus the output generated code does NOT need to > match the GPL-2.0-or-later license of the QAPI schema. We can choose > the code license, and a maximally permissive license looks appropriate. Having this argument confirmed by an actual expert seems advisable. > We want to have API documentation for the Golang bindings and the obvious > way to achieve this is to copy the API docs from the QAPI schema into the > Golang code. It is NOT reasonable to class such *direct copying* of docs > "fair use". IOW, copied docs will be under GPL-2.0-or-later. > > Thus if we pick MIT-0 for the Golang code, and copy across the QAPI docs, > the resulting Golang QAPI project code would be under a compound license > term "MIT-0 AND GPL-2.0-or-later". > > The concern was that this will limit the ability of downstream > applications to consume the Golang bindings, if they don't want their > combined work to contain GPL-2.0-or-later. > > Ignoring whether this fear of GPL-2.0-or-later is sane or not, > in retrospect I believe that this concern in fact has no legal > basis. > > The license of a compiled application binary is the union of all > the object files linked into it. > > Notice I said "Object file" there, **NOT** "Source file". > > This is the crucial distinction that makes the presense of > GPL-2.0-or-later docs a non-issue from a licensing POV. > > > When the compiler takes the "MIT-0 and GPL-2.0-or-later" license > .go source file, and builds an object file, it will be discarding > all the API documentation comments. IOW, all the parts that were > under GPL-2.0-or-later are discarded. The only parts of the source > file that get "compiled" are the Go language constructs which were > MIT-0 licensed [1]. > > IOW, we have a "MIT-0 and GPL-2.0-or-later" .go source file, > and an "MIT-0" object file. > > Thus while there may be a human perception problem with the Golang > bindings being "MIT-0 and GPL-2.0-or-later", there are no legal > licensing limitations, as the combined work for a library or > application linking the bindings will only contain the MIT-0 part. > > The GPL-2.0-or-later docs won't influence the license of the > combined work. > > Note, this interpretation applies only to languages which are > compiled, not interpreted. > > If we are generating python code for example, the there is > no "source file" vs "object file" distinction for licensing. > The combined work in a python app is under the union of all > the source file licenses. > > > TL;DR: I think you can re-add the documentation comments to > the Golang code generator, declare the resulting code as being > "MIT-0 AND GPL-2.0-or-later". > > In the README.md file we need todo two important things: > > * Document our interpretation of the "combined work" license > situation for applications consuming the project. > * Declare that *ALL* manually written code contributions > are exclusively MIT-0. What code do you have in mind? Can you provide an example? > With regards, > Daniel > > [1] NB, there are languages where some code comments can have semantic > impacts on the compiled out. I don't believe that to be the case for > any human targetted API docs that we would be copying over from the > QAPI schema though in the Golang case.
On Thu, Nov 07, 2024 at 01:36:49PM +0100, Markus Armbruster wrote: > Daniel P. Berrangé <berrange@redhat.com> writes: > > > Bringing this thread back from the dead, since I had an in-person > > discussion on a key question below at KVM Forum this year and want > > to record it here. > > [...] > > > To recap the situation > > > > * The license of the code generator itself does not determine the > > license of the output generated code > > For instance, GNU Bison is GPLv3+, but the parts the generator emits > (the parser skeleton) come with a special exception. > > Such exceptions need to be granted by the copyright holder. As long as > the code generating Go is not a derived work, the copyright holder > situation should be simple enough to make this practical. > > > * The license of the inputs to the code generator, may or may > > not, determine the license of the output generated code depending > > on use context > > > > The primary input to the code generator is the QAPI schema, which is part > > of QEMU and thus licensed GPL-2.0-or-later. > > > > The QAPI schema includes both the API definitions AND the API documentation > > text. > > > > We can make the case that as the QEMU public interface, consuming the > > API definitions in the QAPI schema for the purpose of generating code > > is "fair use", and thus the output generated code does NOT need to > > match the GPL-2.0-or-later license of the QAPI schema. We can choose > > the code license, and a maximally permissive license looks appropriate. > > Having this argument confirmed by an actual expert seems advisable. IANAL, but .... :-) The distinction of "source code" license vs "combined work" license, is a quite normal concept. As an example, it is something that is applied in Fedora RPM packaging when deciding the RPM spec "License:" tag. The latter is considered to refer to the licenses on the subset of source is said to be part of the distributed binaries, so we can exclude licenses of any part of the source that is not compiled, and of supporting code - eg license of makefiles, meson file, autoconf, code is excluded from the RPM license tag. The difference is that in most cases, the excluded part of the source is in separate files, or less frequently could be in #ifdef sections, whereas in this case we have two differently licensed parts of work in the same file. The question is thus what can you say is "compiled". Is the whole file "compiled", or are only the souce code constructs "compiled". My proposition is that, in most cases, comments are not used by the "compilation", or "code generation" phase. They are seen by the "parsing" phase only and thus dont contribute to the contents of the output binary code. With C code this is easy to illustrate. Run 'gcc -save-temps' and look at the '.i' file which is what the compiler uses as input for code generation after cpp has run. The '.i' file is devoid of all code comments. I've not proved the same for golang compilers yet, but my argument hinges on that being the true. None the less, we have raised this position/viewpoint with experts for a second opinion. > > We want to have API documentation for the Golang bindings and the obvious > > way to achieve this is to copy the API docs from the QAPI schema into the > > Golang code. It is NOT reasonable to class such *direct copying* of docs > > "fair use". IOW, copied docs will be under GPL-2.0-or-later. > > > > Thus if we pick MIT-0 for the Golang code, and copy across the QAPI docs, > > the resulting Golang QAPI project code would be under a compound license > > term "MIT-0 AND GPL-2.0-or-later". > > > > The concern was that this will limit the ability of downstream > > applications to consume the Golang bindings, if they don't want their > > combined work to contain GPL-2.0-or-later. > > > > Ignoring whether this fear of GPL-2.0-or-later is sane or not, > > in retrospect I believe that this concern in fact has no legal > > basis. > > > > The license of a compiled application binary is the union of all > > the object files linked into it. > > > > Notice I said "Object file" there, **NOT** "Source file". > > > > This is the crucial distinction that makes the presense of > > GPL-2.0-or-later docs a non-issue from a licensing POV. > > > > > > When the compiler takes the "MIT-0 and GPL-2.0-or-later" license > > .go source file, and builds an object file, it will be discarding > > all the API documentation comments. IOW, all the parts that were > > under GPL-2.0-or-later are discarded. The only parts of the source > > file that get "compiled" are the Go language constructs which were > > MIT-0 licensed [1]. > > > > IOW, we have a "MIT-0 and GPL-2.0-or-later" .go source file, > > and an "MIT-0" object file. > > > > Thus while there may be a human perception problem with the Golang > > bindings being "MIT-0 and GPL-2.0-or-later", there are no legal > > licensing limitations, as the combined work for a library or > > application linking the bindings will only contain the MIT-0 part. > > > > The GPL-2.0-or-later docs won't influence the license of the > > combined work. > > > > Note, this interpretation applies only to languages which are > > compiled, not interpreted. > > > > If we are generating python code for example, the there is > > no "source file" vs "object file" distinction for licensing. > > The combined work in a python app is under the union of all > > the source file licenses. > > > > > > TL;DR: I think you can re-add the documentation comments to > > the Golang code generator, declare the resulting code as being > > "MIT-0 AND GPL-2.0-or-later". > > > > In the README.md file we need todo two important things: > > > > * Document our interpretation of the "combined work" license > > situation for applications consuming the project. > > * Declare that *ALL* manually written code contributions > > are exclusively MIT-0. > > What code do you have in mind? Can you provide an example? I don't have any particular examples. Perhaps 100% of the .go code in the repo in question will be auto-generated. I can't predict the future, so there's a decent chance we might end up with some basic scaffolding and/or layered framework code that is manually written. If such a situation arises, it is important to have anticipated this, to make clear it is intended to be under a specific license. With regards, Daniel
On Thu, Nov 07, 2024 at 01:06:36PM +0000, Daniel P. Berrangé wrote: > On Thu, Nov 07, 2024 at 01:36:49PM +0100, Markus Armbruster wrote: > > Daniel P. Berrangé <berrange@redhat.com> writes: > > > > > Bringing this thread back from the dead, since I had an in-person > > > discussion on a key question below at KVM Forum this year and want > > > to record it here. > > > > [...] > > > > > To recap the situation > > > > > > * The license of the code generator itself does not determine the > > > license of the output generated code > > > > For instance, GNU Bison is GPLv3+, but the parts the generator emits > > (the parser skeleton) come with a special exception. > > > > Such exceptions need to be granted by the copyright holder. As long as > > the code generating Go is not a derived work, the copyright holder > > situation should be simple enough to make this practical. > > > > > * The license of the inputs to the code generator, may or may > > > not, determine the license of the output generated code depending > > > on use context > > > > > > The primary input to the code generator is the QAPI schema, which is part > > > of QEMU and thus licensed GPL-2.0-or-later. > > > > > > The QAPI schema includes both the API definitions AND the API documentation > > > text. > > > > > > We can make the case that as the QEMU public interface, consuming the > > > API definitions in the QAPI schema for the purpose of generating code > > > is "fair use", and thus the output generated code does NOT need to > > > match the GPL-2.0-or-later license of the QAPI schema. We can choose > > > the code license, and a maximally permissive license looks appropriate. > > > > Having this argument confirmed by an actual expert seems advisable. > > IANAL, but .... :-) > My proposition is that, in most cases, comments are not used by > the "compilation", or "code generation" phase. They are seen by > the "parsing" phase only and thus dont contribute to the contents > of the output binary code. > None the less, we have raised this position/viewpoint with experts > for a second opinion. Another thought or two... From a QEMU community POV we have no license problem no matter what, so from a legal POV this risk free for us. The licensing question is purely one for application developers consuming our work who want to avoid GPL for some reason. Lets say my interpretation is wrong, or even that it is right, but the app developers disagree, or are none the less unhappy in some way. What impact would this have for QEMU ? Code generation is practically free once the generator is written, so we just run it twice, storing the output in git on different branches with parallel tags, once with comments and once without. eg The "main" branch (and associated versioned vX.Y.Z tags) in qemu-go-qapi.git provide the full QAPI go source + copied QAPI comments under MIT-0 AND GPL-2.0-or-later The "minimal" branch (and associated vX.Y.Z-minimal tags) in qemu-go-qapi.git provide only the QAPI go source, without any comments under MIT-0 only. With regards, Daniel
Daniel P. Berrangé <berrange@redhat.com> writes: > On Thu, Nov 07, 2024 at 01:36:49PM +0100, Markus Armbruster wrote: >> Daniel P. Berrangé <berrange@redhat.com> writes: [...] >> > TL;DR: I think you can re-add the documentation comments to >> > the Golang code generator, declare the resulting code as being >> > "MIT-0 AND GPL-2.0-or-later". >> > >> > In the README.md file we need todo two important things: >> > >> > * Document our interpretation of the "combined work" license >> > situation for applications consuming the project. >> > * Declare that *ALL* manually written code contributions >> > are exclusively MIT-0. >> >> What code do you have in mind? Can you provide an example? > > I don't have any particular examples. Perhaps 100% of the .go code > in the repo in question will be auto-generated. I can't predict the > future, so there's a decent chance we might end up with some basic > scaffolding and/or layered framework code that is manually written. Yes. > If such a situation arises, it is important to have anticipated this, > to make clear it is intended to be under a specific license. Makes sense.
Hi, On Thu, Nov 07, 2024 at 10:43:06AM +0000, Daniel P. Berrangé wrote: > Bringing this thread back from the dead, since I had an > in-person discussion on a key question below at KVM Forum this > year and want to record it here. Thanks for that. > On Fri, Jun 17, 2022 at 02:19:24PM +0200, Victor Toso wrote: > > Hi, > > > > This is the second iteration of RFC v1: > > https://lists.gnu.org/archive/html/qemu-devel/2022-04/msg00226.html > > > > > > # What this is about? > > > > To generate a simple Golang interface that could communicate with QEMU > > over QMP. The Go code that is generated is meant to be used as the bare > > bones to exchange QMP messages. > > > > The goal is to have this as a Go module in QEMU gitlab namespace, > > similar to what have been done to pyhon-qemu-qmp > > https://gitlab.com/qemu-project/python-qemu-qmp > > > > > > # Issues raised in RFC v1 > > > > The leading '*' for issues I addressed in this iteration > > > > * 1) Documentation was removed to avoid License issues, by Daniel > > Thread: https://lists.nongnu.org/archive/html/qemu-devel/2022-05/msg01889.html > > > > It is important for the generated Go module to be compatible with > > Licenses used by projects that would be using this. Copying the > > documentation of the QAPI spec might conflict with GPLv2+. > > > > I have not proposed another license in this iteration, but I'm > > planning to go with MIT No Attribution, aka MIT-0 [0]. Does it make > > sense to bind the generated code's license to MIT-0 already at > > generator level? > > > > [0] https://github.com/aws/mit-0/blob/master/MIT-0 > > To recap the situation > > * The license of the code generator itself does not determine the > license of the output generated code > > * The license of the inputs to the code generator, may or may > not, determine the license of the output generated code depending > on use context > > The primary input to the code generator is the QAPI schema, which is part > of QEMU and thus licensed GPL-2.0-or-later. > > The QAPI schema includes both the API definitions AND the API documentation > text. > > We can make the case that as the QEMU public interface, consuming the > API definitions in the QAPI schema for the purpose of generating code > is "fair use", and thus the output generated code does NOT need to > match the GPL-2.0-or-later license of the QAPI schema. We can choose > the code license, and a maximally permissive license looks appropriate. > > We want to have API documentation for the Golang bindings and the obvious > way to achieve this is to copy the API docs from the QAPI schema into the > Golang code. It is NOT reasonable to class such *direct copying* of docs > "fair use". IOW, copied docs will be under GPL-2.0-or-later. > > Thus if we pick MIT-0 for the Golang code, and copy across the QAPI docs, > the resulting Golang QAPI project code would be under a compound license > term "MIT-0 AND GPL-2.0-or-later". > > The concern was that this will limit the ability of downstream > applications to consume the Golang bindings, if they don't want their > combined work to contain GPL-2.0-or-later. > > Ignoring whether this fear of GPL-2.0-or-later is sane or not, > in retrospect I believe that this concern in fact has no legal > basis. > > The license of a compiled application binary is the union of all > the object files linked into it. > > Notice I said "Object file" there, **NOT** "Source file". > > This is the crucial distinction that makes the presense of > GPL-2.0-or-later docs a non-issue from a licensing POV. > > > When the compiler takes the "MIT-0 and GPL-2.0-or-later" license > .go source file, and builds an object file, it will be discarding > all the API documentation comments. IOW, all the parts that were > under GPL-2.0-or-later are discarded. The only parts of the source > file that get "compiled" are the Go language constructs which were > MIT-0 licensed [1]. > > IOW, we have a "MIT-0 and GPL-2.0-or-later" .go source file, > and an "MIT-0" object file. > > Thus while there may be a human perception problem with the Golang > bindings being "MIT-0 and GPL-2.0-or-later", there are no legal > licensing limitations, as the combined work for a library or > application linking the bindings will only contain the MIT-0 part. > > The GPL-2.0-or-later docs won't influence the license of the > combined work. > > Note, this interpretation applies only to languages which are > compiled, not interpreted. > > If we are generating python code for example, the there is > no "source file" vs "object file" distinction for licensing. > The combined work in a python app is under the union of all > the source file licenses. > > > TL;DR: I think you can re-add the documentation comments to > the Golang code generator, declare the resulting code as being > "MIT-0 AND GPL-2.0-or-later". Yes. I think so too. I plan to add it back and address Andrea's last review and submit it in the next week or so. > In the README.md file we need todo two important things: > > * Document our interpretation of the "combined work" license > situation for applications consuming the project. > * Declare that *ALL* manually written code contributions > are exclusively MIT-0. > > With regards, > Daniel > > [1] NB, there are languages where some code comments can have semantic > impacts on the compiled out. I don't believe that to be the case for > any human targetted API docs that we would be copying over from the > QAPI schema though in the Golang case. Cheers, Victor
Hi, This is the second iteration of RFC v1: https://lists.gnu.org/archive/html/qemu-devel/2022-04/msg00226.html # What this is about? To generate a simple Golang interface that could communicate with QEMU over QMP. The Go code that is generated is meant to be used as the bare bones to exchange QMP messages. The goal is to have this as a Go module in QEMU gitlab namespace, similar to what have been done to pyhon-qemu-qmp https://gitlab.com/qemu-project/python-qemu-qmp # Issues raised in RFC v1 The leading '*' for issues I addressed in this iteration * 1) Documentation was removed to avoid License issues, by Daniel Thread: https://lists.nongnu.org/archive/html/qemu-devel/2022-05/msg01889.html It is important for the generated Go module to be compatible with Licenses used by projects that would be using this. Copying the documentation of the QAPI spec might conflict with GPLv2+. I have not proposed another license in this iteration, but I'm planning to go with MIT No Attribution, aka MIT-0 [0]. Does it make sense to bind the generated code's license to MIT-0 already at generator level? [0] https://github.com/aws/mit-0/blob/master/MIT-0 2) Inconsistent generated Names, by Andrea + Markus Thread: https://lists.gnu.org/archive/html/qemu-devel/2022-04/msg05026.html Example 1: | qapi | Go | Expected | | @logappend | Logappend | LogAppend | Example 2: (acronyms) VncInfo and DisplayReloadOptionsVNC This was not addressed in RFC v2 mainly because it seems to need more metadata support from the QAPI spec to handle specific scenarios. The solution seems either an extra metadata proposal by Andrea [1] or reviving Kevin's work [2] [1] https://lists.gnu.org/archive/html/qemu-devel/2022-05/msg00127.html [2] https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg04703.html * 3) Better type safety, by Andrea + Daniel Thread: https://lists.gnu.org/archive/html/qemu-devel/2022-05/msg01906.html Most of the 'Any' type (interface {}) has been removed. The only place it still exists is for fields that uses QAPI's any type, like with command qom-set or the struct type ObjectPropertyInfo. * 4) QAPI enums mapped to String instead of Int type, by Daniel. Thread: https://lists.gnu.org/archive/html/qemu-devel/2022-05/msg01904.html I'm still over the fence about using string here, mostly by the same issue reported here: https://gitlab.com/libvirt/libvirt-go-module/-/merge_requests/30#note_975517740 * 5) Events and Commands as interface, by Daniel Thread: https://lists.gnu.org/archive/html/qemu-devel/2022-05/msg01914.html So, instead of having a Command/Event struct with a Any type for the Arguments (which could be set with SetPasswordCommand struct type for example); now we have a Command interface which all previous structs that behaved as Arguments implement. I've included Marshal{Command Event} and Unmarshal{Command Event} helper functions that operate on top of each interface. * 6) Removing Any from Unions, by Daniel Thread: https://lists.gnu.org/archive/html/qemu-devel/2022-05/msg01912.html I basically followed the above suggestion to all other types that used Any. Specifically to unions were the removal of the 'discriminator' field, as proposed also in the above link. * 7) Flat structs by removing embed types. Discussion with Andrea Thread: https://lists.gnu.org/archive/html/qemu-devel/2022-05/msg01590.html No one required it but I decided to give it a try. Major issue that I see with this approach is to have generated a few 'Base' structs that are now useless. Overall, less nested structs seems better to me. Opnions? Example: | /* This is now useless, should be removed? */ | type InetSocketAddressBase struct { | Host string `json:"host"` | Port string `json:"port"` | } | | type InetSocketAddress struct { | // Base fields | Host string `json:"host"` | Port string `json:"port"` | | | Numeric *bool `json:"numeric,omitempty"` | To *uint16 `json:"to,omitempty"` | Ipv4 *bool `json:"ipv4,omitempty"` | Ipv6 *bool `json:"ipv6,omitempty"` | KeepAlive *bool `json:"keep-alive,omitempty"` | Mptcp *bool `json:"mptcp,omitempty"` | } 8) Supporting multiple versions Thread: https://lists.gnu.org/archive/html/qemu-devel/2022-05/msg02147.html I'm keen to working on the proposed solution above as it seems a good compromise to make code that can be compatible with multiple versions of qmp/qemu. But the basis needs to be defined first, so this is for the future. * 9) Handling { "error": { ... } } This was missing in the RFC v1. I've added a QapiError type that is included in all CommandReturn types, following Andrea's suggestion: https://lists.gnu.org/archive/html/qemu-devel/2022-04/msg02199.html # victortoso/qapi-go and victortoso/qemu I'm currently hosting the generated code from this RFCv2 at: https://gitlab.com/victortoso/qapi-go/-/tree/main/pkg/qapi This series can also be found at: https://gitlab.com/victortoso/qemu/-/commits/qapi-golang Thanks for taking a look, Victor Victor Toso (8): qapi: golang: Generate qapi's enum types in Go qapi: golang: Generate qapi's alternate types in Go qapi: golang: Generate qapi's struct types in Go qapi: golang: Generate qapi's union types in Go qapi: golang: Generate qapi's event types in Go qapi: golang: Generate qapi's command types in Go qapi: golang: Add CommandResult type to Go qapi: golang: document skip function visit_array_types scripts/qapi/golang.py | 765 +++++++++++++++++++++++++++++++++++++++++ scripts/qapi/main.py | 2 + 2 files changed, 767 insertions(+) create mode 100644 scripts/qapi/golang.py