| Message ID | 20240724221122.54601-1-jwyatt@redhat.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Add SWIG Bindings to libcpupower | expand |
On 7/24/24 16:11, John B. Wyatt IV wrote: > SWIG is a tool packaged in Fedora and other distros that can generate > bindings from C and C++ code for several languages including Python, > Perl, and Go. We at Red Hat are interested in adding binding support to > libcpupower so Python tools like rteval or tuned can make easy use of it. > > This RFC provides a limited subset of bindings as a demonstration. The second > commit provides a Python test script to verify the bindings. I wanted to get > feedback on this before implementing (and possibly testing) the entire library. > > The name raw_pylibcpupower is used because this is a demonstration example that > only provides direct bindings for a few functions. A wrapper `pylibcpupower` > may be needed to make the bindings more 'pythonic'. The bindings folder is used > because Go bindings may be useful for Kubernetes or OpenShift in the future. > > How should the bindings be built? The current example requires the makefile > in cpupower directory be run first to generate the .o files needed before > running the makefile in the python directory in a seperate step. Would the > maintainers prefer the two makefiles integrated? > > Another question is do you want more test files like the .py example? Would > this be used as part of a greater test suite? > > Note that while SWIG itself is GPL v3+ licensed; the resulting output, the > bindings code, is permissively licensed. Please see > https://swig.org/legal.html for more details. > > John B. Wyatt IV (2): > Add SWIG bindings files for libcpupower > Include test_raw_pylibcpupower.py > > Quick note to let you know that I will get to this once the merge window closes. thanks, -- Shuah
Hi, On 7/24/24 16:11, John B. Wyatt IV wrote: > SWIG is a tool packaged in Fedora and other distros that can generate > bindings from C and C++ code for several languages including Python, > Perl, and Go. We at Red Hat are interested in adding binding support to > libcpupower so Python tools like rteval or tuned can make easy use of it. > Can you elaborate on the use-case and what rteval currently does and how it could benefit from using libcpupower with the bindings? > This RFC provides a limited subset of bindings as a demonstration. The second > commit provides a Python test script to verify the bindings. I wanted to get > feedback on this before implementing (and possibly testing) the entire library. > > The name raw_pylibcpupower is used because this is a demonstration example that > only provides direct bindings for a few functions. A wrapper `pylibcpupower` > may be needed to make the bindings more 'pythonic'. The bindings folder is used > because Go bindings may be useful for Kubernetes or OpenShift in the future. > > How should the bindings be built? The current example requires the makefile > in cpupower directory be run first to generate the .o files needed before > running the makefile in the python directory in a seperate step. Would the > maintainers prefer the two makefiles integrated? I can't answer this question until I understand the licensing. However, I would lean towards keeping them separate. > > Another question is do you want more test files like the .py example? Would > this be used as part of a greater test suite? I would like to see document outlining the dependencies and examples of how this would be used. I see the README which says that "Next you will need to install SWIG. Using Fedora:" The document will have to include more than Fedora instructions. Instead of a README I would like to see a document. Before we go any further - we have the licensing implications to figure out. > > Note that while SWIG itself is GPL v3+ licensed; the resulting output, the > bindings code, is permissively licensed. Please see > https://swig.org/legal.html for more details. Adding Linus and Greg for their feedback and input on this proposal. What does it mean by "the resulting output, the bindings code is permissively licensed." I would like to get a better understanding of the licensing angle since this code adds dependency on SWIG which is GPL v3+ to build the proposed python bindings. thanks, -- Shuah
Addressing the licensing first: > On Tue, Jul 30, 2024 at 02:48:41PM -0600, Shuah Khan wrote: > > On 7/24/24 16:11, John B. Wyatt IV wrote: > > Note that while SWIG itself is GPL v3+ licensed; the resulting output, the > > bindings code, is permissively licensed. Please see > > https://swig.org/legal.html for more details. > > Adding Linus and Greg for their feedback and input on this proposal. > > What does it mean by "the resulting output, the bindings code is > permissively licensed." > > I would like to get a better understanding of the licensing angle > since this code adds dependency on SWIG which is GPL v3+ to build > the proposed python bindings. Copying and pasting from the link above: /begin The intention of the SWIG license is to ensure that the SWIG source code (the code that is compiled into the SWIG executable) remains free software by using the GPL license on the SWIG source code. SWIG is a code generator and the intention of the SWIG license is also to enable distribution of the output code under license terms of the user's choice/requirements. [snip html license links] When SWIG is used as it is distributed by the SWIG developers, its output is not governed by SWIG's license (including the GPL). SWIG's output contains code from three sources: * code generated by SWIG, which is not governed by copyright; * code copied from the SWIG library which is permissively licensed to be redistributed without restriction; * code derived from the user's input, which may be governed by the license of he code supplied by the user. So, while the input supplied to SWIG may affect the license of SWIG's output (e.g. if the input code is licensed under a copyleft or proprietary license), SWIG's license does not affect the license of the output. This is consistent with the FSF's FAQ entries on this subject (GPLOutput and WhatCaseIsOutputGPL), because the SWIG code copied into the output by SWIG is not GPL-licensed. /end The output of SWIG depends on SWIG's library which is permissively licensed + the license of the .o files used, which is libcpupower, which is GPLv2 licensed. Therefore, any bindings generated from libcpupower is GPL v2. SWIG Library and Examples license: The SWIG library and examples, under the Lib and Examples top level directories, are distributed under the following terms: You may copy, modify, distribute, and make derivative works based on this software, in source code or object code form, without restriction. If you distribute the software to others, you may do so according to the terms of your choice. This software is offered as is, without warranty of any kind. The snipped license links link to the GPL v3, SWIG's declaration of the GPL 3 or a later version with the above permissive declaration for output, and a link to four university licenses that all look permissive. https://swig.org/Release/LICENSE-UNIVERSITIES The resulting generated code can be linked and distributed with GPL v2 binaries. I am not a lawyer, but this is my understanding of what they declare on their website above. Please let me know if this clarifies this. > > On 7/24/24 16:11, John B. Wyatt IV wrote: > > SWIG is a tool packaged in Fedora and other distros that can generate > > bindings from C and C++ code for several languages including Python, > > Perl, and Go. We at Red Hat are interested in adding binding support to > > libcpupower so Python tools like rteval or tuned can make easy use of it. > > > > Can you elaborate on the use-case and what rteval currently does and > how it could benefit from using libcpupower with the bindings? rteval is a Python program used to measure realtime performance. We wanted to test the effect of enabling some levels of idle-stat to see how it affects latency, and didn't want to reinvent the wheel. We thought that the Python bindings could be useful to other people as well who might want to call cpupower too from Python. I did some testing and was able to achieve this with SWIG. We sent the patchset to see what folks thought about this. > > > > > Another question is do you want more test files like the .py example? Would > > this be used as part of a greater test suite? > > I would like to see document outlining the dependencies and examples of how > this would be used. I see the README which says that > > "Next you will need to install SWIG. Using Fedora:" > > The document will have to include more than Fedora instructions. Instead > of a README I would like to see a document. Understood. I wanted to get an idea of how we would structure the makefile first.
On Thu, Aug 01, 2024 at 05:24:20PM -0400, John B. Wyatt IV wrote: > > On 7/24/24 16:11, John B. Wyatt IV wrote: > > > SWIG is a tool packaged in Fedora and other distros that can generate > > > bindings from C and C++ code for several languages including Python, > > > Perl, and Go. We at Red Hat are interested in adding binding support to > > > libcpupower so Python tools like rteval or tuned can make easy use of it. > > > > > > > Can you elaborate on the use-case and what rteval currently does and > > how it could benefit from using libcpupower with the bindings? > > rteval is a Python program used to measure realtime performance. We wanted to > test the effect of enabling some levels of idle-stat to see how it affects > latency, and didn't want to reinvent the wheel. We thought that the Python > bindings could be useful to other people as well who might want to call > cpupower too from Python. I did some testing and was able to achieve this with > SWIG. We sent the patchset to see what folks thought about this. Is this going to require a built-time dependency on SWIG? If not, when would it be run, and who will be in charge of running it and updating the bindings? And finally, why do we need these at all? You are saying these are new tests that external tools will be using, but why, if external tools are required to run them, are they needed in the kernel tree at all? Why isn't this just another external test-suite that people who care about measuring this type of thing going to just run on their own if desired? thanks, greg k-h
Adding Arnaldo to the CC list. On Sun, Aug 04, 2024 at 10:54:10AM +0200, Greg Kroah-Hartman wrote: > On Thu, Aug 01, 2024 at 05:24:20PM -0400, John B. Wyatt IV wrote: > > > On 7/24/24 16:11, John B. Wyatt IV wrote: > > > > SWIG is a tool packaged in Fedora and other distros that can generate > > > > bindings from C and C++ code for several languages including Python, > > > > Perl, and Go. We at Red Hat are interested in adding binding support to > > > > libcpupower so Python tools like rteval or tuned can make easy use of it. > > > > > > > > > > Can you elaborate on the use-case and what rteval currently does and > > > how it could benefit from using libcpupower with the bindings? > > > > rteval is a Python program used to measure realtime performance. We wanted to > > test the effect of enabling some levels of idle-stat to see how it affects > > latency, and didn't want to reinvent the wheel. We thought that the Python > > bindings could be useful to other people as well who might want to call > > cpupower too from Python. I did some testing and was able to achieve this with > > SWIG. We sent the patchset to see what folks thought about this. > > Is this going to require a built-time dependency on SWIG? If not, when > would it be run It is optional, and based on my conversation with Shuah; the bindings will be in a seperate makefile. It would be ran after running cpupower's makefile, seperately. (But one can call the other.) > and who will be in charge of running it and updating the bindings? That would be myself. If I no longer wish to continue I would reassign it to another person here on the real-time team at Red Hat. John Kacur (whom I am working with on rteval) is fine with being listed as a backup contact. The bindings would need to be updated every time one of the functions or data structures listed in the .i file changes. But it can be as simple as copying the changed declaration from the header file to the .i file to resolve. > And finally, why do we need these at all? To provide bindings for Python programs like rteval to easily interface with libcpupower. It is very common for userspace programs to include bindings to scripting languages. > You are saying these are new > tests that external tools will be using, but why, if external tools are > required to run them, are they needed in the kernel tree at all? Why > isn't this just another external test-suite that people who care about > measuring this type of thing going to just run on their own if desired? SWIG the tool requires the .o files compiled from libcpupower to generate bindings. Since we need these artifacts from a packaging and usability perspective it makes sense to include the bindings source code along in a seperate directory with the cpupower source code to generate the builds for them at the same time. The source code is: the .i definitions file, any future Python wrapper helpers around the bindings needed by SWIG, and the documentation. Why SWIG? It was chosen for this project because of: 1) Many Python bindings generators are strickly Apache2 or GPL v3 licensed; and they may inject code into the bindings binary that will create a license conflict with the GPL v2. Swig was far more clear about the license status of the generated bindings. As described above: https://lore.kernel.org/linux-pm/1f5c24b6-f3ee-4863-8b7a-49344a550206@linuxfoundation.org/T/#mb7170232fb429fc242bb45c8d3d4d5ed47f0c59f 2) It's extensive documentation. This is the Python section: https://www.swig.org/Doc4.2/Python.html#Python 3) Support: SWIG has been around since 1996 and had a release 5 months ago. It's length of time and the acitvity in it's community makes it unlikely SWIG will lose support by it's developers anytime soon. 4) This RFC demonstrates how quickly and with little work you can generate bindings for an existing C project with no code changes to that project. :-) There were issues getting other Python bindings generators to work as described in their documentation.
On Sun, Aug 04, 2024 at 10:54:10AM +0200, Greg Kroah-Hartman wrote: > On Thu, Aug 01, 2024 at 05:24:20PM -0400, John B. Wyatt IV wrote: > > > On 7/24/24 16:11, John B. Wyatt IV wrote: > > > > SWIG is a tool packaged in Fedora and other distros that can generate > > > > bindings from C and C++ code for several languages including Python, > > > > Perl, and Go. We at Red Hat are interested in adding binding support to > > > > libcpupower so Python tools like rteval or tuned can make easy use of it. > > > > > > > > > > Can you elaborate on the use-case and what rteval currently does and > > > how it could benefit from using libcpupower with the bindings? > > > > rteval is a Python program used to measure realtime performance. We wanted to > > test the effect of enabling some levels of idle-stat to see how it affects > > latency, and didn't want to reinvent the wheel. We thought that the Python > > bindings could be useful to other people as well who might want to call > > cpupower too from Python. I did some testing and was able to achieve this with > > SWIG. We sent the patchset to see what folks thought about this. > Is this going to require a built-time dependency on SWIG? If not, when > would it be run, and who will be in charge of running it and updating > the bindings? > And finally, why do we need these at all? You are saying these are new > tests that external tools will be using, but why, if external tools are > required to run them, are they needed in the kernel tree at all? Why > isn't this just another external test-suite that people who care about > measuring this type of thing going to just run on their own if desired? We have a python binding for perf, it was done manually, didn't use something like swig, don't recall why I made that decision at the time, maybe not to get one extra build dependency, as you mention. I did it at the time as another tool developed at Red Hat, tuna, a top like tool, using GTK+, could use the perf infrastructure to get notifications about new threads and threads exiting, as exemplified by test tools in the kernel sources: root@number:/home/acme/git/linux# tools/perf/python/twatch.py cpu: 24, pid: 2787405, tid: 2787405 { type: fork, pid: 2787405, ppid: 2787405, tid: 673409, ptid: 2787405, time: 300460407945015} cpu: 23, pid: 2787405, tid: 673409 { type: comm, pid: 2787405, tid: 673409, comm: StreamT~s #1624 } cpu: 4, pid: 3923381, tid: 3923381 { type: fork, pid: 3923381, ppid: 3923381, tid: 673410, ptid: 3923381, time: 300460862312442} cpu: 23, pid: 2787405, tid: 673409 { type: comm, pid: 2787405, tid: 673409, comm: StreamT~s #1624 } cpu: 23, pid: 3923381, tid: 673410 { type: comm, pid: 3923381, tid: 673410, comm: StreamT~s #3197 } cpu: 23, pid: 3923381, tid: 673410 { type: comm, pid: 3923381, tid: 673410, comm: StreamT~s #3197 } cpu: 13, pid: 2787464, tid: 2787464 { type: fork, pid: 2787464, ppid: 2787464, tid: 673411, ptid: 2787464, time: 300461205531219} cpu: 26, pid: 2787464, tid: 673411 { type: comm, pid: 2787464, tid: 673411, comm: StreamT~s #1624 } cpu: 26, pid: 2787464, tid: 673411 { type: comm, pid: 2787464, tid: 673411, comm: StreamT~s #1624 } ^CTraceback (most recent call last): File "/home/acme/git/linux/tools/perf/python/twatch.py", line 61, in <module> main() File "/home/acme/git/linux/tools/perf/python/twatch.py", line 33, in main evlist.poll(timeout = -1) KeyboardInterrupt root@number:/home/acme/git/linux# root@number:/home/acme/git/linux# head tools/perf/python/twatch.py #! /usr/bin/env python # SPDX-License-Identifier: GPL-2.0-only # -*- python -*- # -*- coding: utf-8 -*- # twatch - Experimental use of the perf python interface # Copyright (C) 2011 Arnaldo Carvalho de Melo <acme@redhat.com> # import perf root@number:/home/acme/git/linux# Another root@number:/home/acme/git/linux# tools/perf/python/tracepoint.py | head -50 | tail time 302243176736731 prev_comm=chromium-browse prev_pid=633048 prev_prio=120 prev_state=0x1 ==> next_comm=swapper/8 next_pid=0 next_prio=120 time 302243177970076 prev_comm=swapper/10 prev_pid=0 prev_prio=120 prev_state=0x0 ==> next_comm=python next_pid=676159 next_prio=120 time 302243177952044 prev_comm=Compositor prev_pid=660727 prev_prio=120 prev_state=0x1 ==> next_comm=migration/17 next_pid=74 next_prio=0 time 302243176742113 prev_comm=swapper/8 prev_pid=0 prev_prio=120 prev_state=0x0 ==> next_comm=chromium-browse next_pid=633048 next_prio=120 time 302243177991642 prev_comm=python prev_pid=676159 prev_prio=120 prev_state=0x1 ==> next_comm=swapper/10 next_pid=0 next_prio=120 time 302243177962820 prev_comm=migration/17 prev_pid=74 prev_prio=0 prev_state=0x1 ==> next_comm=swapper/17 next_pid=0 next_prio=120 time 302243176784747 prev_comm=chromium-browse prev_pid=633048 prev_prio=120 prev_state=0x1 ==> next_comm=swapper/8 next_pid=0 next_prio=120 time 302243177993435 prev_comm=swapper/10 prev_pid=0 prev_prio=120 prev_state=0x0 ==> next_comm=python next_pid=676159 next_prio=120 time 302243177966620 prev_comm=swapper/17 prev_pid=0 prev_prio=120 prev_state=0x0 ==> next_comm=Compositor next_pid=660727 next_prio=120 time 302243176999285 prev_comm=swapper/8 prev_pid=0 prev_prio=120 prev_state=0x0 ==> next_comm=chromium-browse next_pid=633048 next_prio=120 root@number:/home/acme/git/linux# By now I think there are other tools that use that python binding out in the wild and it is packaged by distros as: root@number:/home/acme/git/linux# dnf search python3-perf ============================================================================= Name Exactly Matched: python3-perf ============================================================================== python3-perf.x86_64 : Python bindings for apps which will manipulate perf events root@number:/home/acme/git/linux# Having these bindings opens the doors for projects to use these libraries provided via the kernel sources, allowing other languages to be used than C. Now we have multiple libraries in the kernel sources, if we could have a convenient way to provide those bindings for python, Go, etc, that could help more projects while reducing the cost of supporting those bindings by having more people capable of fixing up when needed as we would be using a common way of providing such bindings. Cheers, - Arnaldo
I wanted to follow up on this since I am close to sending out the v2 of this patchset. 3 points I wanted to raise: 1) Does everyone understand, is okay with the SWIG license, and wants to proceed with me sending in a more complete version of this as a candidate for upstreaming? 2) About maintainership: if I am to be the maintainer of this, how would myself and John Kacur be listed? As a CPU POWER MONITORING SUBSYSTEM maintainer, a separate category below it called CPU POWER MONITORING SUBSYSTEM PYTHON BINDINGS maintainer, or is not needed to be listed at this time? A quick search for bindings shows what I believe to be all of them for device tree. This may establish a new precedent. If I was to be added, I assume it would be a separate commit in the v2 submission? 3) I had to comment out powercap_set_enabled SWIG reported this symbol not being found despite being in powercap.h. I did a quick search and was not able to find it's implementation in powercap.c. The get equivalent powercap_get_enabled is in powercap.c. Wanted to check on this just in case it is a bug or part of future functionality. I am assuming the latter; I would send it v2 with that declaration commented out with a note explaining it for users if there is no objection.
On 8/20/24 00:40, John B. Wyatt IV wrote: > I wanted to follow up on this since I am close to sending out the v2 of > this patchset. > > 3 points I wanted to raise: > > 1) Does everyone understand, is okay with the SWIG license, and wants to > proceed with me sending in a more complete version of this as a candidate for > upstreaming? > > 2) About maintainership: if I am to be the maintainer of this, how would > myself and John Kacur be listed? As a CPU POWER MONITORING SUBSYSTEM > maintainer, a separate category below it called CPU POWER MONITORING SUBSYSTEM > PYTHON BINDINGS maintainer, or is not needed to be listed at this time? > > A quick search for bindings shows what I believe to be all of them for device > tree. This may establish a new precedent. > > If I was to be added, I assume it would be a separate commit in the v2 > submission? > > 3) I had to comment out powercap_set_enabled > > SWIG reported this symbol not being found despite being in powercap.h. I did a > quick search and was not able to find it's implementation in powercap.c. The > get equivalent powercap_get_enabled is in powercap.c. Wanted to check on this > just in case it is a bug or part of future functionality. I am assuming the > latter; I would send it v2 with that declaration commented out with a note > explaining it for users if there is no objection. Yes please send a complete version for review. thanks, -- Shuah
On 8/21/24 01:08, Shuah Khan wrote: > On 8/20/24 00:40, John B. Wyatt IV wrote: >> I wanted to follow up on this since I am close to sending out the v2 of >> this patchset. >> Here is more complete response after my quick response. >> 3 points I wanted to raise: >> >> 1) Does everyone understand, is okay with the SWIG license, and wants to >> proceed with me sending in a more complete version of this as a candidate for >> upstreaming? >> Yes - send patch series without RFC tag. >> 2) About maintainership: if I am to be the maintainer of this, how would >> myself and John Kacur be listed? As a CPU POWER MONITORING SUBSYSTEM >> maintainer, a separate category below it called CPU POWER MONITORING SUBSYSTEM >> PYTHON BINDINGS maintainer, or is not needed to be listed at this time? >> Let's not add a new category at this time. You can add yourself and John Kacur be Reviewers under CPU POWER MONITORING SUBSYSTEM for now. We can revisit as we go forward. >> A quick search for bindings shows what I believe to be all of them for device >> tree. This may establish a new precedent. >> I don't fully understand the above. Will this patch series include device tree changes? I didn't get that from the RFC. >> If I was to be added, I assume it would be a separate commit in the v2 >> submission? >> >> 3) I had to comment out powercap_set_enabled >> >> SWIG reported this symbol not being found despite being in powercap.h. I did a >> quick search and was not able to find it's implementation in powercap.c. The >> get equivalent powercap_get_enabled is in powercap.c. Wanted to check on this >> just in case it is a bug or part of future functionality. I am assuming the >> latter; I would send it v2 with that declaration commented out with a note >> explaining it for users if there is no objection. Good find. It is missing functionality. For now you can define it in powercap.c to simply return 0 with a TODO thanks, -- Shuah
On Wed, Aug 21, 2024 at 09:25:52PM -0600, Shuah Khan wrote: > On 8/21/24 01:08, Shuah Khan wrote: > > On 8/20/24 00:40, John B. Wyatt IV wrote: > > > A quick search for bindings shows what I believe to be all of them for device > > > tree. This may establish a new precedent. > > > > > I don't fully understand the above. Will this patch series include device tree > changes? I didn't get that from the RFC. I miswrote. I meant I could not find anything related to bindings for a user-space program in the kernel repo. Device-bindings was the closest I could find and it is not relevant. My apologies. Thank you for asking about that.