golang/xenlight: Add libxl_utils support
diff mbox series

Message ID 20190718215428.6727-1-george.dunlap@citrix.com
State New
Headers show
Series
  • golang/xenlight: Add libxl_utils support
Related show

Commit Message

George Dunlap July 18, 2019, 9:54 p.m. UTC
The Go bindings for libxl miss functions from libxl_utils, let's start
with the simple libxl_domid_to_name and its counterpart
libxl_name_to_domid.

NB that C.GoString() will return "" if it's passed a NULL; see
https://github.com/golang/go/issues/32734#issuecomment-506835432

Signed-off-by: Nicolas Belouin <nicolas.belouin@gandi.net>
Signed-off-by: George Dunlap <george.dunlap@citrix.com>
---
v3:
 - Wire into build system
 - Add reference to C.GoString() handling NULL to commit message

Nicolas, could you test to see if this actually works for you?

It would be really good also if we could get something that would do
basic unit testing on a live system, and get that running in osstest.

CC: Nicolas Belouin <nicolas.belouin@gandi.net>
CC: Ian Jackson <ian.jackson@citrix.com>
CC: Wei Liu <wl@xen.org>
---
 tools/golang/xenlight/Makefile          |  2 +-
 tools/golang/xenlight/xenlight_utils.go | 55 +++++++++++++++++++++++++
 2 files changed, 56 insertions(+), 1 deletion(-)
 create mode 100644 tools/golang/xenlight/xenlight_utils.go

Comments

Nicolas Belouin July 19, 2019, 7:34 a.m. UTC | #1
On 7/18/19 11:54 PM, George Dunlap wrote:
> The Go bindings for libxl miss functions from libxl_utils, let's start
> with the simple libxl_domid_to_name and its counterpart
> libxl_name_to_domid.
>
> NB that C.GoString() will return "" if it's passed a NULL; see
> https://github.com/golang/go/issues/32734#issuecomment-506835432
>
> Signed-off-by: Nicolas Belouin <nicolas.belouin@gandi.net>
> Signed-off-by: George Dunlap <george.dunlap@citrix.com>
> ---
> v3:
>  - Wire into build system
>  - Add reference to C.GoString() handling NULL to commit message
>
> Nicolas, could you test to see if this actually works for you?
Tested it, it works.

I must confess I do not use that import path as the new modules mechanism
introduced in Go1.11 downloads and compile a versioned copy of every
dependency per project, and this behavior is incompatible with the build
system used here.

Nicolas
George Dunlap July 19, 2019, 8:47 a.m. UTC | #2
> On Jul 19, 2019, at 8:34 AM, Nicolas Belouin <nicolas.belouin@gandi.net> wrote:
> 
> 
> 
> On 7/18/19 11:54 PM, George Dunlap wrote:
>> The Go bindings for libxl miss functions from libxl_utils, let's start
>> with the simple libxl_domid_to_name and its counterpart
>> libxl_name_to_domid.
>> 
>> NB that C.GoString() will return "" if it's passed a NULL; see
>> https://github.com/golang/go/issues/32734#issuecomment-506835432
>> 
>> Signed-off-by: Nicolas Belouin <nicolas.belouin@gandi.net>
>> Signed-off-by: George Dunlap <george.dunlap@citrix.com>
>> ---
>> v3:
>> - Wire into build system
>> - Add reference to C.GoString() handling NULL to commit message
>> 
>> Nicolas, could you test to see if this actually works for you?
> Tested it, it works.
> 
> I must confess I do not use that import path as the new modules mechanism
> introduced in Go1.11 downloads and compile a versioned copy of every
> dependency per project, and this behavior is incompatible with the build
> system used here.

It’s possible that something fundamentally has changed, but I suspect that rather you don’t quite understand how the current build system is supposed to work.  (In which case a write-up in the tree would probably be useful.)

Go has always insisted that there be no binary compatibility between versions; so it’s always been necessary to re-compile all your libraries when upgrading from (say) 1.8 to 1.9.  Which means that any useable distribution must also include all the source files necessary to recompile when you bump the version number.

So the core mechanism of the “install” is actually to copy all the source files necessary into the right local directory such that the go compiler can find them; ATM this is /usr/share/gocode/golang.xenproject.org/xenlight (as documented here: https://github.com/golang/go/wiki/FileTreeDocumentation).  The files which are copied are specified by the PGKSOURCES variable in the makefile.

The purpose of building a binary during the normal “make” process is not actually to have a useable binary; it’s to detect any potential issues when you do the Xen build, rather than whenever you actually build a Go program for the first time (potentially on some end-user’s system).  And “detect any issues” attempts to use “go build” in the situation you expect to use it at run time (i.e., it copies PKGSOURCES to a local directory, then points GO_PATH at them, points to the locally-built libraries, and then runs a plain `go build`).  In this way it should, theoretically, catch any of the following errors:

* Syntax / whatever errors in the Go bindings themselves
* Mismatches between the Go bindings and libxl &c
* Missing system libraries
* Something not right in the installation path system

All that to say — the “real” way to enable a new file in the xenlight package is to add it to PKGSOURCES.

 -George
George Dunlap July 19, 2019, 8:50 a.m. UTC | #3
> On Jul 19, 2019, at 9:47 AM, George Dunlap <george.dunlap@citrix.com> wrote:
> 
> 
> 
>> On Jul 19, 2019, at 8:34 AM, Nicolas Belouin <nicolas.belouin@gandi.net> wrote:
>> 
>> 
>> 
>> On 7/18/19 11:54 PM, George Dunlap wrote:
>>> The Go bindings for libxl miss functions from libxl_utils, let's start
>>> with the simple libxl_domid_to_name and its counterpart
>>> libxl_name_to_domid.
>>> 
>>> NB that C.GoString() will return "" if it's passed a NULL; see
>>> https://github.com/golang/go/issues/32734#issuecomment-506835432
>>> 
>>> Signed-off-by: Nicolas Belouin <nicolas.belouin@gandi.net>
>>> Signed-off-by: George Dunlap <george.dunlap@citrix.com>
>>> ---
>>> v3:
>>> - Wire into build system
>>> - Add reference to C.GoString() handling NULL to commit message
>>> 
>>> Nicolas, could you test to see if this actually works for you?
>> Tested it, it works.
>> 
>> I must confess I do not use that import path as the new modules mechanism
>> introduced in Go1.11 downloads and compile a versioned copy of every
>> dependency per project, and this behavior is incompatible with the build
>> system used here.
> 
> It’s possible that something fundamentally has changed, but I suspect that rather you don’t quite understand how the current build system is supposed to work.  (In which case a write-up in the tree would probably be useful.)
> 
> Go has always insisted that there be no binary compatibility between versions; so it’s always been necessary to re-compile all your libraries when upgrading from (say) 1.8 to 1.9.  Which means that any useable distribution must also include all the source files necessary to recompile when you bump the version number.
> 
> So the core mechanism of the “install” is actually to copy all the source files necessary into the right local directory such that the go compiler can find them; ATM this is /usr/share/gocode/golang.xenproject.org/xenlight

Nit:  This of course should have a `src/` between `gocode/` and `golang.xenproject.org/`.

NB also that this naming scheme was designed so that at some point in the future, we could actually host the xenlight packages at the URL provided.

 -George
George Dunlap July 19, 2019, 10:01 a.m. UTC | #4
On 7/19/19 8:34 AM, Nicolas Belouin wrote:
> 
> 
> On 7/18/19 11:54 PM, George Dunlap wrote:
>> The Go bindings for libxl miss functions from libxl_utils, let's start
>> with the simple libxl_domid_to_name and its counterpart
>> libxl_name_to_domid.
>>
>> NB that C.GoString() will return "" if it's passed a NULL; see
>> https://github.com/golang/go/issues/32734#issuecomment-506835432
>>
>> Signed-off-by: Nicolas Belouin <nicolas.belouin@gandi.net>
>> Signed-off-by: George Dunlap <george.dunlap@citrix.com>
>> ---
>> v3:
>>  - Wire into build system
>>  - Add reference to C.GoString() handling NULL to commit message
>>
>> Nicolas, could you test to see if this actually works for you?
> Tested it, it works.
> 
> I must confess I do not use that import path as the new modules mechanism
> introduced in Go1.11 downloads and compile a versioned copy of every
> dependency per project, and this behavior is incompatible with the build
> system used here.

Also -- I can't check this in without somebody acking my changes; and
you're the most obvious person to do so. :-)

 -George
Nicolas Belouin July 19, 2019, 10:03 a.m. UTC | #5
On 7/19/19 10:50 AM, George Dunlap wrote:
>
>> On Jul 19, 2019, at 9:47 AM, George Dunlap <george.dunlap@citrix.com> wrote:
>>
>>
>>
>>> On Jul 19, 2019, at 8:34 AM, Nicolas Belouin <nicolas.belouin@gandi.net> wrote:
>>>
>>>
>>>
>>> On 7/18/19 11:54 PM, George Dunlap wrote:
>>>> The Go bindings for libxl miss functions from libxl_utils, let's start
>>>> with the simple libxl_domid_to_name and its counterpart
>>>> libxl_name_to_domid.
>>>>
>>>> NB that C.GoString() will return "" if it's passed a NULL; see
>>>> https://github.com/golang/go/issues/32734#issuecomment-506835432
>>>>
>>>> Signed-off-by: Nicolas Belouin <nicolas.belouin@gandi.net>
>>>> Signed-off-by: George Dunlap <george.dunlap@citrix.com>
>>>> ---
>>>> v3:
>>>> - Wire into build system
>>>> - Add reference to C.GoString() handling NULL to commit message
>>>>
>>>> Nicolas, could you test to see if this actually works for you?
>>> Tested it, it works.
>>>
>>> I must confess I do not use that import path as the new modules mechanism
>>> introduced in Go1.11 downloads and compile a versioned copy of every
>>> dependency per project, and this behavior is incompatible with the build
>>> system used here.
>> It’s possible that something fundamentally has changed, but I suspect that rather you don’t quite understand how the current build system is supposed to work.  (In which case a write-up in the tree would probably be useful.)
>>
>> Go has always insisted that there be no binary compatibility between versions; so it’s always been necessary to re-compile all your libraries when upgrading from (say) 1.8 to 1.9.  Which means that any useable distribution must also include all the source files necessary to recompile when you bump the version number.
>>
>> So the core mechanism of the “install” is actually to copy all the source files necessary into the right local directory such that the go compiler can find them; ATM this is /usr/share/gocode/golang.xenproject.org/xenlight
> Nit:  This of course should have a `src/` between `gocode/` and `golang.xenproject.org/`.
>
> NB also that this naming scheme was designed so that at some point in the future, we could actually host the xenlight packages at the URL provided.
>
>  -George
>

This new mechanism of modules is described here:
https://golang.org/cmd/go/#hdr-Modules__module_versions__and_more

The module system is intended to supersede the GOPATH approach and
provide a way to get versioned dependencies, as such
it does not rely on GOPATH at all and doesn't use sources or compiled
packages present in GOPATH elements such as /usr/share/gocode
and systematically fetch (at the asked version) and compile a copy of
the dependency as it might be a different version from the one
in GOPATH.

As far as I tried, I have been unable to build my module even with the
library installed.
I have to use xenbits.xen.org/git-http/xen.git/tools/golang/xenlight (or
one of its mirror) in order to build the module using the new
mechanism (the golang.xenproject.org/xenlight works when building with
modules mode disabled).

Nicolas
George Dunlap July 19, 2019, 10:09 a.m. UTC | #6
On 7/19/19 11:03 AM, Nicolas Belouin wrote:
> 
> 
> On 7/19/19 10:50 AM, George Dunlap wrote:
>>
>>> On Jul 19, 2019, at 9:47 AM, George Dunlap <george.dunlap@citrix.com> wrote:
>>>
>>>
>>>
>>>> On Jul 19, 2019, at 8:34 AM, Nicolas Belouin <nicolas.belouin@gandi.net> wrote:
>>>>
>>>>
>>>>
>>>> On 7/18/19 11:54 PM, George Dunlap wrote:
>>>>> The Go bindings for libxl miss functions from libxl_utils, let's start
>>>>> with the simple libxl_domid_to_name and its counterpart
>>>>> libxl_name_to_domid.
>>>>>
>>>>> NB that C.GoString() will return "" if it's passed a NULL; see
>>>>> https://github.com/golang/go/issues/32734#issuecomment-506835432
>>>>>
>>>>> Signed-off-by: Nicolas Belouin <nicolas.belouin@gandi.net>
>>>>> Signed-off-by: George Dunlap <george.dunlap@citrix.com>
>>>>> ---
>>>>> v3:
>>>>> - Wire into build system
>>>>> - Add reference to C.GoString() handling NULL to commit message
>>>>>
>>>>> Nicolas, could you test to see if this actually works for you?
>>>> Tested it, it works.
>>>>
>>>> I must confess I do not use that import path as the new modules mechanism
>>>> introduced in Go1.11 downloads and compile a versioned copy of every
>>>> dependency per project, and this behavior is incompatible with the build
>>>> system used here.
>>> It’s possible that something fundamentally has changed, but I suspect that rather you don’t quite understand how the current build system is supposed to work.  (In which case a write-up in the tree would probably be useful.)
>>>
>>> Go has always insisted that there be no binary compatibility between versions; so it’s always been necessary to re-compile all your libraries when upgrading from (say) 1.8 to 1.9.  Which means that any useable distribution must also include all the source files necessary to recompile when you bump the version number.
>>>
>>> So the core mechanism of the “install” is actually to copy all the source files necessary into the right local directory such that the go compiler can find them; ATM this is /usr/share/gocode/golang.xenproject.org/xenlight
>> Nit:  This of course should have a `src/` between `gocode/` and `golang.xenproject.org/`.
>>
>> NB also that this naming scheme was designed so that at some point in the future, we could actually host the xenlight packages at the URL provided.
>>
>>  -George
>>
> 
> This new mechanism of modules is described here:
> https://golang.org/cmd/go/#hdr-Modules__module_versions__and_more
> 
> The module system is intended to supersede the GOPATH approach and
> provide a way to get versioned dependencies, as such
> it does not rely on GOPATH at all and doesn't use sources or compiled
> packages present in GOPATH elements such as /usr/share/gocode
> and systematically fetch (at the asked version) and compile a copy of
> the dependency as it might be a different version from the one
> in GOPATH.
> 
> As far as I tried, I have been unable to build my module even with the
> library installed.
> I have to use xenbits.xen.org/git-http/xen.git/tools/golang/xenlight (or
> one of its mirror) in order to build the module using the new
> mechanism (the golang.xenproject.org/xenlight works when building with
> modules mode disabled).

I took a look at the module stuff when it came out, and I was never able
to make sense of how it was supposed to work.

<rant>On the whole, it seems they basically hate the idea of distro
packages, and seem intent on breaking them whenever people manage to
start to get them working.</rant>

 -George
Nicolas Belouin July 19, 2019, 10:24 a.m. UTC | #7
On 7/19/19 12:09 PM, George Dunlap wrote:
> On 7/19/19 11:03 AM, Nicolas Belouin wrote:
>>
>> On 7/19/19 10:50 AM, George Dunlap wrote:
>>>> On Jul 19, 2019, at 9:47 AM, George Dunlap <george.dunlap@citrix.com> wrote:
>>>>
>>>>
>>>>
>>>>> On Jul 19, 2019, at 8:34 AM, Nicolas Belouin <nicolas.belouin@gandi.net> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On 7/18/19 11:54 PM, George Dunlap wrote:
>>>>>> The Go bindings for libxl miss functions from libxl_utils, let's start
>>>>>> with the simple libxl_domid_to_name and its counterpart
>>>>>> libxl_name_to_domid.
>>>>>>
>>>>>> NB that C.GoString() will return "" if it's passed a NULL; see
>>>>>> https://github.com/golang/go/issues/32734#issuecomment-506835432
>>>>>>
>>>>>> Signed-off-by: Nicolas Belouin <nicolas.belouin@gandi.net>
>>>>>> Signed-off-by: George Dunlap <george.dunlap@citrix.com>
>>>>>> ---
>>>>>> v3:
>>>>>> - Wire into build system
>>>>>> - Add reference to C.GoString() handling NULL to commit message
>>>>>>
>>>>>> Nicolas, could you test to see if this actually works for you?
>>>>> Tested it, it works.
>>>>>
>>>>> I must confess I do not use that import path as the new modules mechanism
>>>>> introduced in Go1.11 downloads and compile a versioned copy of every
>>>>> dependency per project, and this behavior is incompatible with the build
>>>>> system used here.
>>>> It’s possible that something fundamentally has changed, but I suspect that rather you don’t quite understand how the current build system is supposed to work.  (In which case a write-up in the tree would probably be useful.)
>>>>
>>>> Go has always insisted that there be no binary compatibility between versions; so it’s always been necessary to re-compile all your libraries when upgrading from (say) 1.8 to 1.9.  Which means that any useable distribution must also include all the source files necessary to recompile when you bump the version number.
>>>>
>>>> So the core mechanism of the “install” is actually to copy all the source files necessary into the right local directory such that the go compiler can find them; ATM this is /usr/share/gocode/golang.xenproject.org/xenlight
>>> Nit:  This of course should have a `src/` between `gocode/` and `golang.xenproject.org/`.
>>>
>>> NB also that this naming scheme was designed so that at some point in the future, we could actually host the xenlight packages at the URL provided.
>>>
>>>  -George
>>>
>> This new mechanism of modules is described here:
>> https://golang.org/cmd/go/#hdr-Modules__module_versions__and_more
>>
>> The module system is intended to supersede the GOPATH approach and
>> provide a way to get versioned dependencies, as such
>> it does not rely on GOPATH at all and doesn't use sources or compiled
>> packages present in GOPATH elements such as /usr/share/gocode
>> and systematically fetch (at the asked version) and compile a copy of
>> the dependency as it might be a different version from the one
>> in GOPATH.
>>
>> As far as I tried, I have been unable to build my module even with the
>> library installed.
>> I have to use xenbits.xen.org/git-http/xen.git/tools/golang/xenlight (or
>> one of its mirror) in order to build the module using the new
>> mechanism (the golang.xenproject.org/xenlight works when building with
>> modules mode disabled).
> I took a look at the module stuff when it came out, and I was never able
> to make sense of how it was supposed to work.
Basically it is the same idea than a python virtualenv with
|include-system-site-packages set to false: never use what is provided
by the system and download everything in the exact version the manifest
tells you to.
|
> <rant>On the whole, it seems they basically hate the idea of distro
> packages, and seem intent on breaking them whenever people manage to
> start to get them working.</rant>
Actually yes because they don't want to be bound to the version provided
by the distro (I will not enter the debate of whether it is a good thing
or not)

Nicolas
Nicolas Belouin July 19, 2019, 10:28 a.m. UTC | #8
On 7/19/19 12:01 PM, George Dunlap wrote:
> On 7/19/19 8:34 AM, Nicolas Belouin wrote:
>>
>> On 7/18/19 11:54 PM, George Dunlap wrote:
>>> The Go bindings for libxl miss functions from libxl_utils, let's start
>>> with the simple libxl_domid_to_name and its counterpart
>>> libxl_name_to_domid.
>>>
>>> NB that C.GoString() will return "" if it's passed a NULL; see
>>> https://github.com/golang/go/issues/32734#issuecomment-506835432
>>>
>>> Signed-off-by: Nicolas Belouin <nicolas.belouin@gandi.net>
>>> Signed-off-by: George Dunlap <george.dunlap@citrix.com>
>>> ---
>>> v3:
>>>  - Wire into build system
>>>  - Add reference to C.GoString() handling NULL to commit message
>>>
>>> Nicolas, could you test to see if this actually works for you?
>> Tested it, it works.
>>
>> I must confess I do not use that import path as the new modules mechanism
>> introduced in Go1.11 downloads and compile a versioned copy of every
>> dependency per project, and this behavior is incompatible with the build
>> system used here.
> Also -- I can't check this in without somebody acking my changes; and
> you're the most obvious person to do so. :-)
>
>  -George
I successfully tested your changes so consider them as acked.

Nicolas
George Dunlap July 19, 2019, 10:35 a.m. UTC | #9
On 7/19/19 11:24 AM, Nicolas Belouin wrote:
> 
> 
> On 7/19/19 12:09 PM, George Dunlap wrote:
>> On 7/19/19 11:03 AM, Nicolas Belouin wrote:
>>>
>>> On 7/19/19 10:50 AM, George Dunlap wrote:
>>>>> On Jul 19, 2019, at 9:47 AM, George Dunlap <george.dunlap@citrix.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>>> On Jul 19, 2019, at 8:34 AM, Nicolas Belouin <nicolas.belouin@gandi.net> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 7/18/19 11:54 PM, George Dunlap wrote:
>>>>>>> The Go bindings for libxl miss functions from libxl_utils, let's start
>>>>>>> with the simple libxl_domid_to_name and its counterpart
>>>>>>> libxl_name_to_domid.
>>>>>>>
>>>>>>> NB that C.GoString() will return "" if it's passed a NULL; see
>>>>>>> https://github.com/golang/go/issues/32734#issuecomment-506835432
>>>>>>>
>>>>>>> Signed-off-by: Nicolas Belouin <nicolas.belouin@gandi.net>
>>>>>>> Signed-off-by: George Dunlap <george.dunlap@citrix.com>
>>>>>>> ---
>>>>>>> v3:
>>>>>>> - Wire into build system
>>>>>>> - Add reference to C.GoString() handling NULL to commit message
>>>>>>>
>>>>>>> Nicolas, could you test to see if this actually works for you?
>>>>>> Tested it, it works.
>>>>>>
>>>>>> I must confess I do not use that import path as the new modules mechanism
>>>>>> introduced in Go1.11 downloads and compile a versioned copy of every
>>>>>> dependency per project, and this behavior is incompatible with the build
>>>>>> system used here.
>>>>> It’s possible that something fundamentally has changed, but I suspect that rather you don’t quite understand how the current build system is supposed to work.  (In which case a write-up in the tree would probably be useful.)
>>>>>
>>>>> Go has always insisted that there be no binary compatibility between versions; so it’s always been necessary to re-compile all your libraries when upgrading from (say) 1.8 to 1.9.  Which means that any useable distribution must also include all the source files necessary to recompile when you bump the version number.
>>>>>
>>>>> So the core mechanism of the “install” is actually to copy all the source files necessary into the right local directory such that the go compiler can find them; ATM this is /usr/share/gocode/golang.xenproject.org/xenlight
>>>> Nit:  This of course should have a `src/` between `gocode/` and `golang.xenproject.org/`.
>>>>
>>>> NB also that this naming scheme was designed so that at some point in the future, we could actually host the xenlight packages at the URL provided.
>>>>
>>>>  -George
>>>>
>>> This new mechanism of modules is described here:
>>> https://golang.org/cmd/go/#hdr-Modules__module_versions__and_more
>>>
>>> The module system is intended to supersede the GOPATH approach and
>>> provide a way to get versioned dependencies, as such
>>> it does not rely on GOPATH at all and doesn't use sources or compiled
>>> packages present in GOPATH elements such as /usr/share/gocode
>>> and systematically fetch (at the asked version) and compile a copy of
>>> the dependency as it might be a different version from the one
>>> in GOPATH.
>>>
>>> As far as I tried, I have been unable to build my module even with the
>>> library installed.
>>> I have to use xenbits.xen.org/git-http/xen.git/tools/golang/xenlight (or
>>> one of its mirror) in order to build the module using the new
>>> mechanism (the golang.xenproject.org/xenlight works when building with
>>> modules mode disabled).
>> I took a look at the module stuff when it came out, and I was never able
>> to make sense of how it was supposed to work.
> Basically it is the same idea than a python virtualenv with
> |include-system-site-packages set to false: never use what is provided
> by the system and download everything in the exact version the manifest
> tells you to.
> |
>> <rant>On the whole, it seems they basically hate the idea of distro
>> packages, and seem intent on breaking them whenever people manage to
>> start to get them working.</rant>
> Actually yes because they don't want to be bound to the version provided
> by the distro (I will not enter the debate of whether it is a good thing
> or not)

If that's a requirement, the distro can provide multiple concurrent
versions.

There are lots of places where build systems aren't allowed to access
the internet.  And distro packages provides lots of useful things:
discoverability, filtering (some level of review has been done to make
sure this code us useful / safe), maintenance (local patches / fixes can
be applied if upstream disappears), decentralization (code is still
available even if upstream goes down / deletes his repositories).

I like Go as a language, but in this particular aspect, the core
developers just seem to be insane.

 -George
George Dunlap July 19, 2019, 11:04 a.m. UTC | #10
On 7/19/19 11:24 AM, Nicolas Belouin wrote:
> 
> 
> On 7/19/19 12:09 PM, George Dunlap wrote:
>> On 7/19/19 11:03 AM, Nicolas Belouin wrote:
>>>
>>> On 7/19/19 10:50 AM, George Dunlap wrote:
>>>>> On Jul 19, 2019, at 9:47 AM, George Dunlap <george.dunlap@citrix.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>>> On Jul 19, 2019, at 8:34 AM, Nicolas Belouin <nicolas.belouin@gandi.net> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 7/18/19 11:54 PM, George Dunlap wrote:
>>>>>>> The Go bindings for libxl miss functions from libxl_utils, let's start
>>>>>>> with the simple libxl_domid_to_name and its counterpart
>>>>>>> libxl_name_to_domid.
>>>>>>>
>>>>>>> NB that C.GoString() will return "" if it's passed a NULL; see
>>>>>>> https://github.com/golang/go/issues/32734#issuecomment-506835432
>>>>>>>
>>>>>>> Signed-off-by: Nicolas Belouin <nicolas.belouin@gandi.net>
>>>>>>> Signed-off-by: George Dunlap <george.dunlap@citrix.com>
>>>>>>> ---
>>>>>>> v3:
>>>>>>> - Wire into build system
>>>>>>> - Add reference to C.GoString() handling NULL to commit message
>>>>>>>
>>>>>>> Nicolas, could you test to see if this actually works for you?
>>>>>> Tested it, it works.
>>>>>>
>>>>>> I must confess I do not use that import path as the new modules mechanism
>>>>>> introduced in Go1.11 downloads and compile a versioned copy of every
>>>>>> dependency per project, and this behavior is incompatible with the build
>>>>>> system used here.
>>>>> It’s possible that something fundamentally has changed, but I suspect that rather you don’t quite understand how the current build system is supposed to work.  (In which case a write-up in the tree would probably be useful.)
>>>>>
>>>>> Go has always insisted that there be no binary compatibility between versions; so it’s always been necessary to re-compile all your libraries when upgrading from (say) 1.8 to 1.9.  Which means that any useable distribution must also include all the source files necessary to recompile when you bump the version number.
>>>>>
>>>>> So the core mechanism of the “install” is actually to copy all the source files necessary into the right local directory such that the go compiler can find them; ATM this is /usr/share/gocode/golang.xenproject.org/xenlight
>>>> Nit:  This of course should have a `src/` between `gocode/` and `golang.xenproject.org/`.
>>>>
>>>> NB also that this naming scheme was designed so that at some point in the future, we could actually host the xenlight packages at the URL provided.
>>>>
>>>>  -George
>>>>
>>> This new mechanism of modules is described here:
>>> https://golang.org/cmd/go/#hdr-Modules__module_versions__and_more
>>>
>>> The module system is intended to supersede the GOPATH approach and
>>> provide a way to get versioned dependencies, as such
>>> it does not rely on GOPATH at all and doesn't use sources or compiled
>>> packages present in GOPATH elements such as /usr/share/gocode
>>> and systematically fetch (at the asked version) and compile a copy of
>>> the dependency as it might be a different version from the one
>>> in GOPATH.
>>>
>>> As far as I tried, I have been unable to build my module even with the
>>> library installed.
>>> I have to use xenbits.xen.org/git-http/xen.git/tools/golang/xenlight (or
>>> one of its mirror) in order to build the module using the new
>>> mechanism

This will break as soon as we have support in golang/xenlight for libxl
features not in the version of Xen you're using.

E.g., suppose you're on Xen 4.12.  Someone introduces a new feature in
Xen 4.13, and plumbs it all the way from Xen, through libxl, *and*
golang/xenlight.  Now when *you* do a build, it will fail, because your
version of golang will expect libxl features which your system doesn't have.

I had always planned on getting golang.xenproject.org set up such that
it could interact with the "normal" go get thing.  If you want to help
us figure out how to get that set up, that would be helpful.

What would be *really* ideal is if we didn't have to link golang against
one particular hypervisor.  Maybe we need to use plugins?
https://golang.org/pkg/plugin/

 -George
Nicolas Belouin July 19, 2019, 1:19 p.m. UTC | #11
On 7/19/19 1:04 PM, George Dunlap wrote:
> On 7/19/19 11:24 AM, Nicolas Belouin wrote:
>>
>> On 7/19/19 12:09 PM, George Dunlap wrote:
>>> On 7/19/19 11:03 AM, Nicolas Belouin wrote:
>>>> On 7/19/19 10:50 AM, George Dunlap wrote:
>>>>>> On Jul 19, 2019, at 9:47 AM, George Dunlap <george.dunlap@citrix.com> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On Jul 19, 2019, at 8:34 AM, Nicolas Belouin <nicolas.belouin@gandi.net> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 7/18/19 11:54 PM, George Dunlap wrote:
>>>>>>>> The Go bindings for libxl miss functions from libxl_utils, let's start
>>>>>>>> with the simple libxl_domid_to_name and its counterpart
>>>>>>>> libxl_name_to_domid.
>>>>>>>>
>>>>>>>> NB that C.GoString() will return "" if it's passed a NULL; see
>>>>>>>> https://github.com/golang/go/issues/32734#issuecomment-506835432
>>>>>>>>
>>>>>>>> Signed-off-by: Nicolas Belouin <nicolas.belouin@gandi.net>
>>>>>>>> Signed-off-by: George Dunlap <george.dunlap@citrix.com>
>>>>>>>> ---
>>>>>>>> v3:
>>>>>>>> - Wire into build system
>>>>>>>> - Add reference to C.GoString() handling NULL to commit message
>>>>>>>>
>>>>>>>> Nicolas, could you test to see if this actually works for you?
>>>>>>> Tested it, it works.
>>>>>>>
>>>>>>> I must confess I do not use that import path as the new modules mechanism
>>>>>>> introduced in Go1.11 downloads and compile a versioned copy of every
>>>>>>> dependency per project, and this behavior is incompatible with the build
>>>>>>> system used here.
>>>>>> It’s possible that something fundamentally has changed, but I suspect that rather you don’t quite understand how the current build system is supposed to work.  (In which case a write-up in the tree would probably be useful.)
>>>>>>
>>>>>> Go has always insisted that there be no binary compatibility between versions; so it’s always been necessary to re-compile all your libraries when upgrading from (say) 1.8 to 1.9.  Which means that any useable distribution must also include all the source files necessary to recompile when you bump the version number.
>>>>>>
>>>>>> So the core mechanism of the “install” is actually to copy all the source files necessary into the right local directory such that the go compiler can find them; ATM this is /usr/share/gocode/golang.xenproject.org/xenlight
>>>>> Nit:  This of course should have a `src/` between `gocode/` and `golang.xenproject.org/`.
>>>>>
>>>>> NB also that this naming scheme was designed so that at some point in the future, we could actually host the xenlight packages at the URL provided.
>>>>>
>>>>>  -George
>>>>>
>>>> This new mechanism of modules is described here:
>>>> https://golang.org/cmd/go/#hdr-Modules__module_versions__and_more
>>>>
>>>> The module system is intended to supersede the GOPATH approach and
>>>> provide a way to get versioned dependencies, as such
>>>> it does not rely on GOPATH at all and doesn't use sources or compiled
>>>> packages present in GOPATH elements such as /usr/share/gocode
>>>> and systematically fetch (at the asked version) and compile a copy of
>>>> the dependency as it might be a different version from the one
>>>> in GOPATH.
>>>>
>>>> As far as I tried, I have been unable to build my module even with the
>>>> library installed.
>>>> I have to use xenbits.xen.org/git-http/xen.git/tools/golang/xenlight (or
>>>> one of its mirror) in order to build the module using the new
>>>> mechanism
> This will break as soon as we have support in golang/xenlight for libxl
> features not in the version of Xen you're using.
>
> E.g., suppose you're on Xen 4.12.  Someone introduces a new feature in
> Xen 4.13, and plumbs it all the way from Xen, through libxl, *and*
> golang/xenlight.  Now when *you* do a build, it will fail, because your
> version of golang will expect libxl features which your system doesn't have.
I know of that, and that can be overcome using modules as you can
specify a branch version of the module you depends on (e.g you can set
your dependency as being xxx/xenlight@stable-4.12).
>
> I had always planned on getting golang.xenproject.org set up such that
> it could interact with the "normal" go get thing.  If you want to help
> us figure out how to get that set up, that would be helpful.

As far as I looked into vanity URLs, you can't serve a subdirectory of a
repository directly, but you can trick the system using a go-proxy.
To do that you need two things. First, you need
https://golang.xenproject.org/xenlight?go-get=1
<https://example.org/pkg/foo?go-get=1> to point to a page containing a
    <meta name="go-import" content="golang.xenproject.org mod
https://golang.xenproject.org/moduleproxy <https://code.org/moduleproxy>">
And have golang.xenproject.org/moduleproxy to follow the specifications
of module proxies by proposing '.zip' files containing the different
versions of the module. This part for sure can be scripted to do the
hard work of dynamically packaging the versions from the git repository
on demand.


>
> What would be *really* ideal is if we didn't have to link golang against
> one particular hypervisor.  Maybe we need to use plugins?
> https://golang.org/pkg/plugin/

This is the Go equivalent of ldopen, I'm not fond of this approach in
general, but something like this can indeed do the trick to keep
from linking against libxl directly, it would need a split in two
modules to be easily usable though.


Nicolas
George Dunlap July 19, 2019, 1:48 p.m. UTC | #12
On 7/19/19 2:19 PM, Nicolas Belouin wrote:
> 
> 
> On 7/19/19 1:04 PM, George Dunlap wrote:
>> On 7/19/19 11:24 AM, Nicolas Belouin wrote:
>>>
>>> On 7/19/19 12:09 PM, George Dunlap wrote:
>>>> On 7/19/19 11:03 AM, Nicolas Belouin wrote:
>>>>> On 7/19/19 10:50 AM, George Dunlap wrote:
>>>>>>> On Jul 19, 2019, at 9:47 AM, George Dunlap <george.dunlap@citrix.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On Jul 19, 2019, at 8:34 AM, Nicolas Belouin <nicolas.belouin@gandi.net> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 7/18/19 11:54 PM, George Dunlap wrote:
>>>>>>>>> The Go bindings for libxl miss functions from libxl_utils, let's start
>>>>>>>>> with the simple libxl_domid_to_name and its counterpart
>>>>>>>>> libxl_name_to_domid.
>>>>>>>>>
>>>>>>>>> NB that C.GoString() will return "" if it's passed a NULL; see
>>>>>>>>> https://github.com/golang/go/issues/32734#issuecomment-506835432
>>>>>>>>>
>>>>>>>>> Signed-off-by: Nicolas Belouin <nicolas.belouin@gandi.net>
>>>>>>>>> Signed-off-by: George Dunlap <george.dunlap@citrix.com>
>>>>>>>>> ---
>>>>>>>>> v3:
>>>>>>>>> - Wire into build system
>>>>>>>>> - Add reference to C.GoString() handling NULL to commit message
>>>>>>>>>
>>>>>>>>> Nicolas, could you test to see if this actually works for you?
>>>>>>>> Tested it, it works.
>>>>>>>>
>>>>>>>> I must confess I do not use that import path as the new modules mechanism
>>>>>>>> introduced in Go1.11 downloads and compile a versioned copy of every
>>>>>>>> dependency per project, and this behavior is incompatible with the build
>>>>>>>> system used here.
>>>>>>> It’s possible that something fundamentally has changed, but I suspect that rather you don’t quite understand how the current build system is supposed to work.  (In which case a write-up in the tree would probably be useful.)
>>>>>>>
>>>>>>> Go has always insisted that there be no binary compatibility between versions; so it’s always been necessary to re-compile all your libraries when upgrading from (say) 1.8 to 1.9.  Which means that any useable distribution must also include all the source files necessary to recompile when you bump the version number.
>>>>>>>
>>>>>>> So the core mechanism of the “install” is actually to copy all the source files necessary into the right local directory such that the go compiler can find them; ATM this is /usr/share/gocode/golang.xenproject.org/xenlight
>>>>>> Nit:  This of course should have a `src/` between `gocode/` and `golang.xenproject.org/`.
>>>>>>
>>>>>> NB also that this naming scheme was designed so that at some point in the future, we could actually host the xenlight packages at the URL provided.
>>>>>>
>>>>>>  -George
>>>>>>
>>>>> This new mechanism of modules is described here:
>>>>> https://golang.org/cmd/go/#hdr-Modules__module_versions__and_more
>>>>>
>>>>> The module system is intended to supersede the GOPATH approach and
>>>>> provide a way to get versioned dependencies, as such
>>>>> it does not rely on GOPATH at all and doesn't use sources or compiled
>>>>> packages present in GOPATH elements such as /usr/share/gocode
>>>>> and systematically fetch (at the asked version) and compile a copy of
>>>>> the dependency as it might be a different version from the one
>>>>> in GOPATH.
>>>>>
>>>>> As far as I tried, I have been unable to build my module even with the
>>>>> library installed.
>>>>> I have to use xenbits.xen.org/git-http/xen.git/tools/golang/xenlight (or
>>>>> one of its mirror) in order to build the module using the new
>>>>> mechanism
>> This will break as soon as we have support in golang/xenlight for libxl
>> features not in the version of Xen you're using.
>>
>> E.g., suppose you're on Xen 4.12.  Someone introduces a new feature in
>> Xen 4.13, and plumbs it all the way from Xen, through libxl, *and*
>> golang/xenlight.  Now when *you* do a build, it will fail, because your
>> version of golang will expect libxl features which your system doesn't have.
> I know of that, and that can be overcome using modules as you can
> specify a branch version of the module you depends on (e.g you can set
> your dependency as being xxx/xenlight@stable-4.12).

That's not terrible I guess.

>> I had always planned on getting golang.xenproject.org set up such that
>> it could interact with the "normal" go get thing.  If you want to help
>> us figure out how to get that set up, that would be helpful.
> 
> As far as I looked into vanity URLs, you can't serve a subdirectory of a
> repository directly, but you can trick the system using a go-proxy.
> To do that you need two things. First, you need
> https://golang.xenproject.org/xenlight?go-get=1
> <https://example.org/pkg/foo?go-get=1> to point to a page containing a
>     <meta name="go-import" content="golang.xenproject.org mod
> https://golang.xenproject.org/moduleproxy <https://code.org/moduleproxy>">
> And have golang.xenproject.org/moduleproxy to follow the specifications
> of module proxies by proposing '.zip' files containing the different
> versions of the module. This part for sure can be scripted to do the
> hard work of dynamically packaging the versions from the git repository
> on demand.

Yes, that's sort of what I had in mind -- although it might be better to
statically update the content with a crontab or something, rather than
regenerating it from scratch every connection.

>> What would be *really* ideal is if we didn't have to link golang against
>> one particular hypervisor.  Maybe we need to use plugins?
>> https://golang.org/pkg/plugin/
> 
> This is the Go equivalent of ldopen, I'm not fond of this approach in
> general, but something like this can indeed do the trick to keep
> from linking against libxl directly, it would need a split in two
> modules to be easily usable though.

It's certainly a lot of work, but it seems a lot nicer than having to
recompile your application every time you upgrade your hypervisor. :-)

But obviously all that requires work and testing, which requires someone
who has enough time to make it happen.

 -George

Patch
diff mbox series

diff --git a/tools/golang/xenlight/Makefile b/tools/golang/xenlight/Makefile
index 0987305224..65923df59a 100644
--- a/tools/golang/xenlight/Makefile
+++ b/tools/golang/xenlight/Makefile
@@ -8,7 +8,7 @@  GOXL_PKG_DIR = /src/$(XEN_GOCODE_URL)/xenlight/
 GOXL_INSTALL_DIR = $(GOCODE_DIR)$(GOXL_PKG_DIR)
 
 # PKGSOURCES: Files which comprise the distributed source package
-PKGSOURCES = xenlight.go
+PKGSOURCES = xenlight.go xenlight_utils.go
 
 GO ?= go
 
diff --git a/tools/golang/xenlight/xenlight_utils.go b/tools/golang/xenlight/xenlight_utils.go
new file mode 100644
index 0000000000..da1636842d
--- /dev/null
+++ b/tools/golang/xenlight/xenlight_utils.go
@@ -0,0 +1,55 @@ 
+/*
+ * Copyright (C) 2019 Nicolas Belouin, Gandi SAS
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation;
+ * version 2.1 of the License.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; If not, see <http://www.gnu.org/licenses/>.
+ */
+package xenlight
+
+/*
+#cgo LDFLAGS: -lxenlight -lyajl -lxentoollog
+#include <stdlib.h>
+#include <libxl_utils.h>
+*/
+import "C"
+
+import (
+	"unsafe"
+)
+
+//char* libxl_domid_to_name(libxl_ctx *ctx, uint32_t domid);
+func (Ctx *Context) DomidToName(id Domid) (name string) {
+	cDomName := C.libxl_domid_to_name(Ctx.ctx, C.uint32_t(id))
+	defer C.free(unsafe.Pointer(cDomName))
+
+	name = C.GoString(cDomName)
+	return
+}
+
+//int libxl_name_to_domid(libxl_ct *ctx, const char *name, uint32_t *domid)
+func (Ctx *Context) NameToDomid(name string) (id Domid, err error) {
+	cname := C.CString(name)
+	defer C.free(unsafe.Pointer(cname))
+
+	var cDomId C.uint32_t
+
+	ret := C.libxl_name_to_domid(Ctx.ctx, cname, &cDomId)
+	if ret != 0 {
+		err = Error(-ret)
+		return
+	}
+
+	id = Domid(cDomId)
+
+	return
+}