| Message ID | 20220221155519.2367-1-damien.hedde@greensocs.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | qmp-shell modifications for non-interactive use | expand |
Damien Hedde <damien.hedde@greensocs.com> writes: > Hi, > > The main idea of this series is to be a bit more user-friendly when > using qmp-shell in a non-interactive way: with an input redirection > from a file containing a list of commands. > > I'm working on dynamic qapi config of a qemu machine, this would > be very useful to provide and reproduce small examples. Why not use plain QMP for that? [...]
On 2/22/22 07:10, Markus Armbruster wrote: > Damien Hedde <damien.hedde@greensocs.com> writes: > >> Hi, >> >> The main idea of this series is to be a bit more user-friendly when >> using qmp-shell in a non-interactive way: with an input redirection >> from a file containing a list of commands. >> >> I'm working on dynamic qapi config of a qemu machine, this would >> be very useful to provide and reproduce small examples. > > Why not use plain QMP for that? > > [...] > What do you mean by plain QMP ? -- Damien
On Tue, Feb 22, 2022 at 08:57:03AM +0100, Damien Hedde wrote: > > > On 2/22/22 07:10, Markus Armbruster wrote: > > Damien Hedde <damien.hedde@greensocs.com> writes: > > > > > Hi, > > > > > > The main idea of this series is to be a bit more user-friendly when > > > using qmp-shell in a non-interactive way: with an input redirection > > > from a file containing a list of commands. > > > > > > I'm working on dynamic qapi config of a qemu machine, this would > > > be very useful to provide and reproduce small examples. > > > > Why not use plain QMP for that? > > > > [...] > > > What do you mean by plain QMP ? Directly connect to the socket and send the QMP JSON commands you have. Essentially qmp-shell is designed for adhoc interactive human usage. For automated / scripted, non-interactive usage, it is expected that QMP is simple enough that tools just directly connect to the QMP socket instead of using a wrapper layer. What is the reason you want to use qmp-shell for this instead of directly using the socket from your scripts ? Regards, Daniel
On 2/22/22 10:21, Daniel P. Berrangé wrote: > On Tue, Feb 22, 2022 at 08:57:03AM +0100, Damien Hedde wrote: >> >> >> On 2/22/22 07:10, Markus Armbruster wrote: >>> Damien Hedde <damien.hedde@greensocs.com> writes: >>> >>>> Hi, >>>> >>>> The main idea of this series is to be a bit more user-friendly when >>>> using qmp-shell in a non-interactive way: with an input redirection >>>> from a file containing a list of commands. >>>> >>>> I'm working on dynamic qapi config of a qemu machine, this would >>>> be very useful to provide and reproduce small examples. >>> >>> Why not use plain QMP for that? >>> >>> [...] >>> >> What do you mean by plain QMP ? > > Directly connect to the socket and send the QMP JSON commands you have. > > Essentially qmp-shell is designed for adhoc interactive human usage. > For automated / scripted, non-interactive usage, it is expected that > QMP is simple enough that tools just directly connect to the QMP > socket instead of using a wrapper layer. > > What is the reason you want to use qmp-shell for this instead of > directly using the socket from your scripts ? > > Regards, > Daniel We have such scripts that interface with QMP directly for our own use. Here I just wanted to propose a simple way to just send a bunch of commands from a source file and stop if something unexpected happens. Only goal is to be able to share a file on the ml and allow people to reproduce easily. We can already redirect the input, but it is almost impossible to see if something failed. -- Damien
Damien Hedde <damien.hedde@greensocs.com> writes: > On 2/22/22 07:10, Markus Armbruster wrote: >> Damien Hedde <damien.hedde@greensocs.com> writes: >> >>> Hi, >>> >>> The main idea of this series is to be a bit more user-friendly when >>> using qmp-shell in a non-interactive way: with an input redirection >>> from a file containing a list of commands. >>> >>> I'm working on dynamic qapi config of a qemu machine, this would >>> be very useful to provide and reproduce small examples. >> Why not use plain QMP for that? >> [...] >> > What do you mean by plain QMP ? Talk straight to QEMU without a translator: $ cat script {"execute": "qmp_capabilities"} {"execute": "quit"} $ socat -t 3 STDIO UNIX-CONNECT:$HOME/work/images/test-qmp <script {"QMP": {"version": {"qemu": {"micro": 50, "minor": 2, "major": 6}, "package": "v6.2.0-1603-gc13b8e9973"}, "capabilities": ["oob"]}} {"error": {"class": "CommandNotFound", "desc": "Expecting capabilities negotiation with 'qmp_capabilities'"}} armbru@dusky:~/work/qemu$ echo -e '{"execute":"qmp_capabilities"}{"execute":"quit"}' >script armbru@dusky:~/work/qemu$ echo -e '{"execute":"qmp_capabilities"}\n{"execute":"quit"}' >script armbru@dusky:~/work/qemu$ socat -t 3 STDIO UNIX-CONNECT:$HOME/work/images/test-qmp <script {"QMP": {"version": {"qemu": {"micro": 50, "minor": 2, "major": 6}, "package": "v6.2.0-1603-gc13b8e9973"}, "capabilities": ["oob"]}} {"return": {}} {"return": {}} {"timestamp": {"seconds": 1645523438, "microseconds": 951702}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}} socat also supports interactive use nicely. Try $ socat "READLINE,history=$HOME/.qmp_history,prompt=QMP>" UNIX-CONNECT:$HOME/path/to/socket Helpfully blinks matching parenthesis for me.
On Tue, Feb 22, 2022 at 10:38:09AM +0100, Damien Hedde wrote: > > > On 2/22/22 10:21, Daniel P. Berrangé wrote: > > On Tue, Feb 22, 2022 at 08:57:03AM +0100, Damien Hedde wrote: > > > > > > > > > On 2/22/22 07:10, Markus Armbruster wrote: > > > > Damien Hedde <damien.hedde@greensocs.com> writes: > > > > > > > > > Hi, > > > > > > > > > > The main idea of this series is to be a bit more user-friendly when > > > > > using qmp-shell in a non-interactive way: with an input redirection > > > > > from a file containing a list of commands. > > > > > > > > > > I'm working on dynamic qapi config of a qemu machine, this would > > > > > be very useful to provide and reproduce small examples. > > > > > > > > Why not use plain QMP for that? > > > > > > > > [...] > > > > > > > What do you mean by plain QMP ? > > > > Directly connect to the socket and send the QMP JSON commands you have. > > > > Essentially qmp-shell is designed for adhoc interactive human usage. > > For automated / scripted, non-interactive usage, it is expected that > > QMP is simple enough that tools just directly connect to the QMP > > socket instead of using a wrapper layer. > > > > What is the reason you want to use qmp-shell for this instead of > > directly using the socket from your scripts ? > > > > Regards, > > Daniel > > We have such scripts that interface with QMP directly for our own use. > > Here I just wanted to propose a simple way to just send a > bunch of commands from a source file and stop if something unexpected > happens. > Only goal is to be able to share a file on the ml and allow people to > reproduce easily. > We can already redirect the input, but it is almost impossible to see > if something failed. Yes, I see what you mean. So the problem with using 'socat' or similar is that we fill the input with commands and response appear asynchronously, so we can't match them up easily. This is actually a problem seen in the block I/O tests which just send QMP stuff in a batch. While you could do this by invoking socat once for each command, that gets silly with the repeated QMP handshake for each command. The thing about using qmp-shell is that it does a bunch of extra stuff targetted at humans on top, and history tells us it isn't a good idea to mix stuff for humans and machines in the same tool/interface. How about instead creating a separate 'qmp-send' command that is not much more than a "QMP-aware socat". By which I mean, it just reads raw QMP commands from stdin, sends each one to the server, but crucially waits for a reply after sending each, and stops on first error reponse. Regards, Daniel
On 2/22/22 11:31, Daniel P. Berrangé wrote: > On Tue, Feb 22, 2022 at 10:38:09AM +0100, Damien Hedde wrote: >> >> >> Here I just wanted to propose a simple way to just send a >> bunch of commands from a source file and stop if something unexpected >> happens. >> Only goal is to be able to share a file on the ml and allow people to >> reproduce easily. >> We can already redirect the input, but it is almost impossible to see >> if something failed. > > Yes, I see what you mean. So the problem with using 'socat' or similar > is that we fill the input with commands and response appear asynchronously, > so we can't match them up easily. This is actually a problem seen in the > block I/O tests which just send QMP stuff in a batch. > > While you could do this by invoking socat once for each command, that > gets silly with the repeated QMP handshake for each command. > > The thing about using qmp-shell is that it does a bunch of extra stuff > targetted at humans on top, and history tells us it isn't a good idea > to mix stuff for humans and machines in the same tool/interface. > > How about instead creating a separate 'qmp-send' command that is not > much more than a "QMP-aware socat". By which I mean, it just reads > raw QMP commands from stdin, sends each one to the server, but > crucially waits for a reply after sending each, and stops on first > error reponse. By 'qmp-send' command, you mean another script in scripts/qmp ? Yes If we go for another script, I would rather do one with wrap feature (like your series) to start qemu as well. -- Damien
On Wed, Feb 23, 2022 at 10:57:29AM +0100, Damien Hedde wrote: > > > On 2/22/22 11:31, Daniel P. Berrangé wrote: > > On Tue, Feb 22, 2022 at 10:38:09AM +0100, Damien Hedde wrote: > > > > > > > > > Here I just wanted to propose a simple way to just send a > > > bunch of commands from a source file and stop if something unexpected > > > happens. > > > Only goal is to be able to share a file on the ml and allow people to > > > reproduce easily. > > > We can already redirect the input, but it is almost impossible to see > > > if something failed. > > > > Yes, I see what you mean. So the problem with using 'socat' or similar > > is that we fill the input with commands and response appear asynchronously, > > so we can't match them up easily. This is actually a problem seen in the > > block I/O tests which just send QMP stuff in a batch. > > > > While you could do this by invoking socat once for each command, that > > gets silly with the repeated QMP handshake for each command. > > > > The thing about using qmp-shell is that it does a bunch of extra stuff > > targetted at humans on top, and history tells us it isn't a good idea > > to mix stuff for humans and machines in the same tool/interface. > > > > How about instead creating a separate 'qmp-send' command that is not > > much more than a "QMP-aware socat". By which I mean, it just reads > > raw QMP commands from stdin, sends each one to the server, but > > crucially waits for a reply after sending each, and stops on first > > error reponse. > > By 'qmp-send' command, you mean another script in scripts/qmp ? > Yes Yep. > If we go for another script, I would rather do one with wrap > feature (like your series) to start qemu as well. Sure, that would certainly make sense. I actually wanted to add the wrap feature directly into the existing qmp-shell, but it was not viable while maintaining back compat. With a new qmp-send command you can easily make "wrap mode" supported from the start. Regards, Daniel
On Wed, Feb 23, 2022 at 6:13 AM Daniel P. Berrangé <berrange@redhat.com> wrote: > > On Wed, Feb 23, 2022 at 10:57:29AM +0100, Damien Hedde wrote: > > > > > > On 2/22/22 11:31, Daniel P. Berrangé wrote: > > > On Tue, Feb 22, 2022 at 10:38:09AM +0100, Damien Hedde wrote: > > > > > > > > > > > > Here I just wanted to propose a simple way to just send a > > > > bunch of commands from a source file and stop if something unexpected > > > > happens. > > > > Only goal is to be able to share a file on the ml and allow people to > > > > reproduce easily. > > > > We can already redirect the input, but it is almost impossible to see > > > > if something failed. > > > > > > Yes, I see what you mean. So the problem with using 'socat' or similar > > > is that we fill the input with commands and response appear asynchronously, > > > so we can't match them up easily. This is actually a problem seen in the > > > block I/O tests which just send QMP stuff in a batch. > > > > > > While you could do this by invoking socat once for each command, that > > > gets silly with the repeated QMP handshake for each command. > > > > > > The thing about using qmp-shell is that it does a bunch of extra stuff > > > targetted at humans on top, and history tells us it isn't a good idea > > > to mix stuff for humans and machines in the same tool/interface. > > > > > > How about instead creating a separate 'qmp-send' command that is not > > > much more than a "QMP-aware socat". By which I mean, it just reads > > > raw QMP commands from stdin, sends each one to the server, but > > > crucially waits for a reply after sending each, and stops on first > > > error reponse. > > > > By 'qmp-send' command, you mean another script in scripts/qmp ? > > Yes > > Yep. > > > If we go for another script, I would rather do one with wrap > > feature (like your series) to start qemu as well. > > Sure, that would certainly make sense. I actually wanted to add > the wrap feature directly into the existing qmp-shell, but it was > not viable while maintaining back compat. With a new qmp-send > command you can easily make "wrap mode" supported from the start. > I'm also wary of adding scriptable interfaces to qmp-shell, but I can see them having some value. I'm not against the ability to add some kind of "load commands from file" interactive command to qmp-shell, for instance. (/LOAD or /PLAY or something?) ... but then we need to worry about what the format of the file is and how exactly that scripting language works. It's a design minefield. I can also see the use in having qmp-shell autoplay a script file at startup and then resume interactive function (Saves you some typing!), but that opens the door to people using it as a script in and of itself and coming to rely on it, which I really would not like to see. A standalone qemu-send that supports launching QEMU (with args) and taking a script file from the CLI sounds great, though. There's still some design questions I have, though: What features do you need out of the script file? What syntax do you intend to use? Does it need event waits? How complex do they need to be? I'm fine with something simple, but I can see cases where we might begin to want slightly more power out of it. I see good utility for test reproductions shared on the ML and in bug trackers, though. --js
On Wed, Feb 23, 2022 at 10:01:05AM -0500, John Snow wrote: > On Wed, Feb 23, 2022 at 6:13 AM Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > On Wed, Feb 23, 2022 at 10:57:29AM +0100, Damien Hedde wrote: > > > > > > > > > On 2/22/22 11:31, Daniel P. Berrangé wrote: > > > > On Tue, Feb 22, 2022 at 10:38:09AM +0100, Damien Hedde wrote: > > > > > > > > > > > > > > > Here I just wanted to propose a simple way to just send a > > > > > bunch of commands from a source file and stop if something unexpected > > > > > happens. > > > > > Only goal is to be able to share a file on the ml and allow people to > > > > > reproduce easily. > > > > > We can already redirect the input, but it is almost impossible to see > > > > > if something failed. > > > > > > > > Yes, I see what you mean. So the problem with using 'socat' or similar > > > > is that we fill the input with commands and response appear asynchronously, > > > > so we can't match them up easily. This is actually a problem seen in the > > > > block I/O tests which just send QMP stuff in a batch. > > > > > > > > While you could do this by invoking socat once for each command, that > > > > gets silly with the repeated QMP handshake for each command. > > > > > > > > The thing about using qmp-shell is that it does a bunch of extra stuff > > > > targetted at humans on top, and history tells us it isn't a good idea > > > > to mix stuff for humans and machines in the same tool/interface. > > > > > > > > How about instead creating a separate 'qmp-send' command that is not > > > > much more than a "QMP-aware socat". By which I mean, it just reads > > > > raw QMP commands from stdin, sends each one to the server, but > > > > crucially waits for a reply after sending each, and stops on first > > > > error reponse. > > > > > > By 'qmp-send' command, you mean another script in scripts/qmp ? > > > Yes > > > > Yep. > > > > > If we go for another script, I would rather do one with wrap > > > feature (like your series) to start qemu as well. > > > > Sure, that would certainly make sense. I actually wanted to add > > the wrap feature directly into the existing qmp-shell, but it was > > not viable while maintaining back compat. With a new qmp-send > > command you can easily make "wrap mode" supported from the start. > > > > I'm also wary of adding scriptable interfaces to qmp-shell, but I can > see them having some value. > > I'm not against the ability to add some kind of "load commands from > file" interactive command to qmp-shell, for instance. (/LOAD or /PLAY > or something?) ... but then we need to worry about what the format of > the file is and how exactly that scripting language works. It's a > design minefield. My concern is that qmp-shell takes command input in a high level data format. I don't want to see that format turn into something that machines use, which is what is proposed initially here. For this reason I prefer to see a separate qmp-send that solves the automation problems, without introducing a new data format, just directly passing raw QMP messages to/fro. Regards, Daniel
On Mon, Feb 21, 2022 at 10:55 AM Damien Hedde <damien.hedde@greensocs.com> wrote: > > Hi, > > The main idea of this series is to be a bit more user-friendly when > using qmp-shell in a non-interactive way: with an input redirection > from a file containing a list of commands. > > I'm working on dynamic qapi config of a qemu machine, this would > be very useful to provide and reproduce small examples. > > This series proposes the following modifications: > + no prompt when input is non-interactive > + an --exit-on-error option so that the shell exits on first > error (disconnection, command parsing error, response is an error) > + support for comment lines and escaping eol to have more reability > in the source files. > > I tested this using QMPShell. I tried HMPShell but did not findout > how to successfully use it with qemu. How do I setup an HMPShell ?. > Should be the same. It's just passing HMP commands through the QMP layer using {"execute": "human-monitor-command", ...}. > ./qmp-shell -H ../../bin/git/monitor.sock Welcome to the HMP shell! Connected to QEMU 6.2.50 (QEMU) help announce_self [interfaces] [id] -- Trigger GARP/RARP announcements ... (and many more lines.) HMP is a completely different interface, with different commands, etc. It's meant for human users, but we are trying to remove any reasons to use it. It has its uses for some debug commands that we don't want to support over the QMP interface, and for some commands that aren't safe and might freeze, etc. Dan has spent a lot of time and effort migrating the save state commands over to QMP, for example. > Another "issue" I have is the handling of integers. I > deal with a lot of addresses and reading/writing them as decimal is > a bit painful (json does not support hexadecimal integer format). Do > you think of any reasonable workaround for this ? Maybe HMP shell > support this ? Take a look at `def _parse_value(cls, val: str) -> object:` int("0x13") is going to raise an Exception, so this won't be perceived as a numerical value. You can use int(val, base=0) to have Python guess the base from the string given (which allows for "0x", "0b" and maybe some others.) This might cause a regression if anyone was using this to pass "0x13" as a string on purpose. It might also be the case that the FuzzyJSON portion of the parser already accepts hexadecimal numbers. You'd have to check, but if you take a look at that piece of the code, you can see that we parse "JSON" arguments as a Python AST and then swap out the true/false/null literals for True/False/None. This gives some flexibility to the syntax being entered here. I don't know off the top of my head what happens to numbers here. This parser is only used when an argument starts with '{' or '[', though. This is related to another problem we have, which is that the qom-set/qom-get scripts are pretty much extremely busted for anything other than strings. The type safety of these interfaces is not ... really there right now. In a beautiful future utopia, we'd be able to get the type information we need directly from QEMU and deterministically convert everything into the right form before sending it out over the wire without guessing around. --js