| Message ID | 20210223003408.964543-3-jsnow@redhat.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | qapi: static typing conversion, pt3 | expand |
John Snow <jsnow@redhat.com> writes: > OrderedDict is a subtype of dict, so we can check for a more general > form. These functions do not themselves depend on it being any > particular type. True. The actual arguments can only be OrderedDict, though. I think we refrained from relaxing to dict in these helpers because we felt "staying ordered" is clearer. We're *this* close to mooting the point, because Changed in version 3.7: Dictionary order is guaranteed to be insertion order. This behavior was an implementation detail of CPython from 3.6. https://docs.python.org/3.7/library/stdtypes.html Is messing with it necessary for later work? If not, is it a worthwhile improvement? > Signed-off-by: John Snow <jsnow@redhat.com> > Reviewed-by: Eduardo Habkost <ehabkost@redhat.com> > Reviewed-by: Cleber Rosa <crosa@redhat.com> > --- > scripts/qapi/expr.py | 5 ++--- > 1 file changed, 2 insertions(+), 3 deletions(-) > > diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py > index 35695c4c653..5694c501fa3 100644 > --- a/scripts/qapi/expr.py > +++ b/scripts/qapi/expr.py > @@ -14,7 +14,6 @@ > # This work is licensed under the terms of the GNU GPL, version 2. > # See the COPYING file in the top-level directory. > > -from collections import OrderedDict > import re > > from .common import c_name > @@ -131,7 +130,7 @@ def check_if_str(ifcond): > > > def normalize_members(members): > - if isinstance(members, OrderedDict): > + if isinstance(members, dict): > for key, arg in members.items(): > if isinstance(arg, dict): > continue > @@ -162,7 +161,7 @@ def check_type(value, info, source, > if not allow_dict: > raise QAPISemError(info, "%s should be a type name" % source) > > - if not isinstance(value, OrderedDict): > + if not isinstance(value, dict): > raise QAPISemError(info, > "%s should be an object or type name" % source)
On 2/24/21 4:30 AM, Markus Armbruster wrote: > John Snow <jsnow@redhat.com> writes: > >> OrderedDict is a subtype of dict, so we can check for a more general >> form. These functions do not themselves depend on it being any >> particular type. > > True. The actual arguments can only be OrderedDict, though. I think we > refrained from relaxing to dict in these helpers because we felt > "staying ordered" is clearer. > As a habit, I tend towards declaring the least specific type possible for input and declaring the most specific type possible for output. > We're *this* close to mooting the point, because > > Changed in version 3.7: Dictionary order is guaranteed to be > insertion order. This behavior was an implementation detail of > CPython from 3.6. > > https://docs.python.org/3.7/library/stdtypes.html > > Is messing with it necessary for later work? If not, is it a worthwhile > improvement? > Not strictly necessary, but if the expression checkers here don't *require* the type be an ordereddict, why bother to enforce that here? It's just a bid to slacken the type (my type hints will look for Dict, not OrderedDict) and leave the use of OrderedDict as an "implementation detail" that only parser.py knows about. (I needed to change it for prototyping using an off-the-shelf parser, so it was annoying to have it check for a stronger type if it doesn't absolutely have to.) >> Signed-off-by: John Snow <jsnow@redhat.com> >> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com> >> Reviewed-by: Cleber Rosa <crosa@redhat.com> >> --- >> scripts/qapi/expr.py | 5 ++--- >> 1 file changed, 2 insertions(+), 3 deletions(-) >> >> diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py >> index 35695c4c653..5694c501fa3 100644 >> --- a/scripts/qapi/expr.py >> +++ b/scripts/qapi/expr.py >> @@ -14,7 +14,6 @@ >> # This work is licensed under the terms of the GNU GPL, version 2. >> # See the COPYING file in the top-level directory. >> >> -from collections import OrderedDict >> import re >> >> from .common import c_name >> @@ -131,7 +130,7 @@ def check_if_str(ifcond): >> >> >> def normalize_members(members): >> - if isinstance(members, OrderedDict): >> + if isinstance(members, dict): >> for key, arg in members.items(): >> if isinstance(arg, dict): >> continue >> @@ -162,7 +161,7 @@ def check_type(value, info, source, >> if not allow_dict: >> raise QAPISemError(info, "%s should be a type name" % source) >> >> - if not isinstance(value, OrderedDict): >> + if not isinstance(value, dict): >> raise QAPISemError(info, >> "%s should be an object or type name" % source)
John Snow <jsnow@redhat.com> writes: > On 2/24/21 4:30 AM, Markus Armbruster wrote: >> John Snow <jsnow@redhat.com> writes: >> >>> OrderedDict is a subtype of dict, so we can check for a more general >>> form. These functions do not themselves depend on it being any >>> particular type. >> >> True. The actual arguments can only be OrderedDict, though. I think we >> refrained from relaxing to dict in these helpers because we felt >> "staying ordered" is clearer. >> > > As a habit, I tend towards declaring the least specific type possible > for input and declaring the most specific type possible for output. This maximimizes generality, which can be quite worthwhile. Maximizing generality by default is not a bad habit, I guess. But cases exist where generality is not needed, and other considerations take precedence. >> We're *this* close to mooting the point, because >> >> Changed in version 3.7: Dictionary order is guaranteed to be >> insertion order. This behavior was an implementation detail of >> CPython from 3.6. >> >> https://docs.python.org/3.7/library/stdtypes.html >> >> Is messing with it necessary for later work? If not, is it a worthwhile >> improvement? >> > > Not strictly necessary, but if the expression checkers here don't > *require* the type be an ordereddict, why bother to enforce that here? > > It's just a bid to slacken the type (my type hints will look for Dict, > not OrderedDict) and leave the use of OrderedDict as an "implementation > detail" that only parser.py knows about. "Orderedness" is anything but a detail only parser.py knows about. Example: { 'command': 'blockdev-insert-medium', 'data': { 'id': 'str', 'node-name': 'str'} } AST: OrderedDict([('command', 'blockdev-insert-medium'), ('data', OrderedDict([('id', {'type': 'str'}), ('node-name', {'type': 'str'})]))]) For the inner dictionary, order matters, because the difference between void qmp_blockdev_insert_medium(const char *id, const char *node_name, Error **errp); and void qmp_blockdev_insert_medium(const char *node_name, const char *id, Error **errp); matters. For the outer dictionary, order carries no semantic meaning. My point is: parser.py fundamentally builds *ordered* dicts. We're certainly free to relax them to more general types wherever "orderedness" is not needed. However, one of the main aims of this typing exercise is to make the code easier to read, and I doubt making things more general helps there. Related: the type aliases for the AST you define later in this series. I figure relaxing these to more general types where possible would actually reduce their value. TopLevelExpression tells me more than dict. I'm not against relaxing types per se. Judicious relaxation is often useful to keep code more generally useful. When to relax isn't always obvious. > (I needed to change it for prototyping using an off-the-shelf parser, so > it was annoying to have it check for a stronger type if it doesn't > absolutely have to.) If your off-the-shelf parse doesn't preserve order, it's not fit for the purpose :) >>> Signed-off-by: John Snow <jsnow@redhat.com> >>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com> >>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
On 2/25/21 5:40 AM, Markus Armbruster wrote: > John Snow <jsnow@redhat.com> writes: > >> On 2/24/21 4:30 AM, Markus Armbruster wrote: >>> John Snow <jsnow@redhat.com> writes: >>> >>>> OrderedDict is a subtype of dict, so we can check for a more general >>>> form. These functions do not themselves depend on it being any >>>> particular type. >>> >>> True. The actual arguments can only be OrderedDict, though. I think we >>> refrained from relaxing to dict in these helpers because we felt >>> "staying ordered" is clearer. >>> >> >> As a habit, I tend towards declaring the least specific type possible >> for input and declaring the most specific type possible for output. > > This maximimizes generality, which can be quite worthwhile. Maximizing > generality by default is not a bad habit, I guess. But cases exist > where generality is not needed, and other considerations take > precedence. > >>> We're *this* close to mooting the point, because >>> >>> Changed in version 3.7: Dictionary order is guaranteed to be >>> insertion order. This behavior was an implementation detail of >>> CPython from 3.6. >>> >>> https://docs.python.org/3.7/library/stdtypes.html >>> >>> Is messing with it necessary for later work? If not, is it a worthwhile >>> improvement? >>> >> >> Not strictly necessary, but if the expression checkers here don't >> *require* the type be an ordereddict, why bother to enforce that here? >> >> It's just a bid to slacken the type (my type hints will look for Dict, >> not OrderedDict) and leave the use of OrderedDict as an "implementation >> detail" that only parser.py knows about. > > "Orderedness" is anything but a detail only parser.py knows about. > > Example: > > { 'command': 'blockdev-insert-medium', > 'data': { 'id': 'str', > 'node-name': 'str'} } > > AST: > > OrderedDict([('command', 'blockdev-insert-medium'), > ('data', > OrderedDict([('id', {'type': 'str'}), > ('node-name', {'type': 'str'})]))]) > > For the inner dictionary, order matters, because the difference between > > void qmp_blockdev_insert_medium(const char *id, const char *node_name, > Error **errp); > > and > > void qmp_blockdev_insert_medium(const char *node_name, const char *id, > Error **errp); > > matters. > > For the outer dictionary, order carries no semantic meaning. > > My point is: parser.py fundamentally builds *ordered* dicts. We're > certainly free to relax them to more general types wherever > "orderedness" is not needed. However, one of the main aims of this > typing exercise is to make the code easier to read, and I doubt making > things more general helps there. > I primarily I saw the writing on the wall that we *will* be abandoning the use of OrderedDict and so I preferred to type in terms of just Dict, using the fact that Dict < OrderedDict anyway, asserting that parser.py uses OrderedDict as an "implementation detail". Later, parser.py may abandon its use of OrderedDict without changes to the rest of the code. The alternative is to use OrderedDict everywhere here in expr.py, but I would *prefer* not to, as it will inhibit prototyping and experimentation efforts where we might use a parser that doesn't use OrderedDict. What I absolutely did not want to do was type in terms of Dict[str, object] but then use isinstance checks for OrderedDict. My preference is still, I think, to just go all-in on dict. I am personally comfortable trusting that parser.py creates an ordered implementation of the type. As for these specific checks: - normalize_members doesn't assert that it has an OrderedDict, it only normalizes *if* it gets one. I don't think this is helpful behavior. - check_type has an error message that doesn't square with the check: we can give it a dict and it will pretend like we didn't give it one. I don't think that's helpful either. > Related: the type aliases for the AST you define later in this series. > I figure relaxing these to more general types where possible would > actually reduce their value. TopLevelExpression tells me more than > dict. > > I'm not against relaxing types per se. Judicious relaxation is often > useful to keep code more generally useful. When to relax isn't always > obvious. > >> (I needed to change it for prototyping using an off-the-shelf parser, so >> it was annoying to have it check for a stronger type if it doesn't >> absolutely have to.) > > If your off-the-shelf parse doesn't preserve order, it's not fit for the > purpose :) > It does, but in 3.6 that might be relying on CPython details. This is a pretty frustrating place to be in, support-wise. >>>> Signed-off-by: John Snow <jsnow@redhat.com> >>>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com> >>>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
John Snow <jsnow@redhat.com> writes: > On 2/25/21 5:40 AM, Markus Armbruster wrote: >> John Snow <jsnow@redhat.com> writes: >> >>> On 2/24/21 4:30 AM, Markus Armbruster wrote: >>>> John Snow <jsnow@redhat.com> writes: >>>> >>>>> OrderedDict is a subtype of dict, so we can check for a more general >>>>> form. These functions do not themselves depend on it being any >>>>> particular type. >>>> >>>> True. The actual arguments can only be OrderedDict, though. I think we >>>> refrained from relaxing to dict in these helpers because we felt >>>> "staying ordered" is clearer. >>>> >>> >>> As a habit, I tend towards declaring the least specific type possible >>> for input and declaring the most specific type possible for output. >> >> This maximimizes generality, which can be quite worthwhile. Maximizing >> generality by default is not a bad habit, I guess. But cases exist >> where generality is not needed, and other considerations take >> precedence. >> >>>> We're *this* close to mooting the point, because >>>> >>>> Changed in version 3.7: Dictionary order is guaranteed to be >>>> insertion order. This behavior was an implementation detail of >>>> CPython from 3.6. >>>> >>>> https://docs.python.org/3.7/library/stdtypes.html >>>> >>>> Is messing with it necessary for later work? If not, is it a worthwhile >>>> improvement? >>>> >>> >>> Not strictly necessary, but if the expression checkers here don't >>> *require* the type be an ordereddict, why bother to enforce that here? >>> >>> It's just a bid to slacken the type (my type hints will look for Dict, >>> not OrderedDict) and leave the use of OrderedDict as an "implementation >>> detail" that only parser.py knows about. >> >> "Orderedness" is anything but a detail only parser.py knows about. >> >> Example: >> >> { 'command': 'blockdev-insert-medium', >> 'data': { 'id': 'str', >> 'node-name': 'str'} } >> >> AST: >> >> OrderedDict([('command', 'blockdev-insert-medium'), >> ('data', >> OrderedDict([('id', {'type': 'str'}), >> ('node-name', {'type': 'str'})]))]) >> >> For the inner dictionary, order matters, because the difference between >> >> void qmp_blockdev_insert_medium(const char *id, const char *node_name, >> Error **errp); >> >> and >> >> void qmp_blockdev_insert_medium(const char *node_name, const char *id, >> Error **errp); >> >> matters. >> >> For the outer dictionary, order carries no semantic meaning. >> >> My point is: parser.py fundamentally builds *ordered* dicts. We're >> certainly free to relax them to more general types wherever >> "orderedness" is not needed. However, one of the main aims of this >> typing exercise is to make the code easier to read, and I doubt making >> things more general helps there. >> > > I primarily I saw the writing on the wall that we *will* be abandoning > the use of OrderedDict and so I preferred to type in terms of just Dict, > using the fact that Dict < OrderedDict anyway, asserting that parser.py > uses OrderedDict as an "implementation detail". > > Later, parser.py may abandon its use of OrderedDict without changes to > the rest of the code. I'm tempted to do it now. This could theoretically break running the QAPI code generator with a Python 3.6 other than CPython. Would we care? If not, we can both have what we want: I can have the same type everywhere (consistent, obvious), and you can have dict in expr.py. It would also cut through the somewhat pedantic argument we're having on "all objects in the AST are OrderedDict, always" [me] vs. "OrderedDict is a detail I'd prefer to forget at first opportunity [totally and intentionally unfair caricature of you ;-P] > The alternative is to use OrderedDict everywhere here in expr.py, but I > would *prefer* not to, as it will inhibit prototyping and > experimentation efforts where we might use a parser that doesn't use > OrderedDict. > > What I absolutely did not want to do was type in terms of Dict[str, > object] but then use isinstance checks for OrderedDict. > > My preference is still, I think, to just go all-in on dict. I am > personally comfortable trusting that parser.py creates an ordered > implementation of the type. ... and nothing else creates instances. > As for these specific checks: > > - normalize_members doesn't assert that it has an OrderedDict, it only > normalizes *if* it gets one. I don't think this is helpful behavior. > > - check_type has an error message that doesn't square with the check: we > can give it a dict and it will pretend like we didn't give it one. I > don't think that's helpful either. I figure you mean raise QAPISemError(info, "%s should be an object or type name" % source) The error message says "object". It's about the part to the right of the ':' in this grammar production: MEMBER = STRING : TYPE-REF | STRING : { 'type': TYPE-REF, '*if': COND, '*features': FEATURES } "Pidgin-JSON object" would be more exact than "object", but also more confusing. (Pidgin-JSON) objects are represented by OrderedDict in the AST. Giving check_type() something that is not an AST is out of spec. check_type() happens to cope with dict as well, but only because the difference to OrderedDict doesn't matter for this particular piece of the AST. I don't understand what "doesn't square". >> Related: the type aliases for the AST you define later in this series. >> I figure relaxing these to more general types where possible would >> actually reduce their value. TopLevelExpression tells me more than >> dict. >> >> I'm not against relaxing types per se. Judicious relaxation is often >> useful to keep code more generally useful. When to relax isn't always >> obvious. >> >>> (I needed to change it for prototyping using an off-the-shelf parser, so >>> it was annoying to have it check for a stronger type if it doesn't >>> absolutely have to.) >> >> If your off-the-shelf parse doesn't preserve order, it's not fit for the >> purpose :) >> > > It does, but in 3.6 that might be relying on CPython details. This is a > pretty frustrating place to be in, support-wise. Santa will bring us Python 3.6 EOL this year. Thanks, Santa, can't wait! (Yes, I know, we'll likely have to endure the frustration even longer for the sake of "distributions with long-lifetime releases".) >>>>> Signed-off-by: John Snow <jsnow@redhat.com> >>>>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com> >>>>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py index 35695c4c653..5694c501fa3 100644 --- a/scripts/qapi/expr.py +++ b/scripts/qapi/expr.py @@ -14,7 +14,6 @@ # This work is licensed under the terms of the GNU GPL, version 2. # See the COPYING file in the top-level directory. -from collections import OrderedDict import re from .common import c_name @@ -131,7 +130,7 @@ def check_if_str(ifcond): def normalize_members(members): - if isinstance(members, OrderedDict): + if isinstance(members, dict): for key, arg in members.items(): if isinstance(arg, dict): continue @@ -162,7 +161,7 @@ def check_type(value, info, source, if not allow_dict: raise QAPISemError(info, "%s should be a type name" % source) - if not isinstance(value, OrderedDict): + if not isinstance(value, dict): raise QAPISemError(info, "%s should be an object or type name" % source)