Message ID | 20240805201719.2345596-1-tavip@google.com (mailing list archive) |
---|---|
Headers | show |
Series | NXP i.MX RT595, ARM SVD and device model unit tests | expand |
On Mon, 5 Aug 2024 at 21:17, Octavian Purdila <tavip@google.com> wrote: > It also introduces unit tests for device models. To allow accessing > registers from unit tests a system bus mock is created. The main > advantage of unit tests for device models over QTest is that device > models can be tested in isolation and do not require a full qemu > machine. On the other hand the disadvantage is that you need to add a bunch of extra code to mock the interfaces that the device connects to, and then you compile into the test binary a subset of C files which weren't written with the expectation that they got compiled into tests like that, so it feels a bit brittle. The nice thing about qtest is that it doesn't need you to do any of that -- you just run the QEMU machine model and prod the devices it already has. Do you have examples in this series of tests which you were able to write with this unit test setup that you wouldn't have been able to write equivalents of with the qtest framework ? thanks -- PMM
On Mon, Aug 12, 2024 at 05:10:52PM +0100, Peter Maydell wrote: > On Mon, 5 Aug 2024 at 21:17, Octavian Purdila <tavip@google.com> wrote: > > It also introduces unit tests for device models. To allow accessing > > registers from unit tests a system bus mock is created. The main > > advantage of unit tests for device models over QTest is that device > > models can be tested in isolation and do not require a full qemu > > machine. > > On the other hand the disadvantage is that you need to add a > bunch of extra code to mock the interfaces that the device > connects to, and then you compile into the test binary a > subset of C files which weren't written with the expectation > that they got compiled into tests like that, so it feels a > bit brittle. The nice thing about qtest is that it doesn't > need you to do any of that -- you just run the QEMU machine > model and prod the devices it already has. Yep, mocking often ends up being rather a double-edged sword. You can do some really powerful things with it, and in particular it can enable you test scenarios / code paths that are hard or impractical to exercise with regular functional tests where fault injection is difficult/impossible. The cost of this though is that the mocks can pose a significant ongoing maint burden on anyone who refactors code in future, as the mocks need to evolve in lockstep with the code refactoring. 12 months down the line, it can be hard for another maintainer to understand the full purpose of the mocks & thus even harder to know what todo as they refactor the code. The future burden of mocks rarely falls on the person who creates them in the first place. > Do you have examples in this series of tests which you > were able to write with this unit test setup that you > wouldn't have been able to write equivalents of with the > qtest framework ? With regards, Daniel
On Mon, Aug 12, 2024 at 9:22 AM Daniel P. Berrangé <berrange@redhat.com> wrote: > > On Mon, Aug 12, 2024 at 05:10:52PM +0100, Peter Maydell wrote: > > On Mon, 5 Aug 2024 at 21:17, Octavian Purdila <tavip@google.com> wrote: > > > It also introduces unit tests for device models. To allow accessing > > > registers from unit tests a system bus mock is created. The main > > > advantage of unit tests for device models over QTest is that device > > > models can be tested in isolation and do not require a full qemu > > > machine. > > > > On the other hand the disadvantage is that you need to add a > > bunch of extra code to mock the interfaces that the device > > connects to, and then you compile into the test binary a > > subset of C files which weren't written with the expectation > > that they got compiled into tests like that, so it feels a > > bit brittle. The nice thing about qtest is that it doesn't > > need you to do any of that -- you just run the QEMU machine > > model and prod the devices it already has. > > Yep, mocking often ends up being rather a double-edged sword. > You can do some really powerful things with it, and in particular > it can enable you test scenarios / code paths that are hard or > impractical to exercise with regular functional tests where fault > injection is difficult/impossible. > > The cost of this though is that the mocks can pose a significant > ongoing maint burden on anyone who refactors code in future, as > the mocks need to evolve in lockstep with the code refactoring. > > 12 months down the line, it can be hard for another maintainer > to understand the full purpose of the mocks & thus even harder > to know what todo as they refactor the code. The future burden > of mocks rarely falls on the person who creates them in the > first place. > I agree that there are potentially maintenance issues. For what is worth, AFAIR, we only ran into one breaking change in about 3 years for which we used this approach which was also simple to fix (gpio was removed from hwcore so we had to pull in gpio.c directly). > > Do you have examples in this series of tests which you > > were able to write with this unit test setup that you > > wouldn't have been able to write equivalents of with the > > qtest framework ? > No, I think all of the tests in the patch set could be written in qtest.