| Message ID | 20231110135348.245156-8-nrb@linux.ibm.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | [kvm-unit-tests,GIT,PULL,01/26] s390x: spec_ex: load full register | expand |
On 10/11/2023 14.52, Nico Boehr wrote: > Until now, kvm-unit-tests has aligned guests to 1 MB in the host virtual > address space. Unfortunately, some s390x environments require guests to > be 2GB aligned in the host virtual address space, preventing > kvm-unit-tests which act as a hypervisor from running there. > > We can't easily put guests at address 0, since we want to be able to run > with MSO/MSL without having to maintain separate page tables for the > guest physical memory. 2GB is also not a good choice, since the > alloc_pages allocator will place its metadata there when the host has > more than 2GB of memory. In addition, we also want a bit of space after > the end of the host physical memory to be able to catch accesses beyond > the end of physical memory. > > The vmalloc allocator unfortunately allocates memory starting at the > highest virtual address which is not suitable for guest memory either > due to additional constraints of some environments. > > The physical page allocator in memalign_pages() is also not a optimal > choice, since every test running SIE would then require at least 4GB+1MB > of physical memory. > > This results in a few quite complex allocation requirements, hence add a > new function sie_guest_alloc() which allocates memory for a guest and > then establishes a properly aligned virtual space mapping. > > Rework snippet test and sie tests to use the new sie_guest_alloc() > function. > > Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com> > Link: https://lore.kernel.org/r/20231106170849.1184162-3-nrb@linux.ibm.com > Signed-off-by: Nico Boehr <nrb@linux.ibm.com> > --- > lib/s390x/sie.h | 2 ++ > lib/s390x/snippet.h | 9 +++------ > lib/s390x/sie.c | 42 ++++++++++++++++++++++++++++++++++++++++++ > s390x/sie.c | 4 ++-- > 4 files changed, 49 insertions(+), 8 deletions(-) Hi Nico! a colleague of mine (Jan) told me today that the current SIE-related tests of the kvm-unit-tests are failing when being run from a KVM guest (i.e. when we're testing a double-nested scenario). I've bisected the problem and ended up with this patch here. Could you please check whether "./run_tests.sh sie mvpg-sie spec_ex-sie" still works for you from within a KVM guest? Thanks, Thomas
On Wed, 2023-11-22 at 12:06 +0100, Thomas Huth wrote: > On 10/11/2023 14.52, Nico Boehr wrote: > > Until now, kvm-unit-tests has aligned guests to 1 MB in the host virtual > > address space. Unfortunately, some s390x environments require guests to > > be 2GB aligned in the host virtual address space, preventing > > kvm-unit-tests which act as a hypervisor from running there. > > > > We can't easily put guests at address 0, since we want to be able to run > > with MSO/MSL without having to maintain separate page tables for the > > guest physical memory. 2GB is also not a good choice, since the > > alloc_pages allocator will place its metadata there when the host has > > more than 2GB of memory. In addition, we also want a bit of space after > > the end of the host physical memory to be able to catch accesses beyond > > the end of physical memory. > > > > The vmalloc allocator unfortunately allocates memory starting at the > > highest virtual address which is not suitable for guest memory either > > due to additional constraints of some environments. > > > > The physical page allocator in memalign_pages() is also not a optimal > > choice, since every test running SIE would then require at least 4GB+1MB > > of physical memory. > > > > This results in a few quite complex allocation requirements, hence add a > > new function sie_guest_alloc() which allocates memory for a guest and > > then establishes a properly aligned virtual space mapping. > > > > Rework snippet test and sie tests to use the new sie_guest_alloc() > > function. > > > > Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com> > > Link: https://lore.kernel.org/r/20231106170849.1184162-3-nrb@linux.ibm.com > > Signed-off-by: Nico Boehr <nrb@linux.ibm.com> > > --- > > lib/s390x/sie.h | 2 ++ > > lib/s390x/snippet.h | 9 +++------ > > lib/s390x/sie.c | 42 ++++++++++++++++++++++++++++++++++++++++++ > > s390x/sie.c | 4 ++-- > > 4 files changed, 49 insertions(+), 8 deletions(-) > > Hi Nico! > > a colleague of mine (Jan) told me today that the current SIE-related tests > of the kvm-unit-tests are failing when being run from a KVM guest (i.e. when > we're testing a double-nested scenario). I've bisected the problem and ended > up with this patch here. > Could you please check whether "./run_tests.sh sie mvpg-sie spec_ex-sie" > still works for you from within a KVM guest? If you're getting a validity intercept, this should be fixed by KVM: s390: vsie: fix wrong VIR 37 when MSO is used https://lore.kernel.org/kvm/20231102153549.53984-1-imbrenda@linux.ibm.com/ > > Thanks, > Thomas > >
On 23/11/2023 10.24, Nina Schoetterl-Glausch wrote: > On Wed, 2023-11-22 at 12:06 +0100, Thomas Huth wrote: >> On 10/11/2023 14.52, Nico Boehr wrote: >>> Until now, kvm-unit-tests has aligned guests to 1 MB in the host virtual >>> address space. Unfortunately, some s390x environments require guests to >>> be 2GB aligned in the host virtual address space, preventing >>> kvm-unit-tests which act as a hypervisor from running there. >>> >>> We can't easily put guests at address 0, since we want to be able to run >>> with MSO/MSL without having to maintain separate page tables for the >>> guest physical memory. 2GB is also not a good choice, since the >>> alloc_pages allocator will place its metadata there when the host has >>> more than 2GB of memory. In addition, we also want a bit of space after >>> the end of the host physical memory to be able to catch accesses beyond >>> the end of physical memory. >>> >>> The vmalloc allocator unfortunately allocates memory starting at the >>> highest virtual address which is not suitable for guest memory either >>> due to additional constraints of some environments. >>> >>> The physical page allocator in memalign_pages() is also not a optimal >>> choice, since every test running SIE would then require at least 4GB+1MB >>> of physical memory. >>> >>> This results in a few quite complex allocation requirements, hence add a >>> new function sie_guest_alloc() which allocates memory for a guest and >>> then establishes a properly aligned virtual space mapping. >>> >>> Rework snippet test and sie tests to use the new sie_guest_alloc() >>> function. >>> >>> Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com> >>> Link: https://lore.kernel.org/r/20231106170849.1184162-3-nrb@linux.ibm.com >>> Signed-off-by: Nico Boehr <nrb@linux.ibm.com> >>> --- >>> lib/s390x/sie.h | 2 ++ >>> lib/s390x/snippet.h | 9 +++------ >>> lib/s390x/sie.c | 42 ++++++++++++++++++++++++++++++++++++++++++ >>> s390x/sie.c | 4 ++-- >>> 4 files changed, 49 insertions(+), 8 deletions(-) >> >> Hi Nico! >> >> a colleague of mine (Jan) told me today that the current SIE-related tests >> of the kvm-unit-tests are failing when being run from a KVM guest (i.e. when >> we're testing a double-nested scenario). I've bisected the problem and ended >> up with this patch here. >> Could you please check whether "./run_tests.sh sie mvpg-sie spec_ex-sie" >> still works for you from within a KVM guest? > > If you're getting a validity intercept, this should be fixed by > KVM: s390: vsie: fix wrong VIR 37 when MSO is used > https://lore.kernel.org/kvm/20231102153549.53984-1-imbrenda@linux.ibm.com/ Yes, thanks, I can confirm that this fixes the issue indeed! Thomas
Quoting Nina Schoetterl-Glausch (2023-11-23 10:24:12) > On Wed, 2023-11-22 at 12:06 +0100, Thomas Huth wrote: > > On 10/11/2023 14.52, Nico Boehr wrote: > > > Until now, kvm-unit-tests has aligned guests to 1 MB in the host virtual > > > address space. Unfortunately, some s390x environments require guests to > > > be 2GB aligned in the host virtual address space, preventing > > > kvm-unit-tests which act as a hypervisor from running there. > > > > > > We can't easily put guests at address 0, since we want to be able to run > > > with MSO/MSL without having to maintain separate page tables for the > > > guest physical memory. 2GB is also not a good choice, since the > > > alloc_pages allocator will place its metadata there when the host has > > > more than 2GB of memory. In addition, we also want a bit of space after > > > the end of the host physical memory to be able to catch accesses beyond > > > the end of physical memory. > > > > > > The vmalloc allocator unfortunately allocates memory starting at the > > > highest virtual address which is not suitable for guest memory either > > > due to additional constraints of some environments. > > > > > > The physical page allocator in memalign_pages() is also not a optimal > > > choice, since every test running SIE would then require at least 4GB+1MB > > > of physical memory. > > > > > > This results in a few quite complex allocation requirements, hence add a > > > new function sie_guest_alloc() which allocates memory for a guest and > > > then establishes a properly aligned virtual space mapping. > > > > > > Rework snippet test and sie tests to use the new sie_guest_alloc() > > > function. > > > > > > Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com> > > > Link: https://lore.kernel.org/r/20231106170849.1184162-3-nrb@linux.ibm.com > > > Signed-off-by: Nico Boehr <nrb@linux.ibm.com> > > > --- > > > lib/s390x/sie.h | 2 ++ > > > lib/s390x/snippet.h | 9 +++------ > > > lib/s390x/sie.c | 42 ++++++++++++++++++++++++++++++++++++++++++ > > > s390x/sie.c | 4 ++-- > > > 4 files changed, 49 insertions(+), 8 deletions(-) > > > > Hi Nico! > > > > a colleague of mine (Jan) told me today that the current SIE-related tests > > of the kvm-unit-tests are failing when being run from a KVM guest (i.e. when > > we're testing a double-nested scenario). I've bisected the problem and ended > > up with this patch here. > > Could you please check whether "./run_tests.sh sie mvpg-sie spec_ex-sie" > > still works for you from within a KVM guest? > > If you're getting a validity intercept, this should be fixed by > KVM: s390: vsie: fix wrong VIR 37 when MSO is used > https://lore.kernel.org/kvm/20231102153549.53984-1-imbrenda@linux.ibm.com/ After discussion with Thomas offline: yes it was a validity and yes the patch mentioned by Nina fixes it. Thanks.
diff --git a/lib/s390x/sie.h b/lib/s390x/sie.h index 147cb0f..c1724cf 100644 --- a/lib/s390x/sie.h +++ b/lib/s390x/sie.h @@ -285,4 +285,6 @@ void sie_guest_sca_create(struct vm *vm); void sie_guest_create(struct vm *vm, uint64_t guest_mem, uint64_t guest_mem_len); void sie_guest_destroy(struct vm *vm); +uint8_t *sie_guest_alloc(uint64_t guest_size); + #endif /* _S390X_SIE_H_ */ diff --git a/lib/s390x/snippet.h b/lib/s390x/snippet.h index 11ec54c..910849a 100644 --- a/lib/s390x/snippet.h +++ b/lib/s390x/snippet.h @@ -125,14 +125,11 @@ static inline void snippet_pv_init(struct vm *vm, const char *gbin, /* Allocates and sets up a snippet based guest */ static inline void snippet_setup_guest(struct vm *vm, bool is_pv) { - u8 *guest; - - /* Allocate 1MB as guest memory */ - guest = alloc_pages(8); - memset(guest, 0, HPAGE_SIZE); + const unsigned long guest_size = SZ_1M; + uint8_t *guest_start = sie_guest_alloc(guest_size); /* Initialize the vm struct and allocate control blocks */ - sie_guest_create(vm, (uint64_t)guest, HPAGE_SIZE); + sie_guest_create(vm, (uint64_t)guest_start, guest_size); if (is_pv) { /* FMT4 needs a ESCA */ diff --git a/lib/s390x/sie.c b/lib/s390x/sie.c index b44febd..97a093b 100644 --- a/lib/s390x/sie.c +++ b/lib/s390x/sie.c @@ -15,6 +15,8 @@ #include <asm/page.h> #include <libcflat.h> #include <alloc_page.h> +#include <vmalloc.h> +#include <sclp.h> void sie_expect_validity(struct vm *vm) { @@ -111,6 +113,46 @@ void sie_guest_create(struct vm *vm, uint64_t guest_mem, uint64_t guest_mem_len) vm->sblk->crycbd = (uint32_t)(uintptr_t)vm->crycb; } +/** + * sie_guest_alloc() - Allocate memory for a guest and map it in virtual address + * space such that it is properly aligned. + * @guest_size: the desired size of the guest in bytes. + */ +uint8_t *sie_guest_alloc(uint64_t guest_size) +{ + static unsigned long guest_counter = 1; + u8 *guest_phys, *guest_virt; + unsigned long i; + pgd_t *root; + + setup_vm(); + root = (pgd_t *)(stctg(1) & PAGE_MASK); + + /* + * Start of guest memory in host virtual space needs to be aligned to + * 2GB for some environments. It also can't be at 2GB since the memory + * allocator stores its page_states metadata there. + * Thus we use the next multiple of 4GB after the end of physical + * mapping. This also leaves space after end of physical memory so the + * page immediately after physical memory is guaranteed not to be + * present. + */ + guest_virt = (uint8_t *)ALIGN(get_ram_size() + guest_counter * 4UL * SZ_1G, SZ_2G); + guest_counter++; + + guest_phys = alloc_pages(get_order(guest_size) - 12); + /* + * Establish a new mapping of the guest memory so it can be 2GB aligned + * without actually requiring 2GB physical memory. + */ + for (i = 0; i < guest_size; i += PAGE_SIZE) { + install_page(root, __pa(guest_phys + i), guest_virt + i); + } + memset(guest_virt, 0, guest_size); + + return guest_virt; +} + /* Frees the memory that was gathered on initialization */ void sie_guest_destroy(struct vm *vm) { diff --git a/s390x/sie.c b/s390x/sie.c index cd3cea1..ce5b606 100644 --- a/s390x/sie.c +++ b/s390x/sie.c @@ -89,8 +89,8 @@ static void setup_guest(void) { setup_vm(); - /* Allocate 1MB as guest memory */ - guest = alloc_pages(8); + guest = sie_guest_alloc(SZ_1M); + /* The first two pages are the lowcore */ guest_instr = guest + PAGE_SIZE * 2;