| Message ID | 20230119212317.8324-13-rick.p.edgecombe@intel.com (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | Shadow stacks for userspace | expand |
On Thu, Jan 19, 2023 at 01:22:50PM -0800, Rick Edgecombe wrote: > From: Yu-cheng Yu <yu-cheng.yu@intel.com> > > When shadow stack is in use, Write=0,Dirty=1 PTE are preserved for > shadow stack. Copy-on-write PTEs then have Write=0,Cow=1. > > When a PTE goes from Write=1,Dirty=1 to Write=0,Cow=1, it could > become a transient shadow stack PTE in two cases: > > 1. Some processors can start a write but end up seeing a Write=0 PTE by > the time they get to the Dirty bit, creating a transient shadow stack > PTE. However, this will not occur on processors supporting shadow > stack, and a TLB flush is not necessary. > > 2. When _PAGE_DIRTY is replaced with _PAGE_COW non-atomically, a transient > shadow stack PTE can be created as a result. Thus, prevent that with > cmpxchg. > > In the case of pmdp_set_wrprotect(), for nopmd configs the ->pmd operated > on does not exist and the logic would need to be different. Although the > extra functionality will normally be optimized out when user shadow > stacks are not configured, also exclude it in the preprocessor stage so > that it will still compile. User shadow stack is not supported there by > Linux anyway. Leave the cpu_feature_enabled() check so that the > functionality also gets disabled based on runtime detection of the > feature. > > Similarly, compile it out in ptep_set_wrprotect() due to a clang warning > on i386. Like above, the code path should get optimized out on i386 > since shadow stack is not supported on 32 bit kernels, but this makes > the compiler happy. > > Dave Hansen, Jann Horn, Andy Lutomirski, and Peter Zijlstra provided many > insights to the issue. Jann Horn provided the cmpxchg solution. > > Tested-by: Pengfei Xu <pengfei.xu@intel.com> > Tested-by: John Allen <john.allen@amd.com> > Signed-off-by: Yu-cheng Yu <yu-cheng.yu@intel.com> Reviewed-by: Kees Cook <keescook@chromium.org>
diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 7942eff2af50..c5047eb5f406 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1232,6 +1232,23 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { +#ifdef CONFIG_X86_USER_SHADOW_STACK + /* + * Avoid accidentally creating shadow stack PTEs + * (Write=0,Dirty=1). Use cmpxchg() to prevent races with + * the hardware setting Dirty=1. + */ + if (cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) { + pte_t old_pte, new_pte; + + old_pte = READ_ONCE(*ptep); + do { + new_pte = pte_wrprotect(old_pte); + } while (!try_cmpxchg(&ptep->pte, &old_pte.pte, new_pte.pte)); + + return; + } +#endif clear_bit(_PAGE_BIT_RW, (unsigned long *)&ptep->pte); } @@ -1284,6 +1301,26 @@ static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm, static inline void pmdp_set_wrprotect(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp) { +#ifdef CONFIG_X86_USER_SHADOW_STACK + /* + * If shadow stack is enabled, pmd_wrprotect() moves _PAGE_DIRTY + * to _PAGE_COW (see comments at pmd_wrprotect()). + * When a thread reads a RW=1, Dirty=0 PMD and before changing it + * to RW=0, Dirty=0, another thread could have written to the page + * and the PMD is RW=1, Dirty=1 now. + */ + if (cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) { + pmd_t old_pmd, new_pmd; + + old_pmd = READ_ONCE(*pmdp); + do { + new_pmd = pmd_wrprotect(old_pmd); + } while (!try_cmpxchg(&pmdp->pmd, &old_pmd.pmd, new_pmd.pmd)); + + return; + } +#endif + clear_bit(_PAGE_BIT_RW, (unsigned long *)pmdp); }