From patchwork Fri May 18 15:55:46 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: David Vrabel X-Patchwork-Id: 10411125 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 47F846031B for ; Fri, 18 May 2018 16:34:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 35B1A28737 for ; Fri, 18 May 2018 16:34:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3246428A55; Fri, 18 May 2018 16:34:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 566D128A84 for ; Fri, 18 May 2018 16:32:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751999AbeERQcB (ORCPT ); Fri, 18 May 2018 12:32:01 -0400 Received: from mx0b-002c1b01.pphosted.com ([148.163.155.12]:53118 "EHLO mx0b-002c1b01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751693AbeERQb7 (ORCPT ); Fri, 18 May 2018 12:31:59 -0400 X-Greylist: delayed 2140 seconds by postgrey-1.27 at vger.kernel.org; Fri, 18 May 2018 12:31:59 EDT Received: from pps.filterd (m0127842.ppops.net [127.0.0.1]) by mx0b-002c1b01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w4IFsolf002088; Fri, 18 May 2018 08:56:13 -0700 Received: from nam01-by2-obe.outbound.protection.outlook.com (mail-by2nam01lp0178.outbound.protection.outlook.com [216.32.181.178]) by mx0b-002c1b01.pphosted.com with ESMTP id 2j21b3g3ha-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Fri, 18 May 2018 08:56:13 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XXrOvKhAIXU8IjrZWLySwDVi5CoPVOZ2MCB8NOODvzo=; b=FXMV0OoF+v5Q7NiFLLB0xQWZTlfKQqtB3CXMcLLhZsrUDroGN8ci9sIQC25kPfD9jrFyeAWYAT39+bv36mBb33N4C/otkjxSu25o3BlXwq2OuM070so11hxP8ZzZyI18nMtLoWYX6Rk6ojewu+TOimkGaJoW3VakOaXjh3n0qKc= Received: from tomato.dev.eng.nutanix.com (62.254.189.133) by SN1PR02MB3759.namprd02.prod.outlook.com (2603:10b6:802:31::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.776.11; Fri, 18 May 2018 15:56:10 +0000 From: David Vrabel To: kvm@vger.kernel.org Cc: David Vrabel , Wanpeng Li , Paolo Bonzini , =?UTF-8?q?Radim=20Kr=C4=8Dm=C3=A1=C5=99?= Subject: [PATCH] x86/kvm: fix LAPIC timer drift when guest uses periodic mode Date: Fri, 18 May 2018 16:55:46 +0100 Message-Id: <20180518155546.19879-1-david.vrabel@nutanix.com> X-Mailer: git-send-email 2.11.0 MIME-Version: 1.0 X-Originating-IP: [62.254.189.133] X-ClientProxiedBy: AM5PR04CA0002.eurprd04.prod.outlook.com (2603:10a6:206:1::15) To SN1PR02MB3759.namprd02.prod.outlook.com (2603:10b6:802:31::17) X-MS-PublicTrafficType: Email X-Microsoft-Antispam: UriScan:(228109839391802); BCL:0; PCL:0; RULEID:(7020095)(4652020)(8989080)(5600026)(4534165)(4627221)(201703031133081)(201702281549075)(8990040)(2017052603328)(7153060)(7193020); SRVR:SN1PR02MB3759; X-Microsoft-Exchange-Diagnostics: 1; SN1PR02MB3759; 3:+Gp3Hf/kRPG4uX3z8vzdqgzQ6GXPWJpA7HWGlF3UAjT02/34eSQcVyhJcnkKGOZlC//9cN4lfQQntGO/wB5j3TEhgNV61t17a7EuWA3NPGTWErp2ljTngMP28VSumd4/pFY0uWcTjHAN3sHulgEYZQTLJyXfFiSKfZyNbt1oQ7+bQrF1SGa1zQ/c0lB5EYNnWmGwh8MVbm8wpO9csjc0Rv76aPl6fg9LAJiKn8kKci1K1XGsh5g77czgjlMS3CxljV9iU1mv9O/dqg4/GokAPHTX7HO587B+59zZ8LvwZVQ=; 25:ZzP0qzln6Bvh0rVYZxkY/SbTTFu6uf2f/3IAw+2Tzx0/S3xI+DlnbmlmzoY855DjQUDpBxNQy9mquvGX3IdcEBrmW1TBykdZqa7Aqey4YSRjDdLPT5rDJLj30herPCsRpSdoeHWnLuOJavJIJpVypB/AiJNcO/nrlAV55ucHXH5XMKj9PF+nH6EpyZOZIGwL3zFfd00Rzs1iLsrFd3tMEqT+2pFTrzlEzusMFAuPQ+0RTCXh9O7SVgROW5YRYixYB84/BpWiLGpoBuvc1hWJ7IUpCmlSHJ8HRQ3i2DKEI5BDU1gB1mn/SRdu310r2QHebzdMVUGS4Csw4UXsoKzFTA==; 31:iAaE8+dr3lHb1T7aHPWYCqo/gr2L3DYfFCC16b7w+EOPgSFQkVWNysWMrJjy6npwOYI38cMjDKt8/JC5Yg63nYbdPWtntZy0jFCbiipvzOABNYAUMpMFF4zLqEFcXtssZjWghDGqB/taXMBhjL5ySGIKzbhGtIZh8BMup9ZEIjr16aYKZhV7wiH3nPcnf7fp1nowvB0ZBH1/ta310OdCH/Ld8MxHGrkoFimlnlXixGk= X-MS-TrafficTypeDiagnostic: SN1PR02MB3759: X-Microsoft-Exchange-Diagnostics: 1; SN1PR02MB3759; 20:UJY9nRnGTIEgcDTQajRXQhexOnoufasrsv1fVm74nRN62Lz5tYWTPQVocZnsgQRW52wD69liURXnTjjThgnJbzcvoRkUbdAvvH+IS1t1Bf3EiuMaraRBACdO6PUHwhxaBWoVvFjgqMO15TxVTEOxxM06XNI4rtGxhLXGU2Ikiq1JAWd4dswAgsPUSa9wv4h6WwLsDyk2eiJ1tiGjpgGmLe/HWu6V6YhSvV1AH7ZIYy4ePkWiqiPZVpSIUZaJ+RNPwNZKfY4plI/EX3tY6yBUhT3HJXb3/GpmMS+CaPuw/1+l8qAg/bkPaJp6QqPwoGPIHYY2szW7RTn0LHUgL5q/RlikkOmlWunPpr5vm8/dL3TCQ4gA8ObB549Okp/aCD9RvLQO5VOWT9l1duaq3BXPhdrwJzSFrSFDxvKF210V8w+9/8ehqg4jnc0Zt94dNySi1yhYhijhaXds5ape3CNr1YPYrBRTsIHPB8uHDdaXMVu2xYqYbBhyrG54kq0HLDbF5NIFAdVF7SyXQRat833egWSFc53tEN1487nMI3L3psF+o3NmOwddcclVxnSEh04oAkJlPmg9EJ+A+gp/wcTObqVqt/Z6kUIEM56FqHUet14= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(52384705835673)(228109839391802)(130873036417446)(194151415913766); X-MS-Exchange-SenderADCheck: 1 X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040522)(2401047)(5005006)(8121501046)(3231254)(944501410)(52105095)(10201501046)(3002001)(93006095)(93001095)(149027)(150027)(6041310)(20161123560045)(20161123564045)(20161123562045)(20161123558120)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(6072148)(201708071742011)(7699016); SRVR:SN1PR02MB3759; BCL:0; PCL:0; RULEID:; SRVR:SN1PR02MB3759; X-Microsoft-Exchange-Diagnostics: 1; SN1PR02MB3759; 4:hV02WJlkMc+y3c+2oaETO4pu+tyQN0n/y19fDNgKYDSSheqrinX1K4IRXYmQrRsKWhhuuT3Nu6w64I0PcODMQL2tY03LlB/uvPw+nkhXLr7/q3omrE+ZaNJYOJGoKkLl5+QsStQCPKiSQ9vHPupSJpj9h6tcPRzBuzl2uJh9xQbwD/tXs8dkT7amSawUDMCksCfdgV2bYH8mYbjm62u93XniA9wHyngI4V6eBufZ0JaUa3xDumhf0uXU44DHMynHl6U0hVCmc8kuMUraF/YS6kBh5+HKvYdNYNGJJVlZLlLazB7BwWPYSORLoFNUOW4VxHf6J1djJyzdiYlxv+3aDa66XoKnCyOKrFOicbw9sg7uy7lvzyMpuWuiVuMsZqY0tvhggoNU5W/6llxs5K7LugEdkakAv6bXUgPQecS3YMI= X-Forefront-PRVS: 0676F530A9 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10019020)(396003)(39380400002)(346002)(39860400002)(366004)(376002)(189003)(199004)(97736004)(54906003)(2870700001)(1857600001)(81166006)(956004)(6666003)(2616005)(6916009)(2906002)(50226002)(316002)(4326008)(25786009)(1076002)(53936002)(105586002)(476003)(8936002)(5660300001)(81156014)(106356001)(6486002)(486006)(47776003)(7736002)(2351001)(305945005)(39060400002)(86362001)(68736007)(2361001)(66066001)(186003)(36756003)(52116002)(16526019)(478600001)(59450400001)(26005)(6116002)(3846002)(45080400002)(7696005)(50466002)(386003)(23676004)(8676002)(44832011)(64030200001); DIR:OUT; SFP:1102; SCL:1; SRVR:SN1PR02MB3759; H:tomato.dev.eng.nutanix.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; Received-SPF: None (protection.outlook.com: nutanix.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtTTjFQUjAyTUIzNzU5OzIzOnNLc0dzSWpyUDJmTXlJbmxnMitkYnhUbUZ0?= =?utf-8?B?a3lKTjFsdWdDQ3ZCR05ZRUtpNkpPSTRpZ1dLZ0hhbHVEVnVjSms2RGhscmFP?= =?utf-8?B?dG1xb3E2WURCK1ZQbkErekVlOHZwWnZRVHlONWVDUVRLdS9pZ252Mm9ZSGFz?= =?utf-8?B?andFdG12NW9Tb1pKam03Z25SVWd1LzJnbGVXMnFlVlFsY1lOTlA4ZkVGZTNL?= =?utf-8?B?MXV0UlV5SExZUXhQTXMvMElpN2lSWTVmVnEzaG5ma0krYnpGVWc3L0RBSktW?= =?utf-8?B?d1g5WFN4M0VEaGh3WnJGdXVsZnBMM3ZOYWdUZ1pzYm1MNitSNkpGZTNuOUc4?= =?utf-8?B?d202aDNRYS9YS1BGa1BUYjdKdjN2dkpuYXpGakN2blF1N0dSbTRVam9mdkdk?= =?utf-8?B?OG5lRkQ1RnkvaFFHaFFjdjY0WS9EUEhVWnNTb3QwNnNTT1dBWW9pdnBXVzZy?= =?utf-8?B?dk1hVTFkaHZlNFIwWngyZGRQVWNxa1g1bDhYZExmQkt5K3I5eGVFeXQ0c2cv?= =?utf-8?B?dXExRW14eGx3aUdOQlNHYXVaT0dSUEEyQzZmZ1VPS0w1ZnI4akFMd2kwWll2?= =?utf-8?B?YmRTVWhxbm5OSWJmYUJVajlQVE5YOEl1ZDRGeFVXS1dCYTZEenBUK3RwMHM1?= =?utf-8?B?MmkrZ0pNVG1ob2swNnd6MU5PMEkvT0lzUkFCMGVYN1BTRmJHbHNHQU9vNjBz?= =?utf-8?B?TGdoUFVDZk9CSnhFRk1lV2dxTGlEN2hEbWxRMXFtZVNkTVQ0ZE1uMzFzMUl2?= =?utf-8?B?OEttV2p5bGx5aGk4UDk1eHY0Ykw4dDRVT3dvUW1jbTAvdnJaY0lCTlczc2VN?= =?utf-8?B?dzBMQnZ1SVBaRmdoYnBaT1RGWE1NeU9Uc1BWcW1KczBxQVhyUmdqSlJMZTFX?= =?utf-8?B?NEx3clBCR0RnMEJWMENUTE1LeE1BbWVOV1BGeTRud2d6YU9rRTlUSWFnMWdV?= =?utf-8?B?TjZqL1BBN3pOcWt6bGRZMXVXSS9TMVpEbk9KaVYvaDJPRjRUUGVBa1hUa3Uw?= =?utf-8?B?Z3BuQm81MDhFUjNtNUo3RGgyaVlyYzZJUjlmZldpRXp4YzduSjdoSGswWFhS?= =?utf-8?B?anlyaGs5cnNLWGJaY3NuQWQxWjN6QXZIS0sxQjB4eFRmZE1tbXpWeWY5Slgv?= =?utf-8?B?RVZFdlJscndHMG5ENnVsbGxxMXpjRFBhc0VVNzU0Y0kyNUF1Y0YrSm1Gek5i?= =?utf-8?B?ak5NRjlLZHA5cEZmTmRwOVRicDJaZHNKVGtvVVJFcGZYaWVIYzFkVFNxTklx?= =?utf-8?B?TnNrZ0FSYnRISFU3T1VyRjdyaExSR2RiWXdITkQ5TjVKTVg0SjNSUlRKTDJu?= =?utf-8?B?aTRLNDhzVldiMEsrUUd0WjRiS0JWUzZ2cnFwQmlvcm9kV2VyaU5yU0ppNW5V?= =?utf-8?B?Q0xzWVY5TFNNMlppL3Rpc1FLTldua290ZTJnNzRYZzIzVDR1a1R6K0hHTW5j?= =?utf-8?B?VExaWHozVW5DWTBSMEprdi9ZY1BYRlJNL3NQZzhnRnIxVW5uUm1VKzJaVytx?= =?utf-8?B?MkdLV1YzUWdWOW1CaGpkeGhFYXZMYUVVd2N5bUM5REozbHhzV01rN2JaeEFO?= =?utf-8?B?eGlId1MrWS9vY1IvN3JUdXJiU2xWbllvRFFwOTZtNm5lY295UnpWU3F1Nkla?= =?utf-8?B?elZiTG5GQ29xS0Y4L1JRT2N5cTdLYnl2eEFOMWtNWWIvaGdpemFWRFMwZFp5?= =?utf-8?B?dEFORDlFeDNkOTczL1hzV3NtNWNqVHRmSFlKMXQ4RFo3WnYwYTR3VmZkUER3?= =?utf-8?Q?EgvMWwMadVqCCk2bdFq9ptu5OR0xOidFT1hZc=3D?= X-Microsoft-Antispam-Message-Info: Ezswhe4kF6Bj298ackGUbJ9oOgUxKP0LSbuttUnh1ejltnsbk/wBSV0voDF4j1MZclrSWtVgiOcli8AuJfahyBsHj8BEiKWg3H6ptKP0ZiL63yM5pK1BlMM04oOC1CVseyaxBmdKYzOabKEf04Tgmp8g281paMR3mcTUId+oJRkBM0Qgl6TnZA7mRlIjhC+K X-Microsoft-Exchange-Diagnostics: 1; SN1PR02MB3759; 6:49kZEtv9/gbMqxiQE7I2T5+jYAxzUt1DdnIDp1FutJj2XsYV7JsnPsmnqLhX6+9YSirKSCmQatekatYcQGULSErl9RONt1Tz9rQKJal9n2sC8fkGXmBFExTyu8xqH8bors0P99TzBmvfbJK2aW2I7gm1xybQnXFBu9QF5SKu2qJKRJos7YeD2soCPqpWNiQzzOxH4BQfDcJW/QOEtXhUSZv/+0ls7Po4oC5s5bareltWP13n+y6ELosaU9bFktd9JxcGtjwdVGarwxsGfNJVrzikuZV7+O/itsIsDgU8YMoQQsU2kLq0pH/4s5JIpqvijrNZRQPFHvebp9FcP0gOM9+JR7izADl309H5So9CZ2AdDsVbuohqyU2HLt5pKuau+Yf0BVJQQo020ZvhOfUqDh7AxwLnQfDozsoGRiOHkCsGBP6cOY2sDS9oTh2ml737l6J2EP0wGfx1qLlJHIQmJQ==; 5:Jte77LjFfbi11ifz2KQThXMCxqr4hNdTClcxaMn4weumwQA4WV2dvEAxXUsrtmMSwuoezWlzXIztBuVd4abIGXBYJ7XqkvQUKgHkNsHXbdr3+GSg4yw/S48xEf+K4k7+mRjoKv8qAI21SpWuFdI4Uc5EnaGGxp/OUeNhHXJ2GTU=; 24:GBeVJdJHVjhtM2RpOFwLF8K6Srd2fxDORg4HHCQhhVz/l61iXvrnwDzMPquWGVYeI7dN560TjXQMAjUIPcDGg8mad/Rp9Mn7baiEbEu2CLc= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; SN1PR02MB3759; 7:CqsSwXgQRCPCYtwFJqWcprFEsM4ouYX1Q29kFwjv34hp/k3RUEpG9HLSTNGrdugzRzDF2robDjD5T6DR/6FaeElIY2MqJpx8+K/FAmDIG531NoYiiiEaK6MuKLjkPuctCoNewzzdxwtzyQXwIT1HqFwgyh3VsEbJaf1pjx2nhfow7CW/5fwUxRdzUi6QPDONhyYhm4yodFfUNEYaevrc90rrMuPLz+JjUB5nuyab4waUi3DAgYKtW1HDn/mJt+ER X-MS-Office365-Filtering-Correlation-Id: c6c879d3-7184-42f9-7b92-08d5bcd7e053 X-OriginatorOrg: nutanix.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 May 2018 15:56:10.0964 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c6c879d3-7184-42f9-7b92-08d5bcd7e053 X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: bb047546-786f-4de1-bd75-24e5b6f79043 X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN1PR02MB3759 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-05-18_06:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=13 phishscore=0 bulkscore=0 spamscore=0 clxscore=1011 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1711220000 definitions=main-1805180173 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Since 4.10, commit 8003c9ae204e (KVM: LAPIC: add APIC Timer periodic/oneshot mode VMX preemption timer support), guests using periodic LAPIC timers (such as FreeBSD 8.4) would see their timers drift significantly over time. Differences in the underlying clocks and numerical errors means the periods of the two timers (hv and sw) are not the same. This difference will accumulate with every expiry resulting in a large error between the hv and sw timer. This means the sw timer may be running slow when compared to the hv timer. When the timer is switched from hv to sw, the now active sw timer will expire late. The guest VCPU is reentered and it switches to using the hv timer. This timer catches up, injecting multiple IRQs into the guest (of which the guest only sees one as it does not get to run until the hv timer has caught up) and thus the guest's timer rate is low (and becomes increasing slower over time as the sw timer lags further and further behind). I believe a similar problem would occur if the hv timer is the slower one, but I have not observed this. Fix this by synchronizing the deadlines for both timers to the same time source on every tick. This prevents the errors from accumulating. Fixes: 8003c9ae204e21204e49816c5ea629357e283b06 Cc: Wanpeng Li Signed-off-by: David Vrabel --- Cc: Paolo Bonzini Cc: "Radim Krčmář" --- arch/x86/kvm/lapic.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index b74c9c1405b9..3773c4625114 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -1522,11 +1522,23 @@ static bool set_target_expiration(struct kvm_lapic *apic) static void advance_periodic_target_expiration(struct kvm_lapic *apic) { - apic->lapic_timer.tscdeadline += - nsec_to_cycles(apic->vcpu, apic->lapic_timer.period); + ktime_t now = ktime_get(); + u64 tscl = rdtsc(); + ktime_t delta; + + /* + * Synchronize both deadlines to the same time source or + * differences in the periods (caused by differences in the + * underlying clocks or numerical approximation errors) will + * cause the two to drift apart over time as the errors + * accumulate. + */ apic->lapic_timer.target_expiration = ktime_add_ns(apic->lapic_timer.target_expiration, apic->lapic_timer.period); + delta = ktime_sub(apic->lapic_timer.target_expiration, now); + apic->lapic_timer.tscdeadline = kvm_read_l1_tsc(apic->vcpu, tscl) + + nsec_to_cycles(apic->vcpu, delta); } static void start_sw_period(struct kvm_lapic *apic)