From patchwork Tue Jul 16 07:37:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jan Beulich X-Patchwork-Id: 11045393 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2457A1510 for ; Tue, 16 Jul 2019 07:44:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 14FF92858A for ; Tue, 16 Jul 2019 07:44:24 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0905B285A8; Tue, 16 Jul 2019 07:44:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 7DC922858A for ; Tue, 16 Jul 2019 07:44:23 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hnI71-0002NA-Az; Tue, 16 Jul 2019 07:42:59 +0000 Received: from us1-rack-dfw2.inumbo.com ([104.130.134.6]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hnI70-0002N0-Eb for xen-devel@lists.xenproject.org; Tue, 16 Jul 2019 07:42:58 +0000 X-Inumbo-ID: 537e95da-a79d-11e9-8980-bc764e045a96 Received: from m9a0001g.houston.softwaregrp.com (unknown [15.124.64.66]) by us1-rack-dfw2.inumbo.com (Halon) with ESMTPS id 537e95da-a79d-11e9-8980-bc764e045a96; Tue, 16 Jul 2019 07:42:56 +0000 (UTC) Received: FROM m9a0001g.houston.softwaregrp.com (15.121.0.190) BY m9a0001g.houston.softwaregrp.com WITH ESMTP; Tue, 16 Jul 2019 07:42:56 +0000 Received: from M4W0335.microfocus.com (2002:f78:1193::f78:1193) by M9W0067.microfocus.com (2002:f79:be::f79:be) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1591.10; Tue, 16 Jul 2019 07:37:37 +0000 Received: from NAM01-BY2-obe.outbound.protection.outlook.com (15.124.8.14) by M4W0335.microfocus.com (15.120.17.147) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1591.10 via Frontend Transport; Tue, 16 Jul 2019 07:37:37 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=V+mxOclmQl/Iwnmb3Ajz3NaVoYkQZnBwRiUvl/8JGlAOvFpNQQKJMCpM4KRDfjrDTdd/tnqUj51lctIUWTTn6E0ow6j0rPzADMIpnbuTcaGxNqyK2LKwMgP1MnKbilCGNAjGWCGEDrxcbS2ME+SeRne1y3q4uz9XLxTNm4TVO1U6A2CtEtE6jE+x9HDHFyqNOxqejhw5s1UntKIFbYHXfikS3lhnaQV/IxvgzQWFZUZ6M8lLJZL7+teF3jsOxW9TijfWzvJs8Dmeuw4s1aZe++vMHVxnoXSN1W0K06ZsuLSc6A7BMjaP9J8ltcQseWOYtvmvHksajlRC0wf38FVS5w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=5XsuRYlo9O9yePuFmC76k1W7jFTXusfwS+bbwzyeAUY=; b=XHfoJod67ahpFbFYqHKlaa6d1sBrAcFPiPF3o7NlBHbxVRtVePlqTIHr0ounVoy0jrkNQI6IOuyrN4YVHLrQG1tBZ0yCZ1x4rEcl+sHTuIJSDFaPR+OT72oZt1Zf5KCSJT8w5u36dCui6UDWdA3Iz3VhucaDtMggn9dr68be4nNuRGc9zuhH8eJEZKjM2zoS+EjLm+5DgsA/0wYKMDxPULmnRbzxAj0hqNg87GyE+XxnHY+zJ8YjiYUOC4AmN5OouQdGCvFP9vEzouTNFeSdylJiaD0N6US6pZHZ8C4bbBR3k2eCTfWIz+m378sPn/vDFswYeiqhlK6Qy0lF5hJU6w== ARC-Authentication-Results: i=1; mx.microsoft.com 1;spf=pass smtp.mailfrom=suse.com;dmarc=pass action=none header.from=suse.com;dkim=pass header.d=suse.com;arc=none Received: from DM6PR18MB3401.namprd18.prod.outlook.com (10.255.174.218) by DM6PR18MB2667.namprd18.prod.outlook.com (20.179.107.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2073.14; Tue, 16 Jul 2019 07:37:36 +0000 Received: from DM6PR18MB3401.namprd18.prod.outlook.com ([fe80::1fe:35f6:faf3:78c7]) by DM6PR18MB3401.namprd18.prod.outlook.com ([fe80::1fe:35f6:faf3:78c7%7]) with mapi id 15.20.2073.012; Tue, 16 Jul 2019 07:37:36 +0000 From: Jan Beulich To: "xen-devel@lists.xenproject.org" Thread-Topic: [PATCH v4 02/13] x86/IRQ: deal with move cleanup count state in fixup_irqs() Thread-Index: AQHVO6lWFnWLjuNRXkiFzFypmuh6cQ== Date: Tue, 16 Jul 2019 07:37:36 +0000 Message-ID: <54a42001-e741-3c4f-ba99-6b5e270099cf@suse.com> References: <5cda711a-b417-76e9-d113-ea838463f225@suse.com> In-Reply-To: <5cda711a-b417-76e9-d113-ea838463f225@suse.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: DB6PR06CA0013.eurprd06.prod.outlook.com (2603:10a6:6:1::26) To DM6PR18MB3401.namprd18.prod.outlook.com (2603:10b6:5:1cc::26) authentication-results: spf=none (sender IP is ) smtp.mailfrom=JBeulich@suse.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [87.234.252.170] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: ac9f597d-d410-421b-bcff-08d709c078ca x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600148)(711020)(4605104)(1401327)(2017052603328)(7193020); SRVR:DM6PR18MB2667; x-ms-traffictypediagnostic: DM6PR18MB2667: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:8882; x-forefront-prvs: 0100732B76 x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(4636009)(39860400002)(346002)(366004)(376002)(396003)(136003)(189003)(199004)(6436002)(102836004)(5640700003)(8676002)(53936002)(6506007)(4326008)(386003)(6512007)(7736002)(186003)(6116002)(6486002)(305945005)(52116002)(2906002)(66066001)(11346002)(81166006)(2616005)(8936002)(316002)(476003)(3846002)(66476007)(446003)(76176011)(2351001)(99286004)(26005)(81156014)(31686004)(68736007)(6916009)(256004)(14444005)(14454004)(486006)(71190400001)(478600001)(71200400001)(31696002)(5660300002)(86362001)(80792005)(2501003)(54906003)(36756003)(25786009)(64756008)(66556008)(66946007)(66446008); DIR:OUT; SFP:1102; SCL:1; SRVR:DM6PR18MB2667; H:DM6PR18MB3401.namprd18.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: suse.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: lOj288oP1WxT1FgfdD5W3Fi7pt+ocLALD2FUj7WhgpTL3b0u6D5yKiNcrxnkA99TUa2tbh5cMwwwqcRVrfAeSXhUNUt7QLiLKxeXL+cm72X2vuoiTqXdeM70obx30WXva9lMU4qojLprnMLrBO6eVVBex/NR3aHFjJ/6LkL4PiLvP9W2VYcj1Atr74MZeiww2omN67yN7+vvAQBm/B9eQWwbfLeQF+Ytxu4rx6y666ycdMRw1n77yj+2863+UtdLOkzvv1qtOngTY7xvqITK1Y0YXozT9Qna2jKH5Z3w4PHPf6lFcNq1EBohMK7Si94T6HyOm8SHIbvT4lkGFpuG8ZyvdwXfVg8YEPHEQroahGl65+T6++Ep/c72rgZFclGKu3YvQedQrMRui80OjRy0sluVPXcG9MXadOXFeMZnYOk= Content-ID: <845375980D633340BB37EF8F1F6F0B21@namprd18.prod.outlook.com> MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: ac9f597d-d410-421b-bcff-08d709c078ca X-MS-Exchange-CrossTenant-originalarrivaltime: 16 Jul 2019 07:37:36.2197 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 856b813c-16e5-49a5-85ec-6f081e13b527 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: JBeulich@suse.com X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR18MB2667 X-OriginatorOrg: suse.com Subject: [Xen-devel] [PATCH v4 02/13] x86/IRQ: deal with move cleanup count state in fixup_irqs() X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , Wei Liu , =?utf-8?q?Roger_Pau_Monn=C3=A9?= Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP The cleanup IPI may get sent immediately before a CPU gets removed from the online map. In such a case the IPI would get handled on the CPU being offlined no earlier than in the interrupts disabled window after fixup_irqs()' main loop. This is too late, however, because a possible affinity change may incur the need for vector assignment, which will fail when the IRQ's move cleanup count is still non-zero. To fix this - record the set of CPUs the cleanup IPIs gets actually sent to alongside setting their count, - adjust the count in fixup_irqs(), accounting for all CPUs that the cleanup IPI was sent to, but that are no longer online, - bail early from the cleanup IPI handler when the CPU is no longer online, to prevent double accounting. Signed-off-by: Jan Beulich Reviewed-by: Roger Pau Monné Acked-by: Andrew Cooper --- a/xen/arch/x86/irq.c +++ b/xen/arch/x86/irq.c @@ -675,6 +675,9 @@ void irq_move_cleanup_interrupt(struct c ack_APIC_irq(); me = smp_processor_id(); + if ( !cpu_online(me) ) + return; + for ( vector = FIRST_DYNAMIC_VECTOR; vector <= LAST_HIPRIORITY_VECTOR; vector++) { @@ -735,11 +738,14 @@ unlock: static void send_cleanup_vector(struct irq_desc *desc) { - cpumask_t cleanup_mask; + cpumask_and(desc->arch.old_cpu_mask, desc->arch.old_cpu_mask, + &cpu_online_map); + desc->arch.move_cleanup_count = cpumask_weight(desc->arch.old_cpu_mask); - cpumask_and(&cleanup_mask, desc->arch.old_cpu_mask, &cpu_online_map); - desc->arch.move_cleanup_count = cpumask_weight(&cleanup_mask); - send_IPI_mask(&cleanup_mask, IRQ_MOVE_CLEANUP_VECTOR); + if ( desc->arch.move_cleanup_count ) + send_IPI_mask(desc->arch.old_cpu_mask, IRQ_MOVE_CLEANUP_VECTOR); + else + release_old_vec(desc); desc->arch.move_in_progress = 0; } @@ -2419,6 +2425,16 @@ void fixup_irqs(const cpumask_t *mask, b vector <= LAST_HIPRIORITY_VECTOR ) cpumask_and(desc->arch.cpu_mask, desc->arch.cpu_mask, mask); + if ( desc->arch.move_cleanup_count ) + { + /* The cleanup IPI may have got sent while we were still online. */ + cpumask_andnot(&affinity, desc->arch.old_cpu_mask, + &cpu_online_map); + desc->arch.move_cleanup_count -= cpumask_weight(&affinity); + if ( !desc->arch.move_cleanup_count ) + release_old_vec(desc); + } + cpumask_copy(&affinity, desc->affinity); if ( !desc->action || cpumask_subset(&affinity, mask) ) {