From patchwork Sun Sep 17 16:45:21 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Martin Blumenstingl X-Patchwork-Id: 9954943 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id D8DB360208 for ; Sun, 17 Sep 2017 16:49:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C4607237F1 for ; Sun, 17 Sep 2017 16:49:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B91A22853D; Sun, 17 Sep 2017 16:49:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, DKIM_VALID, FREEMAIL_FROM, RCVD_IN_DNSWL_MED autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 2377F237F1 for ; Sun, 17 Sep 2017 16:49:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=IkYu8jl06rmIfmu7WcLSFwFYIwgPMyz8knDEeSghLiQ=; b=d8V9yFF0fieDmam6JwX/sKg8ro sKi2dgmr07g9ZTct4XgPwwuJp9FcdEpn3q2ApB8ZeNz7JQFPEP0pYmjMnrd5oBXTyoo4d1XHs+5Jm Ci33+iTXN0eRebt7l6PbjRxL/6D5kZubJUCB+Cw2UuwqoTUQF64iUv5Gd+o0NtXvNoQuVjQfpy9M8 K9hOL5Gewxa3wtuwxyyhK0AlAiYxoiv7m8AV9Y38nk1SGHMk2nO86VIiatkTZi3dYle4eI+KUKWr8 XEOEmDhTfUBOWmeSXSk5MrYTYdn9XAfHJMc8SiELbURbLyz4Ua8qpCeTWUbwkKlAfUnaloujTcHVf Hsrbh6ew==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1dtcl2-0007ot-9P; Sun, 17 Sep 2017 16:49:24 +0000 Received: from mail-wm0-x244.google.com ([2a00:1450:400c:c09::244]) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1dtchy-0005WB-Rk; Sun, 17 Sep 2017 16:46:34 +0000 Received: by mail-wm0-x244.google.com with SMTP id i131so6780086wma.1; Sun, 17 Sep 2017 09:45:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=IOYcEOORRCdyB8ncsoXaq3IXU7vOMgd7r0ONWnQQktU=; b=qk5C6tlxknUHbY0iiYf2cPsFOpo3aeyRkQwYuA2ES1Dm3Q1itTC0A50rFet5p+IQRC nIECzx8rv8sd+7Jnbyj9zr8G+pYOjfrLiM9Astd4cY46IPXNiUIzOJ7jzKc9qAigLz2r RqqJL2Bgsl3H/IRhxpgeNeuFda2/3VGiyfcymPIDRQpdcM7SbKNfwq0PyK1ik1auWo0i gPZTE/4Zzk21TDHEe7CKewZeGqNympCw+r95gx4JX/ZwoWoBDK5tbpn2xfrQRF198xl7 1HVHPlXah3Z/RuMgn42ahErJXcDgnti1uLRO4dJzj4iuUFEwps1tv12dizoEM8cYhfmZ 78vA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=IOYcEOORRCdyB8ncsoXaq3IXU7vOMgd7r0ONWnQQktU=; b=BwdyLJS6dkyYd5WtQtCyd9LsyZO9H/rUMRbXCKk1tawlLc4j0QbpndRfkNS45bnZvH jipp7s/bsmZyMLFBhLypq9WCNs0fdoyTdjQ7E46V9pKkUnMr+LTCVnngIXPyiXrtbXCd VLfOJPFWImB2BYdEMIYW/CWP89vMdCCLkQhaR72OD6lNauUGX51O2TLJEMN31ufMLbid hwtCvHHxHLoDZHicw+SsZi6bjSROME4IbeTV/b78lERFS9QlYUduVA6qXo+3ttWRyHsu GphwNTRexNdNXOVB34po5c+hC0KuO10jYqNXILrRBb5YcV62ZTGSrrLvznZHTP1KfJNz OrLw== X-Gm-Message-State: AHPjjUiM6iQ8ALncwN5A3OmY++jwBwbqjpz4bPXhagrhYtpg3DX4GIqU Pf2XQ5CNZ7PBhwTMG708Y+JAOqrK X-Google-Smtp-Source: AOwi7QBcnIgXhpTZLOPpPOnEeTtosDE/wDy/SjDh4Oplv65WyJwrlfWWhS6ZE97xfZtZTF97C+QcFg== X-Received: by 10.28.99.69 with SMTP id x66mr7443292wmb.30.1505666755107; Sun, 17 Sep 2017 09:45:55 -0700 (PDT) Received: from blackbox.darklights.net (p200300DCD3D04A042D42E2FA95194724.dip0.t-ipconnect.de. [2003:dc:d3d0:4a04:2d42:e2fa:9519:4724]) by smtp.googlemail.com with ESMTPSA id p59sm6323799wrc.75.2017.09.17.09.45.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 17 Sep 2017 09:45:54 -0700 (PDT) From: Martin Blumenstingl To: linux-amlogic@lists.infradead.org, linux-arm-kernel@lists.infradead.org, khilman@baylibre.com, carlo@caione.org, linux@armlinux.org.uk Subject: [PATCH v7 4/6] ARM: meson: Add SMP bringup code for Meson8 and Meson8b Date: Sun, 17 Sep 2017 18:45:21 +0200 Message-Id: <20170917164523.6970-5-martin.blumenstingl@googlemail.com> X-Mailer: git-send-email 2.14.1 In-Reply-To: <20170917164523.6970-1-martin.blumenstingl@googlemail.com> References: <20170917164523.6970-1-martin.blumenstingl@googlemail.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20170917_094616_657146_156C931C X-CRM114-Status: GOOD ( 21.10 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, devicetree@vger.kernel.org, f.fainelli@gmail.com, arnd@arndb.de, Martin Blumenstingl , robh+dt@kernel.org, Carlo Caione MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP This adds the necessary SMP-operations and startup code to use the additional cores on the Amlogic Meson8/Meson8m2 (both are using the same sequence) and Meson8b (using a slightly difference sequence) SoCs. Signed-off-by: Carlo Caione [add Meson8/Meson8m2 support and allow taking CPU cores offline as well] Signed-off-by: Martin Blumenstingl --- arch/arm/Makefile | 1 + arch/arm/mach-meson/Kconfig | 1 + arch/arm/mach-meson/Makefile | 1 + arch/arm/mach-meson/platsmp.c | 440 ++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 443 insertions(+) create mode 100644 arch/arm/mach-meson/platsmp.c diff --git a/arch/arm/Makefile b/arch/arm/Makefile index 47d3a1ab08d2..82faa958ab88 100644 --- a/arch/arm/Makefile +++ b/arch/arm/Makefile @@ -147,6 +147,7 @@ textofs-$(CONFIG_SA1111) := 0x00208000 endif textofs-$(CONFIG_ARCH_MSM8X60) := 0x00208000 textofs-$(CONFIG_ARCH_MSM8960) := 0x00208000 +textofs-$(CONFIG_ARCH_MESON) := 0x00208000 textofs-$(CONFIG_ARCH_AXXIA) := 0x00308000 # Machine directory name. This list is sorted alphanumerically diff --git a/arch/arm/mach-meson/Kconfig b/arch/arm/mach-meson/Kconfig index ee30511849ca..a0c64762d961 100644 --- a/arch/arm/mach-meson/Kconfig +++ b/arch/arm/mach-meson/Kconfig @@ -9,6 +9,7 @@ menuconfig ARCH_MESON select PINCTRL_MESON select COMMON_CLK select COMMON_CLK_AMLOGIC + select HAVE_ARM_SCU if SMP if ARCH_MESON diff --git a/arch/arm/mach-meson/Makefile b/arch/arm/mach-meson/Makefile index 9d7380eeeedd..bc26c85a7e8f 100644 --- a/arch/arm/mach-meson/Makefile +++ b/arch/arm/mach-meson/Makefile @@ -1 +1,2 @@ obj-$(CONFIG_ARCH_MESON) += meson.o +obj-$(CONFIG_SMP) += platsmp.o diff --git a/arch/arm/mach-meson/platsmp.c b/arch/arm/mach-meson/platsmp.c new file mode 100644 index 000000000000..2555f9056a33 --- /dev/null +++ b/arch/arm/mach-meson/platsmp.c @@ -0,0 +1,440 @@ +/* + * Copyright (C) 2015 Carlo Caione + * Copyright (C) 2017 Martin Blumenstingl + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include + +#define MESON_SMP_SRAM_CPU_CTRL_REG (0x00) +#define MESON_SMP_SRAM_CPU_CTRL_ADDR_REG(c) (0x04 + ((c - 1) << 2)) + +#define MESON_CPU_AO_RTI_PWR_A9_CNTL0 (0x00) +#define MESON_CPU_AO_RTI_PWR_A9_CNTL1 (0x04) +#define MESON_CPU_AO_RTI_PWR_A9_MEM_PD0 (0x14) + +#define MESON_CPU_PWR_A9_CNTL0_M(c) (0x03 << ((c * 2) + 16)) +#define MESON_CPU_PWR_A9_CNTL1_M(c) (0x03 << ((c + 1) << 1)) +#define MESON_CPU_PWR_A9_MEM_PD0_M(c) (0x0f << (32 - (c * 4))) +#define MESON_CPU_PWR_A9_CNTL1_ST(c) (0x01 << (c + 16)) + +static void __iomem *sram_base; +static void __iomem *scu_base; +static struct regmap *pmu; + +static struct reset_control *meson_smp_get_core_reset(int cpu) +{ + struct device_node *np = of_get_cpu_node(cpu, 0); + + return of_reset_control_get_exclusive(np, NULL); +} + +static void meson_smp_set_cpu_ctrl(int cpu, bool on_off) +{ + u32 val = readl(sram_base + MESON_SMP_SRAM_CPU_CTRL_REG); + + if (on_off) + val |= BIT(cpu); + else + val &= ~BIT(cpu); + + /* keep bit 0 always enabled */ + val |= BIT(0); + + writel(val, sram_base + MESON_SMP_SRAM_CPU_CTRL_REG); +} + +static void __init meson_smp_prepare_cpus(const char *scu_compatible, + const char *pmu_compatible, + const char *sram_compatible) +{ + static struct device_node *node; + + /* SMP SRAM */ + node = of_find_compatible_node(NULL, NULL, sram_compatible); + if (!node) { + pr_err("Missing SRAM node\n"); + return; + } + + sram_base = of_iomap(node, 0); + if (!sram_base) { + pr_err("Couldn't map SRAM registers\n"); + return; + } + + /* PMU */ + pmu = syscon_regmap_lookup_by_compatible(pmu_compatible); + if (IS_ERR(pmu)) { + pr_err("Couldn't map PMU registers\n"); + return; + } + + /* SCU */ + node = of_find_compatible_node(NULL, NULL, scu_compatible); + if (!node) { + pr_err("Missing SCU node\n"); + return; + } + + scu_base = of_iomap(node, 0); + if (!scu_base) { + pr_err("Couln't map SCU registers\n"); + return; + } + + scu_enable(scu_base); +} + +static void __init meson8b_smp_prepare_cpus(unsigned int max_cpus) +{ + meson_smp_prepare_cpus("arm,cortex-a5-scu", "amlogic,meson8b-pmu", + "amlogic,meson8b-smp-sram"); +} + +static void __init meson8_smp_prepare_cpus(unsigned int max_cpus) +{ + meson_smp_prepare_cpus("arm,cortex-a9-scu", "amlogic,meson8-pmu", + "amlogic,meson8-smp-sram"); +} + +static void meson_smp_begin_secondary_boot(unsigned int cpu) +{ + /* + * Set the entry point before powering on the CPU through the SCU. This + * is needed if the CPU is in "warm" state (= after rebooting the + * system without power-cycling, or when taking the CPU offline and + * then taking it online again. + */ + writel(__pa_symbol(secondary_startup), + sram_base + MESON_SMP_SRAM_CPU_CTRL_ADDR_REG(cpu)); + + /* + * SCU Power on CPU (needs to be done before starting the CPU, + * otherwise the secondary CPU will not start). + */ + scu_cpu_power_enable(scu_base, cpu); +} + +static int meson_smp_finalize_secondary_boot(unsigned int cpu) +{ + unsigned long timeout; + + timeout = jiffies + (10 * HZ); + while (readl(sram_base + MESON_SMP_SRAM_CPU_CTRL_ADDR_REG(cpu))) { + if (!time_before(jiffies, timeout)) { + pr_err("Timeout while waiting for CPU%d status\n", + cpu); + return -ETIMEDOUT; + } + } + + writel(__pa_symbol(secondary_startup), + sram_base + MESON_SMP_SRAM_CPU_CTRL_ADDR_REG(cpu)); + + meson_smp_set_cpu_ctrl(cpu, true); + + return 0; +} + +static int meson8_smp_boot_secondary(unsigned int cpu, + struct task_struct *idle) +{ + struct reset_control *rstc; + int ret; + + rstc = meson_smp_get_core_reset(cpu); + if (IS_ERR(rstc)) { + pr_err("Couldn't get the reset controller for CPU%d\n", cpu); + return PTR_ERR(rstc); + } + + meson_smp_begin_secondary_boot(cpu); + + /* Reset enable */ + ret = reset_control_assert(rstc); + if (ret) { + pr_err("Failed to assert CPU%d reset\n", cpu); + goto out; + } + + /* CPU power ON */ + ret = regmap_update_bits(pmu, MESON_CPU_AO_RTI_PWR_A9_CNTL1, + MESON_CPU_PWR_A9_CNTL1_M(cpu), 0); + if (ret < 0) { + pr_err("Couldn't wake up CPU%d\n", cpu); + goto out; + } + + udelay(10); + + /* Isolation disable */ + ret = regmap_update_bits(pmu, MESON_CPU_AO_RTI_PWR_A9_CNTL0, BIT(cpu), + 0); + if (ret < 0) { + pr_err("Error when disabling isolation of CPU%d\n", cpu); + goto out; + } + + /* Reset disable */ + ret = reset_control_deassert(rstc); + if (ret) { + pr_err("Failed to de-assert CPU%d reset\n", cpu); + goto out; + } + + ret = meson_smp_finalize_secondary_boot(cpu); + if (ret) + goto out; + +out: + reset_control_put(rstc); + + return 0; +} + +static int meson8b_smp_boot_secondary(unsigned int cpu, + struct task_struct *idle) +{ + struct reset_control *rstc; + int ret; + u32 val; + + rstc = meson_smp_get_core_reset(cpu); + if (IS_ERR(rstc)) { + pr_err("Couldn't get the reset controller for CPU%d\n", cpu); + return PTR_ERR(rstc); + } + + meson_smp_begin_secondary_boot(cpu); + + /* CPU power UP */ + ret = regmap_update_bits(pmu, MESON_CPU_AO_RTI_PWR_A9_CNTL0, + MESON_CPU_PWR_A9_CNTL0_M(cpu), 0); + if (ret < 0) { + pr_err("Couldn't power up CPU%d\n", cpu); + goto out; + } + + udelay(5); + + /* Reset enable */ + ret = reset_control_assert(rstc); + if (ret) { + pr_err("Failed to assert CPU%d reset\n", cpu); + goto out; + } + + /* Memory power UP */ + ret = regmap_update_bits(pmu, MESON_CPU_AO_RTI_PWR_A9_MEM_PD0, + MESON_CPU_PWR_A9_MEM_PD0_M(cpu), 0); + if (ret < 0) { + pr_err("Couldn't power up the memory for CPU%d\n", cpu); + goto out; + } + + /* Wake up CPU */ + ret = regmap_update_bits(pmu, MESON_CPU_AO_RTI_PWR_A9_CNTL1, + MESON_CPU_PWR_A9_CNTL1_M(cpu), 0); + if (ret < 0) { + pr_err("Couldn't wake up CPU%d\n", cpu); + goto out; + } + + udelay(10); + + ret = regmap_read_poll_timeout(pmu, MESON_CPU_AO_RTI_PWR_A9_CNTL1, val, + val & MESON_CPU_PWR_A9_CNTL1_ST(cpu), + 10, 10000); + if (ret) { + pr_err("Timeout while polling PMU for CPU%d status\n", cpu); + goto out; + } + + /* Isolation disable */ + ret = regmap_update_bits(pmu, MESON_CPU_AO_RTI_PWR_A9_CNTL0, BIT(cpu), + 0); + if (ret < 0) { + pr_err("Error when disabling isolation of CPU%d\n", cpu); + goto out; + } + + /* Reset disable */ + ret = reset_control_deassert(rstc); + if (ret) { + pr_err("Failed to de-assert CPU%d reset\n", cpu); + goto out; + } + + ret = meson_smp_finalize_secondary_boot(cpu); + if (ret) + goto out; + +out: + reset_control_put(rstc); + + return 0; +} + +#ifdef CONFIG_HOTPLUG_CPU +static void meson8_smp_cpu_die(unsigned int cpu) +{ + meson_smp_set_cpu_ctrl(cpu, false); + + v7_exit_coherency_flush(louis); + + scu_power_mode(scu_base, SCU_PM_POWEROFF); + + dsb(); + wfi(); + + /* we should never get here */ + WARN_ON(1); +} + +static int meson8_smp_cpu_kill(unsigned int cpu) +{ + int ret, power_mode; + unsigned long timeout; + + timeout = jiffies + (50 * HZ); + do { + power_mode = scu_get_cpu_power_mode(scu_base, cpu); + + if (power_mode == SCU_PM_POWEROFF) + break; + + usleep_range(10000, 15000); + } while (time_before(jiffies, timeout)); + + if (power_mode != SCU_PM_POWEROFF) { + pr_err("Error while waiting for SCU power-off on CPU%d\n", + cpu); + return -ETIMEDOUT; + } + + msleep(30); + + /* Isolation enable */ + ret = regmap_update_bits(pmu, MESON_CPU_AO_RTI_PWR_A9_CNTL0, BIT(cpu), + 0x3); + if (ret < 0) { + pr_err("Error when enabling isolation for CPU%d\n", cpu); + return ret; + } + + udelay(10); + + /* CPU power OFF */ + ret = regmap_update_bits(pmu, MESON_CPU_AO_RTI_PWR_A9_CNTL1, + MESON_CPU_PWR_A9_CNTL1_M(cpu), 0x3); + if (ret < 0) { + pr_err("Couldn't change sleep status of CPU%d\n", cpu); + return ret; + } + + return 1; +} + +static int meson8b_smp_cpu_kill(unsigned int cpu) +{ + int ret, power_mode, count = 5000; + + do { + power_mode = scu_get_cpu_power_mode(scu_base, cpu); + + if (power_mode == SCU_PM_POWEROFF) + break; + + udelay(10); + } while (++count); + + if (power_mode != SCU_PM_POWEROFF) { + pr_err("Error while waiting for SCU power-off on CPU%d\n", + cpu); + return -ETIMEDOUT; + } + + udelay(10); + + /* CPU power DOWN */ + ret = regmap_update_bits(pmu, MESON_CPU_AO_RTI_PWR_A9_CNTL0, + MESON_CPU_PWR_A9_CNTL0_M(cpu), 0x3); + if (ret < 0) { + pr_err("Couldn't power down CPU%d\n", cpu); + return ret; + } + + /* Isolation enable */ + ret = regmap_update_bits(pmu, MESON_CPU_AO_RTI_PWR_A9_CNTL0, BIT(cpu), + 0x3); + if (ret < 0) { + pr_err("Error when enabling isolation for CPU%d\n", cpu); + return ret; + } + + udelay(10); + + /* Sleep status */ + ret = regmap_update_bits(pmu, MESON_CPU_AO_RTI_PWR_A9_CNTL1, + MESON_CPU_PWR_A9_CNTL1_M(cpu), 0x3); + if (ret < 0) { + pr_err("Couldn't change sleep status of CPU%d\n", cpu); + return ret; + } + + /* Memory power DOWN */ + ret = regmap_update_bits(pmu, MESON_CPU_AO_RTI_PWR_A9_MEM_PD0, + MESON_CPU_PWR_A9_MEM_PD0_M(cpu), 0xf); + if (ret < 0) { + pr_err("Couldn't power down the memory of CPU%d\n", cpu); + return ret; + } + + return 1; +} +#endif + +static struct smp_operations meson8_smp_ops __initdata = { + .smp_prepare_cpus = meson8_smp_prepare_cpus, + .smp_boot_secondary = meson8_smp_boot_secondary, +#ifdef CONFIG_HOTPLUG_CPU + .cpu_die = meson8_smp_cpu_die, + .cpu_kill = meson8_smp_cpu_kill, +#endif +}; + +static struct smp_operations meson8b_smp_ops __initdata = { + .smp_prepare_cpus = meson8b_smp_prepare_cpus, + .smp_boot_secondary = meson8b_smp_boot_secondary, +#ifdef CONFIG_HOTPLUG_CPU + .cpu_die = meson8_smp_cpu_die, + .cpu_kill = meson8b_smp_cpu_kill, +#endif +}; + +CPU_METHOD_OF_DECLARE(meson8_smp, "amlogic,meson8-smp", &meson8_smp_ops); +CPU_METHOD_OF_DECLARE(meson8b_smp, "amlogic,meson8b-smp", &meson8b_smp_ops);