diff mbox

[v2,2/3] mailbox: Add support for Hi3660 mailbox

Message ID 1509084904-2505-3-git-send-email-zhongkaihua@huawei.com (mailing list archive)
State New, archived
Headers show

Commit Message

Zhong Kaihua Oct. 27, 2017, 6:15 a.m. UTC
Hi3660 mailbox controller is used to send message within multiple
processors, MCU, HIFI, etc.  It supports 32 mailbox channels and every
channel can only be used for single transferring direction.  Once the
channel is enabled, it needs to specify the destination interrupt and
acknowledge interrupt, these two interrupt vectors are used to create
the connection between the mailbox and interrupt controllers.

The application processor (or from point of view of kernel) is not the
only one master which can launch the data transferring, other
processors or MCU/DSP also can kick off the data transferring.  So this
driver implements a locking mechanism to support exclusive accessing.

The data transferring supports two modes, one is named as "automatic
acknowledge" mode so after send message the kernel doesn't need to wait
for acknowledge from remote and directly return; there have another mode
is to rely on handling interrupt for acknowledge.

This commit is for initial version driver, which only supports
"automatic acknowledge" mode to support CPU clock, which is the only
one consumer to use mailbox and has been verified.  Later may enhance
this driver for interrupt mode (e.g. for supporting HIFI).

Cc: John Stultz <john.stultz@linaro.org>
Cc: Guodong Xu <guodong.xu@linaro.org>
Cc: Haojian Zhuang <haojian.zhuang@linaro.org>
Cc: Niranjan Yadla <nyadla@cadence.com>
Cc: Raj Pawate <pawateb@cadence.com>
Signed-off-by: Leo Yan <leo.yan@linaro.org>
Signed-off-by: Ruyi Wang <wangruyi@huawei.com>
Signed-off-by: Kaihua Zhong <zhongkaihua@huawei.com>
Signed-off-by: Kevin Wang <kevin.wangtao@hisilicon.com>
---
 drivers/mailbox/Kconfig          |   8 +
 drivers/mailbox/Makefile         |   2 +
 drivers/mailbox/hi3660-mailbox.c | 331 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 341 insertions(+)
 create mode 100644 drivers/mailbox/hi3660-mailbox.c

Comments

Julien Thierry Oct. 27, 2017, 10:40 a.m. UTC | #1
Hi Kaihua,

On 27/10/17 07:15, Kaihua Zhong wrote:
> Hi3660 mailbox controller is used to send message within multiple
> processors, MCU, HIFI, etc.  It supports 32 mailbox channels and every
> channel can only be used for single transferring direction.  Once the
> channel is enabled, it needs to specify the destination interrupt and
> acknowledge interrupt, these two interrupt vectors are used to create
> the connection between the mailbox and interrupt controllers.
> 
> The application processor (or from point of view of kernel) is not the
> only one master which can launch the data transferring, other
> processors or MCU/DSP also can kick off the data transferring.  So this
> driver implements a locking mechanism to support exclusive accessing.
> 
> The data transferring supports two modes, one is named as "automatic
> acknowledge" mode so after send message the kernel doesn't need to wait
> for acknowledge from remote and directly return; there have another mode
> is to rely on handling interrupt for acknowledge.
> 
> This commit is for initial version driver, which only supports
> "automatic acknowledge" mode to support CPU clock, which is the only
> one consumer to use mailbox and has been verified.  Later may enhance
> this driver for interrupt mode (e.g. for supporting HIFI).
> 
> Cc: John Stultz <john.stultz@linaro.org>
> Cc: Guodong Xu <guodong.xu@linaro.org>
> Cc: Haojian Zhuang <haojian.zhuang@linaro.org>
> Cc: Niranjan Yadla <nyadla@cadence.com>
> Cc: Raj Pawate <pawateb@cadence.com>
> Signed-off-by: Leo Yan <leo.yan@linaro.org>
> Signed-off-by: Ruyi Wang <wangruyi@huawei.com>
> Signed-off-by: Kaihua Zhong <zhongkaihua@huawei.com>
> Signed-off-by: Kevin Wang <kevin.wangtao@hisilicon.com>
> ---
>   drivers/mailbox/Kconfig          |   8 +
>   drivers/mailbox/Makefile         |   2 +
>   drivers/mailbox/hi3660-mailbox.c | 331 +++++++++++++++++++++++++++++++++++++++
>   3 files changed, 341 insertions(+)
>   create mode 100644 drivers/mailbox/hi3660-mailbox.c
> 
> diff --git a/drivers/mailbox/Kconfig b/drivers/mailbox/Kconfig
> index c5731e5..4b5d6e9 100644
> --- a/drivers/mailbox/Kconfig
> +++ b/drivers/mailbox/Kconfig
> @@ -108,6 +108,14 @@ config TI_MESSAGE_MANAGER
>   	  multiple processors within the SoC. Select this driver if your
>   	  platform has support for the hardware block.
>   
> +config HI3660_MBOX
> +	tristate "Hi3660 Mailbox"
> +	depends on ARCH_HISI && OF
> +	help
> +	  An implementation of the hi3660 mailbox. It is used to send message
> +	  between application processors and other processors/MCU/DSP. Select
> +	  Y here if you want to use Hi3660 mailbox controller.
> +
>   config HI6220_MBOX
>   	tristate "Hi6220 Mailbox"
>   	depends on ARCH_HISI
> diff --git a/drivers/mailbox/Makefile b/drivers/mailbox/Makefile
> index d54e412..7d1bd51 100644
> --- a/drivers/mailbox/Makefile
> +++ b/drivers/mailbox/Makefile
> @@ -26,6 +26,8 @@ obj-$(CONFIG_TI_MESSAGE_MANAGER) += ti-msgmgr.o
>   
>   obj-$(CONFIG_XGENE_SLIMPRO_MBOX) += mailbox-xgene-slimpro.o
>   
> +obj-$(CONFIG_HI3660_MBOX)	+= hi3660-mailbox.o
> +
>   obj-$(CONFIG_HI6220_MBOX)	+= hi6220-mailbox.o
>   
>   obj-$(CONFIG_BCM_PDC_MBOX)	+= bcm-pdc-mailbox.o
> diff --git a/drivers/mailbox/hi3660-mailbox.c b/drivers/mailbox/hi3660-mailbox.c
> new file mode 100644
> index 0000000..67df8f8
> --- /dev/null
> +++ b/drivers/mailbox/hi3660-mailbox.c
> @@ -0,0 +1,331 @@
> +/*
> + * Hisilicon's Hi3660 mailbox controller driver
> + *
> + * Copyright (c) 2017 Hisilicon Limited.
> + * Copyright (c) 2017 Linaro Limited.
> + *
> + * Author: Leo Yan <leo.yan@linaro.org>
> + *
> + * This program is free software: you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation, version 2 of the License.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + */
> +
> +#include <linux/bitops.h>
> +#include <linux/delay.h>
> +#include <linux/device.h>
> +#include <linux/err.h>
> +#include <linux/interrupt.h>
> +#include <linux/io.h>
> +#include <linux/iopoll.h>
> +#include <linux/mailbox_controller.h>
> +#include <linux/module.h>
> +#include <linux/platform_device.h>
> +#include <linux/slab.h>
> +
> +#include "mailbox.h"
> +
> +#define MBOX_CHAN_MAX			32
> +
> +#define MBOX_RX				(0x0)
> +#define MBOX_TX				(0x1)
> +
> +#define MBOX_BASE(mbox, ch)		((mbox)->base + ((ch) * 0x40))
> +#define MBOX_SRC_REG			(0x00)
> +#define MBOX_DST_REG			(0x04)
> +#define MBOX_DCLR_REG			(0x08)
> +#define MBOX_DSTAT_REG			(0x0c)
> +#define MBOX_MODE_REG			(0x10)
> +#define MBOX_IMASK_REG			(0x14)
> +#define MBOX_ICLR_REG			(0x18)
> +#define MBOX_SEND_REG			(0x1c)
> +#define MBOX_DATA_REG			(0x20)
> +
> +#define MBOX_IPC_LOCK_REG		(0xa00)
> +#define MBOX_IPC_UNLOCK			(0x1acce551)
> +
> +#define MBOX_AUTOMATIC_ACK		(1)
> +
> +#define MBOX_STATE_IDLE			BIT(4)
> +#define MBOX_STATE_ACK			BIT(7)
> +
> +#define MBOX_MSG_LEN			8
> +
> +/**
> + * Hi3660 mailbox channel device data
> + *
> + * A channel can be used for TX or RX, it can trigger remote
> + * processor interrupt to notify remote processor and can receive
> + * interrupt if has incoming message.
> + *
> + * @dst_irq:	Interrupt vector for remote processor
> + * @ack_irq:	Interrupt vector for local processor
> + */
> +struct hi3660_mbox_dev {
> +	unsigned int dst_irq;
> +	unsigned int ack_irq;
> +};
> +
> +/**
> + * Hi3660 mailbox controller data
> + *
> + * Mailbox controller includes 32 channels and can allocate
> + * channel for message transferring.
> + *
> + * @dev:	Device to which it is attached
> + * @base:	Base address of the register mapping region
> + * @chan:	Representation of channels in mailbox controller
> + * @mdev:	Representation of channel device data
> + * @controller:	Representation of a communication channel controller
> + */
> +struct hi3660_mbox {
> +	struct device *dev;
> +	void __iomem *base;
> +	struct mbox_chan chan[MBOX_CHAN_MAX];
> +	struct hi3660_mbox_dev mdev[MBOX_CHAN_MAX];
> +	struct mbox_controller controller;
> +};
> +
> +static inline struct hi3660_mbox *to_hi3660_mbox(struct mbox_chan *chan)
> +{
> +	return container_of(chan->mbox, struct hi3660_mbox, controller);
> +}
> +
> +static int hi3660_mbox_check_state(struct mbox_chan *chan)
> +{
> +	unsigned long ch = (unsigned long)chan->con_priv;
> +	struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
> +	struct hi3660_mbox_dev *mdev = &mbox->mdev[ch];
> +	void __iomem *base = MBOX_BASE(mbox, ch);
> +	unsigned long val;
> +	unsigned int state, ret;
> +
> +	/* Mailbox is idle so directly bail out */
> +	state = readl_relaxed(base + MBOX_MODE_REG);
> +	if (state & MBOX_STATE_IDLE)
> +		return 0;
> +
> +	/* Wait for acknowledge from remote */
> +	ret = readx_poll_timeout_atomic(readl_relaxed, base + MBOX_MODE_REG,
> +			val, (val & MBOX_STATE_ACK), 1000, 300000);
> +	if (ret) {
> +		dev_err(mbox->dev, "%s: timeout for receiving ack\n", __func__);
> +		return ret;
> +	}
> +
> +	/* Ensure channel is released */
> +	writel_relaxed(0xffffffff, base + MBOX_IMASK_REG);
> +	writel_relaxed(BIT(mdev->ack_irq), base + MBOX_SRC_REG);
> +	__asm__ volatile ("sev");

There is an existing sev() macro for this.

> +	return 0;
> +}
> +
> +static int hi3660_mbox_unlock(struct mbox_chan *chan)
> +{
> +	struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
> +	unsigned int val, retry = 3;
> +
> +	do {
> +		writel_relaxed(MBOX_IPC_UNLOCK, mbox->base + MBOX_IPC_LOCK_REG);
> +
> +		val = readl_relaxed(mbox->base + MBOX_IPC_LOCK_REG);
> +		if (!val)
> +			break;
> +
> +		udelay(10);
> +	} while (retry--);
> +
> +	return (!val) ? 0 : -ETIMEDOUT;
> +}
> +
> +static int hi3660_mbox_acquire_channel(struct mbox_chan *chan)
> +{
> +	unsigned long ch = (unsigned long)chan->con_priv;
> +	struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
> +	struct hi3660_mbox_dev *mdev = &mbox->mdev[ch];
> +	void __iomem *base = MBOX_BASE(mbox, ch);
> +	unsigned int val = 0, retry = 10;
> +
> +	/*
> +	 * Hardware locking for exclusive accessing within CPUs
> +	 * without exclusive monitor mechanism.
> +	 */
> +	do {
> +		val = readl_relaxed(base + MBOX_MODE_REG);
> +		if (!(val & MBOX_STATE_IDLE)) {
> +			__asm__ volatile ("wfe");
> +			continue;

So this is going to "wfe" when retry == 0 and the continue statement 
will take us out of the loop?

Also, there is a wfe() macro for arm and arm64 which could be used here.

> +		}
> +
> +		/* Check if channel has be acquired */
> +		writel_relaxed(BIT(mdev->ack_irq), base + MBOX_SRC_REG);
> +		val = readl_relaxed(base + MBOX_SRC_REG) & BIT(mdev->ack_irq);
> +		if (val)
> +			break;
> +
> +	} while (retry--);
> +
> +	return (val) ? 0 : -ETIMEDOUT;

If timeout occurs while waiting for the MBOX to be idle, val will hold 
the last value from MBOX_MODE_REG, and if this can be different than 0, 
this will hide the fact it timed out.

Maybe the following would be more appropriate:

return (retry >= 0) ? 0 : -ETIMEDOUT;

Also, your do {} while loops for the timeouts falsely give the 
impression we do "retry" attempts when we actually do "retry + 1" 
attempts (but I guess it doesn't make a big difference from the 
functional point of view).

> +}
> +
> +static int hi3660_mbox_send(struct mbox_chan *chan, u32 *msg)
> +{
> +	unsigned long ch = (unsigned long)chan->con_priv;
> +	struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
> +	struct hi3660_mbox_dev *mdev = &mbox->mdev[ch];
> +	void __iomem *base = MBOX_BASE(mbox, ch);
> +	unsigned int i;
> +
> +	/* Clear mask for destination interrupt */
> +	writel_relaxed(~BIT(mdev->dst_irq), base + MBOX_IMASK_REG);
> +
> +	/* Config destination for interrupt vector */
> +	writel_relaxed(BIT(mdev->dst_irq), base + MBOX_DST_REG);
> +
> +	/* Automatic acknowledge mode */
> +	writel_relaxed(MBOX_AUTOMATIC_ACK, base + MBOX_MODE_REG);
> +
> +	/* Fill message data */
> +	for (i = 0; i < MBOX_MSG_LEN; i++)
> +		writel_relaxed(msg[i], base + MBOX_DATA_REG + i * 4);
> +
> +	/* Trigger data transferring */
> +	writel_relaxed(BIT(mdev->ack_irq), base + MBOX_SEND_REG);
> +	return 0;
> +}
> +
> +static int hi3660_mbox_send_data(struct mbox_chan *chan, void *msg)
> +{
> +	struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
> +	int err;
> +
> +	err = hi3660_mbox_check_state(chan);
> +	if (err) {
> +		dev_err(mbox->dev, "checking state failed\n");
> +		return err;
> +	}
> +
> +	err = hi3660_mbox_unlock(chan);
> +	if (err) {
> +		dev_err(mbox->dev, "unlocking mailbox failed\n");
> +		return err;
> +	}
> +
> +	err = hi3660_mbox_acquire_channel(chan);
> +	if (err) {
> +		dev_err(mbox->dev, "acquiring channel failed\n");
> +		return err;
> +	}
> +
> +	return hi3660_mbox_send(chan, msg);
> +}
> +
> +static struct mbox_chan_ops hi3660_mbox_ops = {
> +	.send_data    = hi3660_mbox_send_data,
> +};
> +
> +static struct mbox_chan *hi3660_mbox_xlate(struct mbox_controller *controller,
> +					   const struct of_phandle_args *spec)
> +{
> +	struct hi3660_mbox *mbox = dev_get_drvdata(controller->dev);

nit:
In to_hi3660_mbox, you use the fact that the mbox_controller structure 
is part of hi3660_mbox and use container of to retrieve hi3660_mbox.

I think it would be nice to be consistent about this, either use 
container_of or dev drvdata, but not both.

Cheers,
Mark Rutland Oct. 27, 2017, 10:46 a.m. UTC | #2
On Fri, Oct 27, 2017 at 02:15:03PM +0800, Kaihua Zhong wrote:
> Hi3660 mailbox controller is used to send message within multiple
> processors, MCU, HIFI, etc.  It supports 32 mailbox channels and every
> channel can only be used for single transferring direction.  Once the
> channel is enabled, it needs to specify the destination interrupt and
> acknowledge interrupt, these two interrupt vectors are used to create
> the connection between the mailbox and interrupt controllers.
> 
> The application processor (or from point of view of kernel) is not the
> only one master which can launch the data transferring, other
> processors or MCU/DSP also can kick off the data transferring.  So this
> driver implements a locking mechanism to support exclusive accessing.

... and that locking mechanism is what precisely?

Where is the protocol defined?

> +static int hi3660_mbox_check_state(struct mbox_chan *chan)
> +{
> +	unsigned long ch = (unsigned long)chan->con_priv;
> +	struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
> +	struct hi3660_mbox_dev *mdev = &mbox->mdev[ch];
> +	void __iomem *base = MBOX_BASE(mbox, ch);
> +	unsigned long val;
> +	unsigned int state, ret;
> +
> +	/* Mailbox is idle so directly bail out */
> +	state = readl_relaxed(base + MBOX_MODE_REG);
> +	if (state & MBOX_STATE_IDLE)
> +		return 0;
> +
> +	/* Wait for acknowledge from remote */
> +	ret = readx_poll_timeout_atomic(readl_relaxed, base + MBOX_MODE_REG,
> +			val, (val & MBOX_STATE_ACK), 1000, 300000);
> +	if (ret) {
> +		dev_err(mbox->dev, "%s: timeout for receiving ack\n", __func__);
> +		return ret;
> +	}
> +
> +	/* Ensure channel is released */
> +	writel_relaxed(0xffffffff, base + MBOX_IMASK_REG);
> +	writel_relaxed(BIT(mdev->ack_irq), base + MBOX_SRC_REG);
> +	__asm__ volatile ("sev");
> +	return 0;
> +}

Drivers really shouldn't be using SEV directly (even if via the sev() macro)...

This SEV isn't ordered w.r.t. anything, and it's unclear what ordering you
need, so this simply does not work.

> +static int hi3660_mbox_acquire_channel(struct mbox_chan *chan)
> +{
> +	unsigned long ch = (unsigned long)chan->con_priv;
> +	struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
> +	struct hi3660_mbox_dev *mdev = &mbox->mdev[ch];
> +	void __iomem *base = MBOX_BASE(mbox, ch);
> +	unsigned int val = 0, retry = 10;
> +
> +	/*
> +	 * Hardware locking for exclusive accessing within CPUs
> +	 * without exclusive monitor mechanism.
> +	 */
> +	do {
> +		val = readl_relaxed(base + MBOX_MODE_REG);
> +		if (!(val & MBOX_STATE_IDLE)) {
> +			__asm__ volatile ("wfe");

... and likewise for WFE / wfe().

> +			continue;
> +		}
> +
> +		/* Check if channel has be acquired */
> +		writel_relaxed(BIT(mdev->ack_irq), base + MBOX_SRC_REG);
> +		val = readl_relaxed(base + MBOX_SRC_REG) & BIT(mdev->ack_irq);
> +		if (val)
> +			break;
> +
> +	} while (retry--);
> +
> +	return (val) ? 0 : -ETIMEDOUT;
> +}

Please elaborate on how this synchronisation mechanism is expected to work, and
why it is necessary in the first place.

Thanks,
Mark.
Leo Yan Oct. 30, 2017, 4:45 a.m. UTC | #3
Hi Mark,

On Fri, Oct 27, 2017 at 11:46:00AM +0100, Mark Rutland wrote:
> On Fri, Oct 27, 2017 at 02:15:03PM +0800, Kaihua Zhong wrote:
> > Hi3660 mailbox controller is used to send message within multiple
> > processors, MCU, HIFI, etc.  It supports 32 mailbox channels and every
> > channel can only be used for single transferring direction.  Once the
> > channel is enabled, it needs to specify the destination interrupt and
> > acknowledge interrupt, these two interrupt vectors are used to create
> > the connection between the mailbox and interrupt controllers.
> > 
> > The application processor (or from point of view of kernel) is not the
> > only one master which can launch the data transferring, other
> > processors or MCU/DSP also can kick off the data transferring.  So this
> > driver implements a locking mechanism to support exclusive accessing.
> 
> ... and that locking mechanism is what precisely?
> 
> Where is the protocol defined?
> 
> > +static int hi3660_mbox_check_state(struct mbox_chan *chan)
> > +{
> > +	unsigned long ch = (unsigned long)chan->con_priv;
> > +	struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
> > +	struct hi3660_mbox_dev *mdev = &mbox->mdev[ch];
> > +	void __iomem *base = MBOX_BASE(mbox, ch);
> > +	unsigned long val;
> > +	unsigned int state, ret;
> > +
> > +	/* Mailbox is idle so directly bail out */
> > +	state = readl_relaxed(base + MBOX_MODE_REG);
> > +	if (state & MBOX_STATE_IDLE)
> > +		return 0;
> > +
> > +	/* Wait for acknowledge from remote */
> > +	ret = readx_poll_timeout_atomic(readl_relaxed, base + MBOX_MODE_REG,
> > +			val, (val & MBOX_STATE_ACK), 1000, 300000);
> > +	if (ret) {
> > +		dev_err(mbox->dev, "%s: timeout for receiving ack\n", __func__);
> > +		return ret;
> > +	}
> > +
> > +	/* Ensure channel is released */
> > +	writel_relaxed(0xffffffff, base + MBOX_IMASK_REG);
> > +	writel_relaxed(BIT(mdev->ack_irq), base + MBOX_SRC_REG);
> > +	__asm__ volatile ("sev");
> > +	return 0;
> > +}
> 
> Drivers really shouldn't be using SEV directly (even if via the sev() macro)...
> 
> This SEV isn't ordered w.r.t. anything, and it's unclear what ordering you
> need, so this simply does not work.

I will leave your questions for Hisilicon colleagues, essentially
your questions are related with mailbox mechanism.

But I'd like to firstly get clear your question for "This SEV isn't
ordered w.r.t. anything". From my understanding, ARMv8 architecture
natually adds DMB before SEV so all previous register writing
opreations should be ensured to endpoint before SEV?

[...]

Thanks,
Leo Yan
Mark Rutland Oct. 30, 2017, 10:19 a.m. UTC | #4
Hi,

On Mon, Oct 30, 2017 at 12:45:06PM +0800, Leo Yan wrote:
> On Fri, Oct 27, 2017 at 11:46:00AM +0100, Mark Rutland wrote:
> > On Fri, Oct 27, 2017 at 02:15:03PM +0800, Kaihua Zhong wrote:
> > > +static int hi3660_mbox_check_state(struct mbox_chan *chan)
> > > +{

> > > +	/* Ensure channel is released */
> > > +	writel_relaxed(0xffffffff, base + MBOX_IMASK_REG);
> > > +	writel_relaxed(BIT(mdev->ack_irq), base + MBOX_SRC_REG);
> > > +	__asm__ volatile ("sev");
> > > +	return 0;
> > > +}
> > 
> > Drivers really shouldn't be using SEV directly (even if via the
> > sev() macro)...
> > 
> > This SEV isn't ordered w.r.t. anything, and it's unclear what
> > ordering you need, so this simply does not work.
> 
> I will leave your questions for Hisilicon colleagues, essentially your
> questions are related with mailbox mechanism.
> 
> But I'd like to firstly get clear your question for "This SEV isn't
> ordered w.r.t. anything". From my understanding, ARMv8 architecture
> natually adds DMB before SEV so all previous register writing
> opreations should be ensured to endpoint before SEV?

This is not the case; SEV does not add any implicit memory barrier, and
is not ordered w.r.t. memory accesses.

See ARM DDI 0487B.b, page D1-1905, "The Send Event instructions":

    The PE is not required to guarantee the ordering of this event with
    respect to the completion of memory accesses by instructions before
    the SEV instruction. Therefore, ARM recommends that software
    includes a DSB instruction before any SEV instruction.

Note that a DMB is not sufficient, as SEV is not a memory access.

Thanks,
Mark.
Leo Yan Oct. 30, 2017, 11:13 a.m. UTC | #5
Hi Mark,

On Mon, Oct 30, 2017 at 10:19:40AM +0000, Mark Rutland wrote:
> Hi,
> 
> On Mon, Oct 30, 2017 at 12:45:06PM +0800, Leo Yan wrote:
> > On Fri, Oct 27, 2017 at 11:46:00AM +0100, Mark Rutland wrote:
> > > On Fri, Oct 27, 2017 at 02:15:03PM +0800, Kaihua Zhong wrote:
> > > > +static int hi3660_mbox_check_state(struct mbox_chan *chan)
> > > > +{
> 
> > > > +	/* Ensure channel is released */
> > > > +	writel_relaxed(0xffffffff, base + MBOX_IMASK_REG);
> > > > +	writel_relaxed(BIT(mdev->ack_irq), base + MBOX_SRC_REG);
> > > > +	__asm__ volatile ("sev");
> > > > +	return 0;
> > > > +}
> > > 
> > > Drivers really shouldn't be using SEV directly (even if via the
> > > sev() macro)...
> > > 
> > > This SEV isn't ordered w.r.t. anything, and it's unclear what
> > > ordering you need, so this simply does not work.
> > 
> > I will leave your questions for Hisilicon colleagues, essentially your
> > questions are related with mailbox mechanism.
> > 
> > But I'd like to firstly get clear your question for "This SEV isn't
> > ordered w.r.t. anything". From my understanding, ARMv8 architecture
> > natually adds DMB before SEV so all previous register writing
> > opreations should be ensured to endpoint before SEV?
> 
> This is not the case; SEV does not add any implicit memory barrier, and
> is not ordered w.r.t. memory accesses.
> 
> See ARM DDI 0487B.b, page D1-1905, "The Send Event instructions":
> 
>     The PE is not required to guarantee the ordering of this event with
>     respect to the completion of memory accesses by instructions before
>     the SEV instruction. Therefore, ARM recommends that software
>     includes a DSB instruction before any SEV instruction.

My fault and thanks for explanation.

> Note that a DMB is not sufficient, as SEV is not a memory access.

Understood now, so below code should be safe?

wmb();  -> dsb(st);
sev();

Thanks,
Leo Yan
Mark Rutland Oct. 30, 2017, 11:37 a.m. UTC | #6
On Mon, Oct 30, 2017 at 07:13:13PM +0800, Leo Yan wrote:
> Hi Mark,
> 
> On Mon, Oct 30, 2017 at 10:19:40AM +0000, Mark Rutland wrote:
> > Hi,
> > 
> > On Mon, Oct 30, 2017 at 12:45:06PM +0800, Leo Yan wrote:
> > > On Fri, Oct 27, 2017 at 11:46:00AM +0100, Mark Rutland wrote:
> > > > On Fri, Oct 27, 2017 at 02:15:03PM +0800, Kaihua Zhong wrote:
> > > > > +static int hi3660_mbox_check_state(struct mbox_chan *chan)
> > > > > +{
> > 
> > > > > +	/* Ensure channel is released */
> > > > > +	writel_relaxed(0xffffffff, base + MBOX_IMASK_REG);
> > > > > +	writel_relaxed(BIT(mdev->ack_irq), base + MBOX_SRC_REG);
> > > > > +	__asm__ volatile ("sev");
> > > > > +	return 0;
> > > > > +}
> > > > 
> > > > Drivers really shouldn't be using SEV directly (even if via the
> > > > sev() macro)...
> > > > 
> > > > This SEV isn't ordered w.r.t. anything, and it's unclear what
> > > > ordering you need, so this simply does not work.
> > > 
> > > I will leave your questions for Hisilicon colleagues, essentially your
> > > questions are related with mailbox mechanism.
> > > 
> > > But I'd like to firstly get clear your question for "This SEV isn't
> > > ordered w.r.t. anything". From my understanding, ARMv8 architecture
> > > natually adds DMB before SEV so all previous register writing
> > > opreations should be ensured to endpoint before SEV?
> > 
> > This is not the case; SEV does not add any implicit memory barrier, and
> > is not ordered w.r.t. memory accesses.
> > 
> > See ARM DDI 0487B.b, page D1-1905, "The Send Event instructions":
> > 
> >     The PE is not required to guarantee the ordering of this event with
> >     respect to the completion of memory accesses by instructions before
> >     the SEV instruction. Therefore, ARM recommends that software
> >     includes a DSB instruction before any SEV instruction.
> 
> My fault and thanks for explanation.
> 
> > Note that a DMB is not sufficient, as SEV is not a memory access.
> 
> Understood now, so below code should be safe?
> 
> wmb();  -> dsb(st);
> sev();

Whether that is safe depends on what you are trying to ensure is
ordered, and what the other side (with the WFE) is doing.

For example, my understanding is that in general, WFE is also not
ordered w.r.t.  memory accesses.

This is a very subtle part of the architecture, and I'm very much not
keen on using WFE and SEV outside of architecture code implementing
locking primitives.

Thanks,
Mark.
Zhong Kaihua Nov. 2, 2017, 9:15 a.m. UTC | #7
Hi, Julien,

On 2017/10/27 18:40, Julien Thierry wrote:
> Hi Kaihua,
> 
> On 27/10/17 07:15, Kaihua Zhong wrote:
>> Hi3660 mailbox controller is used to send message within multiple
>> processors, MCU, HIFI, etc.  It supports 32 mailbox channels and every
>> channel can only be used for single transferring direction.  Once the
>> channel is enabled, it needs to specify the destination interrupt and
>> acknowledge interrupt, these two interrupt vectors are used to create
>> the connection between the mailbox and interrupt controllers.
>>
>> The application processor (or from point of view of kernel) is not the
>> only one master which can launch the data transferring, other
>> processors or MCU/DSP also can kick off the data transferring.  So this
>> driver implements a locking mechanism to support exclusive accessing.
>>
>> The data transferring supports two modes, one is named as "automatic
>> acknowledge" mode so after send message the kernel doesn't need to wait
>> for acknowledge from remote and directly return; there have another mode
>> is to rely on handling interrupt for acknowledge.
>>
>> This commit is for initial version driver, which only supports
>> "automatic acknowledge" mode to support CPU clock, which is the only
>> one consumer to use mailbox and has been verified.  Later may enhance
>> this driver for interrupt mode (e.g. for supporting HIFI).
>>
>> Cc: John Stultz <john.stultz@linaro.org>
>> Cc: Guodong Xu <guodong.xu@linaro.org>
>> Cc: Haojian Zhuang <haojian.zhuang@linaro.org>
>> Cc: Niranjan Yadla <nyadla@cadence.com>
>> Cc: Raj Pawate <pawateb@cadence.com>
>> Signed-off-by: Leo Yan <leo.yan@linaro.org>
>> Signed-off-by: Ruyi Wang <wangruyi@huawei.com>
>> Signed-off-by: Kaihua Zhong <zhongkaihua@huawei.com>
>> Signed-off-by: Kevin Wang <kevin.wangtao@hisilicon.com>
>> ---
>>   drivers/mailbox/Kconfig          |   8 +
>>   drivers/mailbox/Makefile         |   2 +
>>   drivers/mailbox/hi3660-mailbox.c | 331 +++++++++++++++++++++++++++++++++++++++
>>   3 files changed, 341 insertions(+)
>>   create mode 100644 drivers/mailbox/hi3660-mailbox.c
>>
>> diff --git a/drivers/mailbox/Kconfig b/drivers/mailbox/Kconfig
>> index c5731e5..4b5d6e9 100644
>> --- a/drivers/mailbox/Kconfig
>> +++ b/drivers/mailbox/Kconfig
>> @@ -108,6 +108,14 @@ config TI_MESSAGE_MANAGER
>>         multiple processors within the SoC. Select this driver if your
>>         platform has support for the hardware block.
>>   +config HI3660_MBOX
>> +    tristate "Hi3660 Mailbox"
>> +    depends on ARCH_HISI && OF
>> +    help
>> +      An implementation of the hi3660 mailbox. It is used to send message
>> +      between application processors and other processors/MCU/DSP. Select
>> +      Y here if you want to use Hi3660 mailbox controller.
>> +
>>   config HI6220_MBOX
>>       tristate "Hi6220 Mailbox"
>>       depends on ARCH_HISI
>> diff --git a/drivers/mailbox/Makefile b/drivers/mailbox/Makefile
>> index d54e412..7d1bd51 100644
>> --- a/drivers/mailbox/Makefile
>> +++ b/drivers/mailbox/Makefile
>> @@ -26,6 +26,8 @@ obj-$(CONFIG_TI_MESSAGE_MANAGER) += ti-msgmgr.o
>>     obj-$(CONFIG_XGENE_SLIMPRO_MBOX) += mailbox-xgene-slimpro.o
>>   +obj-$(CONFIG_HI3660_MBOX)    += hi3660-mailbox.o
>> +
>>   obj-$(CONFIG_HI6220_MBOX)    += hi6220-mailbox.o
>>     obj-$(CONFIG_BCM_PDC_MBOX)    += bcm-pdc-mailbox.o
>> diff --git a/drivers/mailbox/hi3660-mailbox.c b/drivers/mailbox/hi3660-mailbox.c
>> new file mode 100644
>> index 0000000..67df8f8
>> --- /dev/null
>> +++ b/drivers/mailbox/hi3660-mailbox.c
>> @@ -0,0 +1,331 @@
>> +/*
>> + * Hisilicon's Hi3660 mailbox controller driver
>> + *
>> + * Copyright (c) 2017 Hisilicon Limited.
>> + * Copyright (c) 2017 Linaro Limited.
>> + *
>> + * Author: Leo Yan <leo.yan@linaro.org>
>> + *
>> + * This program is free software: you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation, version 2 of the License.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + */
>> +
>> +#include <linux/bitops.h>
>> +#include <linux/delay.h>
>> +#include <linux/device.h>
>> +#include <linux/err.h>
>> +#include <linux/interrupt.h>
>> +#include <linux/io.h>
>> +#include <linux/iopoll.h>
>> +#include <linux/mailbox_controller.h>
>> +#include <linux/module.h>
>> +#include <linux/platform_device.h>
>> +#include <linux/slab.h>
>> +
>> +#include "mailbox.h"
>> +
>> +#define MBOX_CHAN_MAX            32
>> +
>> +#define MBOX_RX                (0x0)
>> +#define MBOX_TX                (0x1)
>> +
>> +#define MBOX_BASE(mbox, ch)        ((mbox)->base + ((ch) * 0x40))
>> +#define MBOX_SRC_REG            (0x00)
>> +#define MBOX_DST_REG            (0x04)
>> +#define MBOX_DCLR_REG            (0x08)
>> +#define MBOX_DSTAT_REG            (0x0c)
>> +#define MBOX_MODE_REG            (0x10)
>> +#define MBOX_IMASK_REG            (0x14)
>> +#define MBOX_ICLR_REG            (0x18)
>> +#define MBOX_SEND_REG            (0x1c)
>> +#define MBOX_DATA_REG            (0x20)
>> +
>> +#define MBOX_IPC_LOCK_REG        (0xa00)
>> +#define MBOX_IPC_UNLOCK            (0x1acce551)
>> +
>> +#define MBOX_AUTOMATIC_ACK        (1)
>> +
>> +#define MBOX_STATE_IDLE            BIT(4)
>> +#define MBOX_STATE_ACK            BIT(7)
>> +
>> +#define MBOX_MSG_LEN            8
>> +
>> +/**
>> + * Hi3660 mailbox channel device data
>> + *
>> + * A channel can be used for TX or RX, it can trigger remote
>> + * processor interrupt to notify remote processor and can receive
>> + * interrupt if has incoming message.
>> + *
>> + * @dst_irq:    Interrupt vector for remote processor
>> + * @ack_irq:    Interrupt vector for local processor
>> + */
>> +struct hi3660_mbox_dev {
>> +    unsigned int dst_irq;
>> +    unsigned int ack_irq;
>> +};
>> +
>> +/**
>> + * Hi3660 mailbox controller data
>> + *
>> + * Mailbox controller includes 32 channels and can allocate
>> + * channel for message transferring.
>> + *
>> + * @dev:    Device to which it is attached
>> + * @base:    Base address of the register mapping region
>> + * @chan:    Representation of channels in mailbox controller
>> + * @mdev:    Representation of channel device data
>> + * @controller:    Representation of a communication channel controller
>> + */
>> +struct hi3660_mbox {
>> +    struct device *dev;
>> +    void __iomem *base;
>> +    struct mbox_chan chan[MBOX_CHAN_MAX];
>> +    struct hi3660_mbox_dev mdev[MBOX_CHAN_MAX];
>> +    struct mbox_controller controller;
>> +};
>> +
>> +static inline struct hi3660_mbox *to_hi3660_mbox(struct mbox_chan *chan)
>> +{
>> +    return container_of(chan->mbox, struct hi3660_mbox, controller);
>> +}
>> +
>> +static int hi3660_mbox_check_state(struct mbox_chan *chan)
>> +{
>> +    unsigned long ch = (unsigned long)chan->con_priv;
>> +    struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
>> +    struct hi3660_mbox_dev *mdev = &mbox->mdev[ch];
>> +    void __iomem *base = MBOX_BASE(mbox, ch);
>> +    unsigned long val;
>> +    unsigned int state, ret;
>> +
>> +    /* Mailbox is idle so directly bail out */
>> +    state = readl_relaxed(base + MBOX_MODE_REG);
>> +    if (state & MBOX_STATE_IDLE)
>> +        return 0;
>> +
>> +    /* Wait for acknowledge from remote */
>> +    ret = readx_poll_timeout_atomic(readl_relaxed, base + MBOX_MODE_REG,
>> +            val, (val & MBOX_STATE_ACK), 1000, 300000);
>> +    if (ret) {
>> +        dev_err(mbox->dev, "%s: timeout for receiving ack\n", __func__);
>> +        return ret;
>> +    }
>> +
>> +    /* Ensure channel is released */
>> +    writel_relaxed(0xffffffff, base + MBOX_IMASK_REG);
>> +    writel_relaxed(BIT(mdev->ack_irq), base + MBOX_SRC_REG);
>> +    __asm__ volatile ("sev");
> 
> There is an existing sev() macro for this.
> 
Macro will be used in next patch,if this function is not redesigned.
Mark and Leo is discussing memory barrier protection issues.

>> +    return 0;
>> +}
>> +
>> +static int hi3660_mbox_unlock(struct mbox_chan *chan)
>> +{
>> +    struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
>> +    unsigned int val, retry = 3;
>> +
>> +    do {
>> +        writel_relaxed(MBOX_IPC_UNLOCK, mbox->base + MBOX_IPC_LOCK_REG);
>> +
>> +        val = readl_relaxed(mbox->base + MBOX_IPC_LOCK_REG);
>> +        if (!val)
>> +            break;
>> +
>> +        udelay(10);
>> +    } while (retry--);
>> +
>> +    return (!val) ? 0 : -ETIMEDOUT;
>> +}
>> +
>> +static int hi3660_mbox_acquire_channel(struct mbox_chan *chan)
>> +{
>> +    unsigned long ch = (unsigned long)chan->con_priv;
>> +    struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
>> +    struct hi3660_mbox_dev *mdev = &mbox->mdev[ch];
>> +    void __iomem *base = MBOX_BASE(mbox, ch);
>> +    unsigned int val = 0, retry = 10;
>> +
>> +    /*
>> +     * Hardware locking for exclusive accessing within CPUs
>> +     * without exclusive monitor mechanism.
>> +     */
>> +    do {
>> +        val = readl_relaxed(base + MBOX_MODE_REG);
>> +        if (!(val & MBOX_STATE_IDLE)) {
>> +            __asm__ volatile ("wfe");
>> +            continue;
> 
> So this is going to "wfe" when retry == 0 and the continue statement will take us out of the loop?
> 
> Also, there is a wfe() macro for arm and arm64 which could be used here.
> 
>> +        }
>> +
>> +        /* Check if channel has be acquired */
>> +        writel_relaxed(BIT(mdev->ack_irq), base + MBOX_SRC_REG);
>> +        val = readl_relaxed(base + MBOX_SRC_REG) & BIT(mdev->ack_irq);
>> +        if (val)
>> +            break;
>> +
>> +    } while (retry--);
>> +
>> +    return (val) ? 0 : -ETIMEDOUT;
> 
> If timeout occurs while waiting for the MBOX to be idle, val will hold the last value from MBOX_MODE_REG, and if this can be different than 0, this will hide the fact it timed out.
> 
> Maybe the following would be more appropriate:
> 
> return (retry >= 0) ? 0 : -ETIMEDOUT;
> 
> Also, your do {} while loops for the timeouts falsely give the impression we do "retry" attempts when we actually do "retry + 1" attempts (but I guess it doesn't make a big difference from the functional point of view).
> 

Agreed. The current logic of code is confusing as val is used twice.

>> +}
>> +
>> +static int hi3660_mbox_send(struct mbox_chan *chan, u32 *msg)
>> +{
>> +    unsigned long ch = (unsigned long)chan->con_priv;
>> +    struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
>> +    struct hi3660_mbox_dev *mdev = &mbox->mdev[ch];
>> +    void __iomem *base = MBOX_BASE(mbox, ch);
>> +    unsigned int i;
>> +
>> +    /* Clear mask for destination interrupt */
>> +    writel_relaxed(~BIT(mdev->dst_irq), base + MBOX_IMASK_REG);
>> +
>> +    /* Config destination for interrupt vector */
>> +    writel_relaxed(BIT(mdev->dst_irq), base + MBOX_DST_REG);
>> +
>> +    /* Automatic acknowledge mode */
>> +    writel_relaxed(MBOX_AUTOMATIC_ACK, base + MBOX_MODE_REG);
>> +
>> +    /* Fill message data */
>> +    for (i = 0; i < MBOX_MSG_LEN; i++)
>> +        writel_relaxed(msg[i], base + MBOX_DATA_REG + i * 4);
>> +
>> +    /* Trigger data transferring */
>> +    writel_relaxed(BIT(mdev->ack_irq), base + MBOX_SEND_REG);
>> +    return 0;
>> +}
>> +
>> +static int hi3660_mbox_send_data(struct mbox_chan *chan, void *msg)
>> +{
>> +    struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
>> +    int err;
>> +
>> +    err = hi3660_mbox_check_state(chan);
>> +    if (err) {
>> +        dev_err(mbox->dev, "checking state failed\n");
>> +        return err;
>> +    }
>> +
>> +    err = hi3660_mbox_unlock(chan);
>> +    if (err) {
>> +        dev_err(mbox->dev, "unlocking mailbox failed\n");
>> +        return err;
>> +    }
>> +
>> +    err = hi3660_mbox_acquire_channel(chan);
>> +    if (err) {
>> +        dev_err(mbox->dev, "acquiring channel failed\n");
>> +        return err;
>> +    }
>> +
>> +    return hi3660_mbox_send(chan, msg);
>> +}
>> +
>> +static struct mbox_chan_ops hi3660_mbox_ops = {
>> +    .send_data    = hi3660_mbox_send_data,
>> +};
>> +
>> +static struct mbox_chan *hi3660_mbox_xlate(struct mbox_controller *controller,
>> +                       const struct of_phandle_args *spec)
>> +{
>> +    struct hi3660_mbox *mbox = dev_get_drvdata(controller->dev);
> 
> nit:
> In to_hi3660_mbox, you use the fact that the mbox_controller structure is part of hi3660_mbox and use container of to retrieve hi3660_mbox.
> 
> I think it would be nice to be consistent about this, either use container_of or dev drvdata, but not both.
> 
> Cheers,
> 
I will discuss with Leo.

Thanks for your comments.

Best Regards,
diff mbox

Patch

diff --git a/drivers/mailbox/Kconfig b/drivers/mailbox/Kconfig
index c5731e5..4b5d6e9 100644
--- a/drivers/mailbox/Kconfig
+++ b/drivers/mailbox/Kconfig
@@ -108,6 +108,14 @@  config TI_MESSAGE_MANAGER
 	  multiple processors within the SoC. Select this driver if your
 	  platform has support for the hardware block.
 
+config HI3660_MBOX
+	tristate "Hi3660 Mailbox"
+	depends on ARCH_HISI && OF
+	help
+	  An implementation of the hi3660 mailbox. It is used to send message
+	  between application processors and other processors/MCU/DSP. Select
+	  Y here if you want to use Hi3660 mailbox controller.
+
 config HI6220_MBOX
 	tristate "Hi6220 Mailbox"
 	depends on ARCH_HISI
diff --git a/drivers/mailbox/Makefile b/drivers/mailbox/Makefile
index d54e412..7d1bd51 100644
--- a/drivers/mailbox/Makefile
+++ b/drivers/mailbox/Makefile
@@ -26,6 +26,8 @@  obj-$(CONFIG_TI_MESSAGE_MANAGER) += ti-msgmgr.o
 
 obj-$(CONFIG_XGENE_SLIMPRO_MBOX) += mailbox-xgene-slimpro.o
 
+obj-$(CONFIG_HI3660_MBOX)	+= hi3660-mailbox.o
+
 obj-$(CONFIG_HI6220_MBOX)	+= hi6220-mailbox.o
 
 obj-$(CONFIG_BCM_PDC_MBOX)	+= bcm-pdc-mailbox.o
diff --git a/drivers/mailbox/hi3660-mailbox.c b/drivers/mailbox/hi3660-mailbox.c
new file mode 100644
index 0000000..67df8f8
--- /dev/null
+++ b/drivers/mailbox/hi3660-mailbox.c
@@ -0,0 +1,331 @@ 
+/*
+ * Hisilicon's Hi3660 mailbox controller driver
+ *
+ * Copyright (c) 2017 Hisilicon Limited.
+ * Copyright (c) 2017 Linaro Limited.
+ *
+ * Author: Leo Yan <leo.yan@linaro.org>
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, version 2 of the License.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ */
+
+#include <linux/bitops.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/err.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/iopoll.h>
+#include <linux/mailbox_controller.h>
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <linux/slab.h>
+
+#include "mailbox.h"
+
+#define MBOX_CHAN_MAX			32
+
+#define MBOX_RX				(0x0)
+#define MBOX_TX				(0x1)
+
+#define MBOX_BASE(mbox, ch)		((mbox)->base + ((ch) * 0x40))
+#define MBOX_SRC_REG			(0x00)
+#define MBOX_DST_REG			(0x04)
+#define MBOX_DCLR_REG			(0x08)
+#define MBOX_DSTAT_REG			(0x0c)
+#define MBOX_MODE_REG			(0x10)
+#define MBOX_IMASK_REG			(0x14)
+#define MBOX_ICLR_REG			(0x18)
+#define MBOX_SEND_REG			(0x1c)
+#define MBOX_DATA_REG			(0x20)
+
+#define MBOX_IPC_LOCK_REG		(0xa00)
+#define MBOX_IPC_UNLOCK			(0x1acce551)
+
+#define MBOX_AUTOMATIC_ACK		(1)
+
+#define MBOX_STATE_IDLE			BIT(4)
+#define MBOX_STATE_ACK			BIT(7)
+
+#define MBOX_MSG_LEN			8
+
+/**
+ * Hi3660 mailbox channel device data
+ *
+ * A channel can be used for TX or RX, it can trigger remote
+ * processor interrupt to notify remote processor and can receive
+ * interrupt if has incoming message.
+ *
+ * @dst_irq:	Interrupt vector for remote processor
+ * @ack_irq:	Interrupt vector for local processor
+ */
+struct hi3660_mbox_dev {
+	unsigned int dst_irq;
+	unsigned int ack_irq;
+};
+
+/**
+ * Hi3660 mailbox controller data
+ *
+ * Mailbox controller includes 32 channels and can allocate
+ * channel for message transferring.
+ *
+ * @dev:	Device to which it is attached
+ * @base:	Base address of the register mapping region
+ * @chan:	Representation of channels in mailbox controller
+ * @mdev:	Representation of channel device data
+ * @controller:	Representation of a communication channel controller
+ */
+struct hi3660_mbox {
+	struct device *dev;
+	void __iomem *base;
+	struct mbox_chan chan[MBOX_CHAN_MAX];
+	struct hi3660_mbox_dev mdev[MBOX_CHAN_MAX];
+	struct mbox_controller controller;
+};
+
+static inline struct hi3660_mbox *to_hi3660_mbox(struct mbox_chan *chan)
+{
+	return container_of(chan->mbox, struct hi3660_mbox, controller);
+}
+
+static int hi3660_mbox_check_state(struct mbox_chan *chan)
+{
+	unsigned long ch = (unsigned long)chan->con_priv;
+	struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
+	struct hi3660_mbox_dev *mdev = &mbox->mdev[ch];
+	void __iomem *base = MBOX_BASE(mbox, ch);
+	unsigned long val;
+	unsigned int state, ret;
+
+	/* Mailbox is idle so directly bail out */
+	state = readl_relaxed(base + MBOX_MODE_REG);
+	if (state & MBOX_STATE_IDLE)
+		return 0;
+
+	/* Wait for acknowledge from remote */
+	ret = readx_poll_timeout_atomic(readl_relaxed, base + MBOX_MODE_REG,
+			val, (val & MBOX_STATE_ACK), 1000, 300000);
+	if (ret) {
+		dev_err(mbox->dev, "%s: timeout for receiving ack\n", __func__);
+		return ret;
+	}
+
+	/* Ensure channel is released */
+	writel_relaxed(0xffffffff, base + MBOX_IMASK_REG);
+	writel_relaxed(BIT(mdev->ack_irq), base + MBOX_SRC_REG);
+	__asm__ volatile ("sev");
+	return 0;
+}
+
+static int hi3660_mbox_unlock(struct mbox_chan *chan)
+{
+	struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
+	unsigned int val, retry = 3;
+
+	do {
+		writel_relaxed(MBOX_IPC_UNLOCK, mbox->base + MBOX_IPC_LOCK_REG);
+
+		val = readl_relaxed(mbox->base + MBOX_IPC_LOCK_REG);
+		if (!val)
+			break;
+
+		udelay(10);
+	} while (retry--);
+
+	return (!val) ? 0 : -ETIMEDOUT;
+}
+
+static int hi3660_mbox_acquire_channel(struct mbox_chan *chan)
+{
+	unsigned long ch = (unsigned long)chan->con_priv;
+	struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
+	struct hi3660_mbox_dev *mdev = &mbox->mdev[ch];
+	void __iomem *base = MBOX_BASE(mbox, ch);
+	unsigned int val = 0, retry = 10;
+
+	/*
+	 * Hardware locking for exclusive accessing within CPUs
+	 * without exclusive monitor mechanism.
+	 */
+	do {
+		val = readl_relaxed(base + MBOX_MODE_REG);
+		if (!(val & MBOX_STATE_IDLE)) {
+			__asm__ volatile ("wfe");
+			continue;
+		}
+
+		/* Check if channel has be acquired */
+		writel_relaxed(BIT(mdev->ack_irq), base + MBOX_SRC_REG);
+		val = readl_relaxed(base + MBOX_SRC_REG) & BIT(mdev->ack_irq);
+		if (val)
+			break;
+
+	} while (retry--);
+
+	return (val) ? 0 : -ETIMEDOUT;
+}
+
+static int hi3660_mbox_send(struct mbox_chan *chan, u32 *msg)
+{
+	unsigned long ch = (unsigned long)chan->con_priv;
+	struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
+	struct hi3660_mbox_dev *mdev = &mbox->mdev[ch];
+	void __iomem *base = MBOX_BASE(mbox, ch);
+	unsigned int i;
+
+	/* Clear mask for destination interrupt */
+	writel_relaxed(~BIT(mdev->dst_irq), base + MBOX_IMASK_REG);
+
+	/* Config destination for interrupt vector */
+	writel_relaxed(BIT(mdev->dst_irq), base + MBOX_DST_REG);
+
+	/* Automatic acknowledge mode */
+	writel_relaxed(MBOX_AUTOMATIC_ACK, base + MBOX_MODE_REG);
+
+	/* Fill message data */
+	for (i = 0; i < MBOX_MSG_LEN; i++)
+		writel_relaxed(msg[i], base + MBOX_DATA_REG + i * 4);
+
+	/* Trigger data transferring */
+	writel_relaxed(BIT(mdev->ack_irq), base + MBOX_SEND_REG);
+	return 0;
+}
+
+static int hi3660_mbox_send_data(struct mbox_chan *chan, void *msg)
+{
+	struct hi3660_mbox *mbox = to_hi3660_mbox(chan);
+	int err;
+
+	err = hi3660_mbox_check_state(chan);
+	if (err) {
+		dev_err(mbox->dev, "checking state failed\n");
+		return err;
+	}
+
+	err = hi3660_mbox_unlock(chan);
+	if (err) {
+		dev_err(mbox->dev, "unlocking mailbox failed\n");
+		return err;
+	}
+
+	err = hi3660_mbox_acquire_channel(chan);
+	if (err) {
+		dev_err(mbox->dev, "acquiring channel failed\n");
+		return err;
+	}
+
+	return hi3660_mbox_send(chan, msg);
+}
+
+static struct mbox_chan_ops hi3660_mbox_ops = {
+	.send_data    = hi3660_mbox_send_data,
+};
+
+static struct mbox_chan *hi3660_mbox_xlate(struct mbox_controller *controller,
+					   const struct of_phandle_args *spec)
+{
+	struct hi3660_mbox *mbox = dev_get_drvdata(controller->dev);
+	struct hi3660_mbox_dev *mdev;
+	unsigned int ch = spec->args[0];
+
+	if (ch >= MBOX_CHAN_MAX) {
+		dev_err(mbox->dev, "Invalid channel idx %d\n", ch);
+		return ERR_PTR(-EINVAL);
+	}
+
+	mdev = &mbox->mdev[ch];
+	mdev->dst_irq = spec->args[1];
+	mdev->ack_irq = spec->args[2];
+
+	return &mbox->chan[ch];
+}
+
+static const struct of_device_id hi3660_mbox_of_match[] = {
+	{ .compatible = "hisilicon,hi3660-mbox", },
+	{},
+};
+
+MODULE_DEVICE_TABLE(of, hi3660_mbox_of_match);
+
+static int hi3660_mbox_probe(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct hi3660_mbox *mbox;
+	struct mbox_chan *chan;
+	struct resource *res;
+	unsigned long ch;
+	int err;
+
+	mbox = devm_kzalloc(dev, sizeof(*mbox), GFP_KERNEL);
+	if (!mbox)
+		return -ENOMEM;
+
+	res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+	mbox->base = devm_ioremap_resource(dev, res);
+	if (IS_ERR(mbox->base))
+		return PTR_ERR(mbox->base);
+
+	mbox->dev = dev;
+	mbox->controller.dev = dev;
+	mbox->controller.chans = mbox->chan;
+	mbox->controller.num_chans = MBOX_CHAN_MAX;
+	mbox->controller.ops = &hi3660_mbox_ops;
+	mbox->controller.of_xlate = hi3660_mbox_xlate;
+
+	/* Initialize mailbox channel data */
+	chan = mbox->chan;
+	for (ch = 0; ch < MBOX_CHAN_MAX; ch++)
+		chan[ch].con_priv = (void *)ch;
+
+	err = mbox_controller_register(&mbox->controller);
+	if (err) {
+		dev_err(dev, "Failed to register mailbox %d\n", err);
+		return err;
+	}
+
+	platform_set_drvdata(pdev, mbox);
+	dev_info(dev, "Mailbox enabled\n");
+	return 0;
+}
+
+static int hi3660_mbox_remove(struct platform_device *pdev)
+{
+	struct hi3660_mbox *mbox = platform_get_drvdata(pdev);
+
+	mbox_controller_unregister(&mbox->controller);
+	return 0;
+}
+
+static struct platform_driver hi3660_mbox_driver = {
+	.probe  = hi3660_mbox_probe,
+	.remove = hi3660_mbox_remove,
+	.driver = {
+		.name = "hi3660-mbox",
+		.of_match_table = hi3660_mbox_of_match,
+	},
+};
+
+static int __init hi3660_mbox_init(void)
+{
+	return platform_driver_register(&hi3660_mbox_driver);
+}
+core_initcall(hi3660_mbox_init);
+
+static void __exit hi3660_mbox_exit(void)
+{
+	platform_driver_unregister(&hi3660_mbox_driver);
+}
+module_exit(hi3660_mbox_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("Hisilicon Hi3660 Mailbox Controller");
+MODULE_AUTHOR("Leo Yan <leo.yan@linaro.org>");