| Message ID | 20220822155130.2491006-5-Frank.Li@nxp.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | [v7,1/4] irqchip: allow pass down .pm field at IRQCHIP_PLATFORM_DRIVER_END | expand |
Hi Frank,
I love your patch! Yet something to improve:
[auto build test ERROR on jonmason-ntb/ntb-next]
[also build test ERROR on robh/for-next linus/master v6.0-rc2 next-20220822]
[cannot apply to tip/irq/core]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Frank-Li/PCI-EP-driver-support-MSI-doorbell-from-host/20220822-235323
base: https://github.com/jonmason/ntb ntb-next
config: microblaze-randconfig-r005-20220821 (https://download.01.org/0day-ci/archive/20220823/202208230543.fAaLXJy7-lkp@intel.com/config)
compiler: microblaze-linux-gcc (GCC) 12.1.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# https://github.com/intel-lab-lkp/linux/commit/d98704aefa5b57814d7b9b1b40160df34977b2b6
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Frank-Li/PCI-EP-driver-support-MSI-doorbell-from-host/20220822-235323
git checkout d98704aefa5b57814d7b9b1b40160df34977b2b6
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=microblaze SHELL=/bin/bash drivers/pci/endpoint/functions/
If you fix the issue, kindly add following tag where applicable
Reported-by: kernel test robot <lkp@intel.com>
All errors (new ones prefixed by >>):
drivers/pci/endpoint/functions/pci-epf-vntb.c: In function 'epf_ntb_epc_msi_init':
>> drivers/pci/endpoint/functions/pci-epf-vntb.c:786:13: error: implicit declaration of function 'platform_msi_domain_alloc_irqs' [-Werror=implicit-function-declaration]
786 | if (platform_msi_domain_alloc_irqs(&ntb->epf->dev,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1: some warnings being treated as errors
vim +/platform_msi_domain_alloc_irqs +786 drivers/pci/endpoint/functions/pci-epf-vntb.c
771
772 static void epf_ntb_epc_msi_init(struct epf_ntb *ntb)
773 {
774 struct device *dev = &ntb->epf->dev;
775 struct irq_domain *domain;
776 int virq;
777 int ret;
778 int i;
779
780 domain = dev_get_msi_domain(ntb->epf->epc->dev.parent);
781 if (!domain)
782 return;
783
784 dev_set_msi_domain(dev, domain);
785
> 786 if (platform_msi_domain_alloc_irqs(&ntb->epf->dev,
787 ntb->db_count,
788 epf_ntb_write_msi_msg)) {
789 dev_info(dev, "Can't allocate MSI, fall back to poll mode\n");
790 return;
791 }
792
793 dev_info(dev, "vntb use MSI as doorbell\n");
794
795 for (i = 0; i < ntb->db_count; i++) {
796 virq = msi_get_virq(dev, i);
797 ret = devm_request_irq(dev, virq,
798 epf_ntb_interrupt_handler, 0,
799 "ntb", ntb);
800
801 if (ret) {
802 dev_err(dev, "devm_request_irq() failure, fall back to poll mode\n");
803 ntb->epf_db_phy = 0;
804 break;
805 }
806
807 if (!i)
808 ntb->msi_virqbase = virq;
809 }
810 }
811
Hi Frank,
I love your patch! Perhaps something to improve:
[auto build test WARNING on jonmason-ntb/ntb-next]
[also build test WARNING on robh/for-next linus/master v6.0-rc2 next-20220822]
[cannot apply to tip/irq/core]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Frank-Li/PCI-EP-driver-support-MSI-doorbell-from-host/20220822-235323
base: https://github.com/jonmason/ntb ntb-next
config: s390-randconfig-s053-20220821 (https://download.01.org/0day-ci/archive/20220823/202208230844.D25Fw8sg-lkp@intel.com/config)
compiler: s390-linux-gcc (GCC) 12.1.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# apt-get install sparse
# sparse version: v0.6.4-39-gce1a6720-dirty
# https://github.com/intel-lab-lkp/linux/commit/d98704aefa5b57814d7b9b1b40160df34977b2b6
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Frank-Li/PCI-EP-driver-support-MSI-doorbell-from-host/20220822-235323
git checkout d98704aefa5b57814d7b9b1b40160df34977b2b6
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross C=1 CF='-fdiagnostic-prefix -D__CHECK_ENDIAN__' O=build_dir ARCH=s390 SHELL=/bin/bash drivers/pci/endpoint/functions/
If you fix the issue, kindly add following tag where applicable
Reported-by: kernel test robot <lkp@intel.com>
sparse warnings: (new ones prefixed by >>)
>> drivers/pci/endpoint/functions/pci-epf-vntb.c:567:25: sparse: sparse: incorrect type in assignment (different address spaces) @@ expected void [noderef] __iomem *[assigned] mw_addr @@ got void * @@
drivers/pci/endpoint/functions/pci-epf-vntb.c:567:25: sparse: expected void [noderef] __iomem *[assigned] mw_addr
drivers/pci/endpoint/functions/pci-epf-vntb.c:567:25: sparse: got void *
drivers/pci/endpoint/functions/pci-epf-vntb.c:600:41: sparse: sparse: incorrect type in argument 2 (different address spaces) @@ expected void *addr @@ got void [noderef] __iomem *epf_db @@
drivers/pci/endpoint/functions/pci-epf-vntb.c:600:41: sparse: expected void *addr
drivers/pci/endpoint/functions/pci-epf-vntb.c:600:41: sparse: got void [noderef] __iomem *epf_db
drivers/pci/endpoint/functions/pci-epf-vntb.c:1201:33: sparse: sparse: incorrect type in initializer (different address spaces) @@ expected void [noderef] __iomem *base @@ got struct epf_ntb_ctrl *reg @@
drivers/pci/endpoint/functions/pci-epf-vntb.c:1201:33: sparse: expected void [noderef] __iomem *base
drivers/pci/endpoint/functions/pci-epf-vntb.c:1201:33: sparse: got struct epf_ntb_ctrl *reg
drivers/pci/endpoint/functions/pci-epf-vntb.c:1212:33: sparse: sparse: incorrect type in initializer (different address spaces) @@ expected void [noderef] __iomem *base @@ got struct epf_ntb_ctrl *reg @@
drivers/pci/endpoint/functions/pci-epf-vntb.c:1212:33: sparse: expected void [noderef] __iomem *base
drivers/pci/endpoint/functions/pci-epf-vntb.c:1212:33: sparse: got struct epf_ntb_ctrl *reg
drivers/pci/endpoint/functions/pci-epf-vntb.c:1223:33: sparse: sparse: incorrect type in initializer (different address spaces) @@ expected void [noderef] __iomem *base @@ got struct epf_ntb_ctrl *reg @@
drivers/pci/endpoint/functions/pci-epf-vntb.c:1223:33: sparse: expected void [noderef] __iomem *base
drivers/pci/endpoint/functions/pci-epf-vntb.c:1223:33: sparse: got struct epf_ntb_ctrl *reg
drivers/pci/endpoint/functions/pci-epf-vntb.c:1235:33: sparse: sparse: incorrect type in initializer (different address spaces) @@ expected void [noderef] __iomem *base @@ got struct epf_ntb_ctrl *reg @@
drivers/pci/endpoint/functions/pci-epf-vntb.c:1235:33: sparse: expected void [noderef] __iomem *base
drivers/pci/endpoint/functions/pci-epf-vntb.c:1235:33: sparse: got struct epf_ntb_ctrl *reg
vim +567 drivers/pci/endpoint/functions/pci-epf-vntb.c
d98704aefa5b578 Frank Li 2022-08-22 536
e35f56bb03304ab Frank Li 2022-02-22 537 /**
e35f56bb03304ab Frank Li 2022-02-22 538 * epf_ntb_db_bar_init() - Configure Doorbell window BARs
e35f56bb03304ab Frank Li 2022-02-22 539 * @ntb: NTB device that facilitates communication between HOST and vHOST
e35f56bb03304ab Frank Li 2022-02-22 540 */
e35f56bb03304ab Frank Li 2022-02-22 541 static int epf_ntb_db_bar_init(struct epf_ntb *ntb)
e35f56bb03304ab Frank Li 2022-02-22 542 {
e35f56bb03304ab Frank Li 2022-02-22 543 const struct pci_epc_features *epc_features;
e35f56bb03304ab Frank Li 2022-02-22 544 u32 align;
e35f56bb03304ab Frank Li 2022-02-22 545 struct device *dev = &ntb->epf->dev;
e35f56bb03304ab Frank Li 2022-02-22 546 int ret;
e35f56bb03304ab Frank Li 2022-02-22 547 struct pci_epf_bar *epf_bar;
e35f56bb03304ab Frank Li 2022-02-22 548 void __iomem *mw_addr;
e35f56bb03304ab Frank Li 2022-02-22 549 enum pci_barno barno;
d98704aefa5b578 Frank Li 2022-08-22 550 size_t size;
e35f56bb03304ab Frank Li 2022-02-22 551
e35f56bb03304ab Frank Li 2022-02-22 552 epc_features = pci_epc_get_features(ntb->epf->epc,
e35f56bb03304ab Frank Li 2022-02-22 553 ntb->epf->func_no,
e35f56bb03304ab Frank Li 2022-02-22 554 ntb->epf->vfunc_no);
e35f56bb03304ab Frank Li 2022-02-22 555 align = epc_features->align;
d98704aefa5b578 Frank Li 2022-08-22 556 size = epf_ntb_db_size(ntb);
e35f56bb03304ab Frank Li 2022-02-22 557
e35f56bb03304ab Frank Li 2022-02-22 558 barno = ntb->epf_ntb_bar[BAR_DB];
d98704aefa5b578 Frank Li 2022-08-22 559 epf_bar = &ntb->epf->bar[barno];
e35f56bb03304ab Frank Li 2022-02-22 560
d98704aefa5b578 Frank Li 2022-08-22 561 if (ntb->epf_db_phy) {
d98704aefa5b578 Frank Li 2022-08-22 562 mw_addr = NULL;
d98704aefa5b578 Frank Li 2022-08-22 563 epf_bar->phys_addr = ntb->epf_db_phy;
d98704aefa5b578 Frank Li 2022-08-22 564 epf_bar->barno = barno;
d98704aefa5b578 Frank Li 2022-08-22 565 epf_bar->size = size;
d98704aefa5b578 Frank Li 2022-08-22 566 } else {
e35f56bb03304ab Frank Li 2022-02-22 @567 mw_addr = pci_epf_alloc_space(ntb->epf, size, barno, align, 0);
e35f56bb03304ab Frank Li 2022-02-22 568 if (!mw_addr) {
e35f56bb03304ab Frank Li 2022-02-22 569 dev_err(dev, "Failed to allocate OB address\n");
e35f56bb03304ab Frank Li 2022-02-22 570 return -ENOMEM;
e35f56bb03304ab Frank Li 2022-02-22 571 }
d98704aefa5b578 Frank Li 2022-08-22 572 }
e35f56bb03304ab Frank Li 2022-02-22 573
e35f56bb03304ab Frank Li 2022-02-22 574 ntb->epf_db = mw_addr;
e35f56bb03304ab Frank Li 2022-02-22 575
e35f56bb03304ab Frank Li 2022-02-22 576 ret = pci_epc_set_bar(ntb->epf->epc, ntb->epf->func_no, ntb->epf->vfunc_no, epf_bar);
e35f56bb03304ab Frank Li 2022-02-22 577 if (ret) {
e35f56bb03304ab Frank Li 2022-02-22 578 dev_err(dev, "Doorbell BAR set failed\n");
e35f56bb03304ab Frank Li 2022-02-22 579 goto err_alloc_peer_mem;
e35f56bb03304ab Frank Li 2022-02-22 580 }
e35f56bb03304ab Frank Li 2022-02-22 581 return ret;
e35f56bb03304ab Frank Li 2022-02-22 582
e35f56bb03304ab Frank Li 2022-02-22 583 err_alloc_peer_mem:
e35f56bb03304ab Frank Li 2022-02-22 584 pci_epc_mem_free_addr(ntb->epf->epc, epf_bar->phys_addr, mw_addr, epf_bar->size);
e35f56bb03304ab Frank Li 2022-02-22 585 return -1;
e35f56bb03304ab Frank Li 2022-02-22 586 }
e35f56bb03304ab Frank Li 2022-02-22 587
diff --git a/drivers/pci/endpoint/functions/pci-epf-vntb.c b/drivers/pci/endpoint/functions/pci-epf-vntb.c index 1466dd1904175..f6e0b682da000 100644 --- a/drivers/pci/endpoint/functions/pci-epf-vntb.c +++ b/drivers/pci/endpoint/functions/pci-epf-vntb.c @@ -44,6 +44,7 @@ #include <linux/pci-epc.h> #include <linux/pci-epf.h> #include <linux/ntb.h> +#include <linux/msi.h> static struct workqueue_struct *kpcintb_workqueue; @@ -143,6 +144,8 @@ struct epf_ntb { void __iomem *vpci_mw_addr[MAX_MW]; struct delayed_work cmd_handler; + + int msi_virqbase; }; #define to_epf_ntb(epf_group) container_of((epf_group), struct epf_ntb, group) @@ -253,13 +256,15 @@ static void epf_ntb_cmd_handler(struct work_struct *work) ntb = container_of(work, struct epf_ntb, cmd_handler.work); - for (i = 1; i < ntb->db_count; i++) { - if (readl(ntb->epf_db + i * 4)) { - if (readl(ntb->epf_db + i * 4)) - ntb->db |= 1 << (i - 1); + if (!ntb->epf_db_phy) { + for (i = 1; i < ntb->db_count; i++) { + if (readl(ntb->epf_db + i * 4)) { + if (readl(ntb->epf_db + i * 4)) + ntb->db |= 1 << (i - 1); - ntb_db_event(&ntb->ntb, i); - writel(0, ntb->epf_db + i * 4); + ntb_db_event(&ntb->ntb, i); + writel(0, ntb->epf_db + i * 4); + } } } @@ -454,11 +459,9 @@ static int epf_ntb_config_spad_bar_alloc(struct epf_ntb *ntb) ctrl->num_mws = ntb->num_mws; ntb->spad_size = spad_size; - ctrl->db_entry_size = 4; - for (i = 0; i < ntb->db_count; i++) { ntb->reg->db_data[i] = 1 + i; - ntb->reg->db_offset[i] = 0; + ntb->reg->db_offset[i] = 4 * i; } return 0; @@ -509,6 +512,28 @@ static int epf_ntb_configure_interrupt(struct epf_ntb *ntb) return 0; } +static int epf_ntb_db_size(struct epf_ntb *ntb) +{ + const struct pci_epc_features *epc_features; + size_t size = 4 * ntb->db_count; + u32 align; + + epc_features = pci_epc_get_features(ntb->epf->epc, + ntb->epf->func_no, + ntb->epf->vfunc_no); + align = epc_features->align; + + if (size < 128) + size = 128; + + if (align) + size = ALIGN(size, align); + else + size = roundup_pow_of_two(size); + + return size; +} + /** * epf_ntb_db_bar_init() - Configure Doorbell window BARs * @ntb: NTB device that facilitates communication between HOST and vHOST @@ -522,33 +547,32 @@ static int epf_ntb_db_bar_init(struct epf_ntb *ntb) struct pci_epf_bar *epf_bar; void __iomem *mw_addr; enum pci_barno barno; - size_t size = 4 * ntb->db_count; + size_t size; epc_features = pci_epc_get_features(ntb->epf->epc, ntb->epf->func_no, ntb->epf->vfunc_no); align = epc_features->align; - - if (size < 128) - size = 128; - - if (align) - size = ALIGN(size, align); - else - size = roundup_pow_of_two(size); + size = epf_ntb_db_size(ntb); barno = ntb->epf_ntb_bar[BAR_DB]; + epf_bar = &ntb->epf->bar[barno]; - mw_addr = pci_epf_alloc_space(ntb->epf, size, barno, align, 0); - if (!mw_addr) { - dev_err(dev, "Failed to allocate OB address\n"); - return -ENOMEM; + if (ntb->epf_db_phy) { + mw_addr = NULL; + epf_bar->phys_addr = ntb->epf_db_phy; + epf_bar->barno = barno; + epf_bar->size = size; + } else { + mw_addr = pci_epf_alloc_space(ntb->epf, size, barno, align, 0); + if (!mw_addr) { + dev_err(dev, "Failed to allocate OB address\n"); + return -ENOMEM; + } } ntb->epf_db = mw_addr; - epf_bar = &ntb->epf->bar[barno]; - ret = pci_epc_set_bar(ntb->epf->epc, ntb->epf->func_no, ntb->epf->vfunc_no, epf_bar); if (ret) { dev_err(dev, "Doorbell BAR set failed\n"); @@ -704,6 +728,77 @@ static int epf_ntb_init_epc_bar(struct epf_ntb *ntb) return 0; } +static void epf_ntb_write_msi_msg(struct msi_desc *desc, struct msi_msg *msg) +{ + struct epf_ntb *ntb = dev_get_drvdata(desc->dev); + struct epf_ntb_ctrl *reg = ntb->reg; + int size = epf_ntb_db_size(ntb); + u64 addr; + + addr = msg->address_hi; + addr <<= 32; + addr |= msg->address_lo; + + reg->db_data[desc->msi_index] = msg->data; + + if (desc->msi_index == 0) + ntb->epf_db_phy = round_down(addr, size); + + reg->db_offset[desc->msi_index] = addr - ntb->epf_db_phy; +} + +static irqreturn_t epf_ntb_interrupt_handler(int irq, void *data) +{ + struct epf_ntb *ntb = data; + int index; + + index = irq - ntb->msi_virqbase; + ntb->db |= 1 << (index - 1); + ntb_db_event(&ntb->ntb, index); + + return IRQ_HANDLED; +} + +static void epf_ntb_epc_msi_init(struct epf_ntb *ntb) +{ + struct device *dev = &ntb->epf->dev; + struct irq_domain *domain; + int virq; + int ret; + int i; + + domain = dev_get_msi_domain(ntb->epf->epc->dev.parent); + if (!domain) + return; + + dev_set_msi_domain(dev, domain); + + if (platform_msi_domain_alloc_irqs(&ntb->epf->dev, + ntb->db_count, + epf_ntb_write_msi_msg)) { + dev_info(dev, "Can't allocate MSI, fall back to poll mode\n"); + return; + } + + dev_info(dev, "vntb use MSI as doorbell\n"); + + for (i = 0; i < ntb->db_count; i++) { + virq = msi_get_virq(dev, i); + ret = devm_request_irq(dev, virq, + epf_ntb_interrupt_handler, 0, + "ntb", ntb); + + if (ret) { + dev_err(dev, "devm_request_irq() failure, fall back to poll mode\n"); + ntb->epf_db_phy = 0; + break; + } + + if (!i) + ntb->msi_virqbase = virq; + } +} + /** * epf_ntb_epc_init() - Initialize NTB interface * @ntb: NTB device that facilitates communication between HOST and vHOST2 @@ -1299,14 +1394,15 @@ static int epf_ntb_bind(struct pci_epf *epf) goto err_bar_alloc; } + epf_set_drvdata(epf, ntb); + epf_ntb_epc_msi_init(ntb); + ret = epf_ntb_epc_init(ntb); if (ret) { dev_err(dev, "Failed to initialize EPC\n"); goto err_bar_alloc; } - epf_set_drvdata(epf, ntb); - pci_space[0] = (ntb->vntb_pid << 16) | ntb->vntb_vid; pci_vntb_table[0].vendor = ntb->vntb_vid; pci_vntb_table[0].device = ntb->vntb_pid;
┌───────┐ ┌──────────┐ │ │ │ │ ┌─────────────┐ │ │ │ PCI Host │ │ MSI │◄┐ │ │ │ │ │ Controller │ │ │ │ MSI Write │ │ └─────────────┘ └─┼───────┼───────────┼─BAR0 │ │ PCI │ │ BAR1 │ │ EP │ │ BAR2 │ │ │ │ BAR3 │ │ │ │ BAR4 │ │ ├──────────►│ │ └───────┘ └──────────┘ Linux supports endpoint functions. PCI RC write BAR<n> space like write to memory. The EP side can't know memory changed by the host driver. PCI Spec has not defined a standard method to do that. Only define MSI(x) to let EP notified RC status change. The basic idea is to trigger an IRQ when PCI RC writes to a memory address. That's what MSI controller provided. EP drivers just need to request a platform MSI interrupt, struct msi_msg *msg will pass down a memory address and data. EP driver will map such memory address to one of PCI BAR<n>. PCI RC writes to such memory to trigger EP side IRQ. Add MSI support for pci-epf-vntb. Query if system has an MSI controller. Set up doorbell address according to struct msi_msg. So PCI RC can write this doorbell address to trigger EP side's IRQ. If no MSI controller exists, fall back to software polling. Signed-off-by: Frank Li <Frank.Li@nxp.com> --- drivers/pci/endpoint/functions/pci-epf-vntb.c | 148 +++++++++++++++--- 1 file changed, 122 insertions(+), 26 deletions(-)