| Message ID | 20200211121135.30064-11-ming.lei@redhat.com (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | scsi: tracking device queue depth via sbitmap | expand |
Hi Ming,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on scsi/for-next]
[also build test ERROR on mkp-scsi/for-next block/for-next v5.6-rc1 next-20200213]
[if your patch is applied to the wrong git tree, please drop us a note to help
improve the system. BTW, we also suggest to use '--base' option to specify the
base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/Ming-Lei/scsi-tracking-device-queue-depth-via-sbitmap/20200213-215727
base: https://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi.git for-next
config: i386-allyesconfig (attached as .config)
compiler: gcc-7 (Debian 7.5.0-4) 7.5.0
reproduce:
# save the attached .config to linux build tree
make ARCH=i386
If you fix the issue, kindly add following tag
Reported-by: kbuild test robot <lkp@intel.com>
All error/warnings (new ones prefixed by >>):
In file included from drivers/message/fusion/mptbase.h:839:0,
from drivers/message/fusion/mptsas.c:63:
drivers/message/fusion/mptsas.c: In function 'mptsas_send_link_status_event':
>> drivers/message/fusion/mptsas.c:3759:26: error: 'struct scsi_device' has no member named 'device_busy'; did you mean 'device_blocked'?
atomic_read(&sdev->device_busy)));
^
drivers/message/fusion/mptdebug.h:72:3: note: in definition of macro 'MPT_CHECK_LOGGING'
CMD; \
^~~
>> drivers/message/fusion/mptsas.c:3755:7: note: in expansion of macro 'devtprintk'
devtprintk(ioc,
^~~~~~~~~~
vim +3759 drivers/message/fusion/mptsas.c
e6b2d76a49f0ee4 Moore, Eric 2006-03-14 3675
0c33b27deb93178 Christoph Hellwig 2005-09-09 3676 static void
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3677 mptsas_send_link_status_event(struct fw_event_work *fw_event)
0c33b27deb93178 Christoph Hellwig 2005-09-09 3678 {
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3679 MPT_ADAPTER *ioc;
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3680 MpiEventDataSasPhyLinkStatus_t *link_data;
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3681 struct mptsas_portinfo *port_info;
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3682 struct mptsas_phyinfo *phy_info = NULL;
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3683 __le64 sas_address;
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3684 u8 phy_num;
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3685 u8 link_rate;
0c33b27deb93178 Christoph Hellwig 2005-09-09 3686
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3687 ioc = fw_event->ioc;
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3688 link_data = (MpiEventDataSasPhyLinkStatus_t *)fw_event->event_data;
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3689
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3690 memcpy(&sas_address, &link_data->SASAddress, sizeof(__le64));
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3691 sas_address = le64_to_cpu(sas_address);
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3692 link_rate = link_data->LinkRates >> 4;
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3693 phy_num = link_data->PhyNum;
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3694
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3695 port_info = mptsas_find_portinfo_by_sas_address(ioc, sas_address);
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3696 if (port_info) {
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3697 phy_info = &port_info->phy_info[phy_num];
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3698 if (phy_info)
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3699 phy_info->negotiated_link_rate = link_rate;
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3700 }
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3701
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3702 if (link_rate == MPI_SAS_IOUNIT0_RATE_1_5 ||
d75733d51fdab5c Kashyap, Desai 2011-02-10 3703 link_rate == MPI_SAS_IOUNIT0_RATE_3_0 ||
d75733d51fdab5c Kashyap, Desai 2011-02-10 3704 link_rate == MPI_SAS_IOUNIT0_RATE_6_0) {
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3705
eedf92b99806aef Kashyap, Desai 2009-05-29 3706 if (!port_info) {
eedf92b99806aef Kashyap, Desai 2009-05-29 3707 if (ioc->old_sas_discovery_protocal) {
eedf92b99806aef Kashyap, Desai 2009-05-29 3708 port_info = mptsas_expander_add(ioc,
eedf92b99806aef Kashyap, Desai 2009-05-29 3709 le16_to_cpu(link_data->DevHandle));
eedf92b99806aef Kashyap, Desai 2009-05-29 3710 if (port_info)
eedf92b99806aef Kashyap, Desai 2009-05-29 3711 goto out;
eedf92b99806aef Kashyap, Desai 2009-05-29 3712 }
f44e5461d919a34 Moore, Eric 2006-03-14 3713 goto out;
eedf92b99806aef Kashyap, Desai 2009-05-29 3714 }
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3715
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3716 if (port_info == ioc->hba_port_info)
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3717 mptsas_probe_hba_phys(ioc);
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3718 else
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3719 mptsas_expander_refresh(ioc, port_info);
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3720 } else if (phy_info && phy_info->phy) {
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3721 if (link_rate == MPI_SAS_IOUNIT0_RATE_PHY_DISABLED)
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3722 phy_info->phy->negotiated_linkrate =
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3723 SAS_PHY_DISABLED;
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3724 else if (link_rate ==
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3725 MPI_SAS_IOUNIT0_RATE_FAILED_SPEED_NEGOTIATION)
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3726 phy_info->phy->negotiated_linkrate =
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3727 SAS_LINK_RATE_FAILED;
c9de7dc48307395 Kashyap, Desai 2010-07-26 3728 else {
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3729 phy_info->phy->negotiated_linkrate =
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3730 SAS_LINK_RATE_UNKNOWN;
c9de7dc48307395 Kashyap, Desai 2010-07-26 3731 if (ioc->device_missing_delay &&
c9de7dc48307395 Kashyap, Desai 2010-07-26 3732 mptsas_is_end_device(&phy_info->attached)) {
c9de7dc48307395 Kashyap, Desai 2010-07-26 3733 struct scsi_device *sdev;
c9de7dc48307395 Kashyap, Desai 2010-07-26 3734 VirtDevice *vdevice;
c9de7dc48307395 Kashyap, Desai 2010-07-26 3735 u8 channel, id;
c9de7dc48307395 Kashyap, Desai 2010-07-26 3736 id = phy_info->attached.id;
c9de7dc48307395 Kashyap, Desai 2010-07-26 3737 channel = phy_info->attached.channel;
c9de7dc48307395 Kashyap, Desai 2010-07-26 3738 devtprintk(ioc, printk(MYIOC_s_DEBUG_FMT
c9de7dc48307395 Kashyap, Desai 2010-07-26 3739 "Link down for fw_id %d:fw_channel %d\n",
c9de7dc48307395 Kashyap, Desai 2010-07-26 3740 ioc->name, phy_info->attached.id,
c9de7dc48307395 Kashyap, Desai 2010-07-26 3741 phy_info->attached.channel));
c9de7dc48307395 Kashyap, Desai 2010-07-26 3742
c9de7dc48307395 Kashyap, Desai 2010-07-26 3743 shost_for_each_device(sdev, ioc->sh) {
c9de7dc48307395 Kashyap, Desai 2010-07-26 3744 vdevice = sdev->hostdata;
c9de7dc48307395 Kashyap, Desai 2010-07-26 3745 if ((vdevice == NULL) ||
c9de7dc48307395 Kashyap, Desai 2010-07-26 3746 (vdevice->vtarget == NULL))
c9de7dc48307395 Kashyap, Desai 2010-07-26 3747 continue;
c9de7dc48307395 Kashyap, Desai 2010-07-26 3748 if ((vdevice->vtarget->tflags &
c9de7dc48307395 Kashyap, Desai 2010-07-26 3749 MPT_TARGET_FLAGS_RAID_COMPONENT ||
c9de7dc48307395 Kashyap, Desai 2010-07-26 3750 vdevice->vtarget->raidVolume))
c9de7dc48307395 Kashyap, Desai 2010-07-26 3751 continue;
c9de7dc48307395 Kashyap, Desai 2010-07-26 3752 if (vdevice->vtarget->id == id &&
c9de7dc48307395 Kashyap, Desai 2010-07-26 3753 vdevice->vtarget->channel ==
c9de7dc48307395 Kashyap, Desai 2010-07-26 3754 channel)
c9de7dc48307395 Kashyap, Desai 2010-07-26 @3755 devtprintk(ioc,
c9de7dc48307395 Kashyap, Desai 2010-07-26 3756 printk(MYIOC_s_DEBUG_FMT
c9de7dc48307395 Kashyap, Desai 2010-07-26 3757 "SDEV OUTSTANDING CMDS"
c9de7dc48307395 Kashyap, Desai 2010-07-26 3758 "%d\n", ioc->name,
71e75c97f97a964 Christoph Hellwig 2014-04-11 @3759 atomic_read(&sdev->device_busy)));
c9de7dc48307395 Kashyap, Desai 2010-07-26 3760 }
c9de7dc48307395 Kashyap, Desai 2010-07-26 3761
c9de7dc48307395 Kashyap, Desai 2010-07-26 3762 }
c9de7dc48307395 Kashyap, Desai 2010-07-26 3763 }
f44e5461d919a34 Moore, Eric 2006-03-14 3764 }
f44e5461d919a34 Moore, Eric 2006-03-14 3765 out:
f9c34022eae9c76 Kashyap, Desai 2009-05-29 3766 mptsas_free_fw_event(ioc, fw_event);
e6b2d76a49f0ee4 Moore, Eric 2006-03-14 3767 }
e6b2d76a49f0ee4 Moore, Eric 2006-03-14 3768
:::::: The code at line 3759 was first introduced by commit
:::::: 71e75c97f97a9645d25fbf3d8e4165a558f18747 scsi: convert device_busy to atomic_t
:::::: TO: Christoph Hellwig <hch@lst.de>
:::::: CC: Christoph Hellwig <hch@lst.de>
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org
Hi Ming,
Thank you for the patch! Perhaps something to improve:
[auto build test WARNING on scsi/for-next]
[also build test WARNING on mkp-scsi/for-next block/for-next v5.6-rc1 next-20200213]
[if your patch is applied to the wrong git tree, please drop us a note to help
improve the system. BTW, we also suggest to use '--base' option to specify the
base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/Ming-Lei/scsi-tracking-device-queue-depth-via-sbitmap/20200213-215727
base: https://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi.git for-next
config: arm-allmodconfig (attached as .config)
compiler: arm-linux-gnueabi-gcc (GCC) 7.5.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=7.5.0 make.cross ARCH=arm
If you fix the issue, kindly add following tag
Reported-by: kbuild test robot <lkp@intel.com>
All warnings (new ones prefixed by >>):
In file included from drivers/message/fusion/mptbase.h:839:0,
from drivers/message/fusion/mptsas.c:63:
drivers/message/fusion/mptsas.c: In function 'mptsas_send_link_status_event':
drivers/message/fusion/mptsas.c:3759:26: error: 'struct scsi_device' has no member named 'device_busy'; did you mean 'device_blocked'?
atomic_read(&sdev->device_busy)));
^
drivers/message/fusion/mptdebug.h:72:3: note: in definition of macro 'MPT_CHECK_LOGGING'
CMD; \
^~~
drivers/message/fusion/mptsas.c:3755:7: note: in expansion of macro 'devtprintk'
devtprintk(ioc,
^~~~~~~~~~
include/linux/compiler.h:269:22: note: in expansion of macro '__READ_ONCE'
#define READ_ONCE(x) __READ_ONCE(x, 1)
^~~~~~~~~~~
>> arch/arm/include/asm/atomic.h:27:24: note: in expansion of macro 'READ_ONCE'
#define atomic_read(v) READ_ONCE((v)->counter)
^~~~~~~~~
drivers/message/fusion/mptsas.c:3759:7: note: in expansion of macro 'atomic_read'
atomic_read(&sdev->device_busy)));
^~~~~~~~~~~
drivers/message/fusion/mptsas.c:3759:26: error: 'struct scsi_device' has no member named 'device_busy'; did you mean 'device_blocked'?
atomic_read(&sdev->device_busy)));
^
drivers/message/fusion/mptdebug.h:72:3: note: in definition of macro 'MPT_CHECK_LOGGING'
CMD; \
^~~
drivers/message/fusion/mptsas.c:3755:7: note: in expansion of macro 'devtprintk'
devtprintk(ioc,
^~~~~~~~~~
include/linux/compiler.h:269:22: note: in expansion of macro '__READ_ONCE'
#define READ_ONCE(x) __READ_ONCE(x, 1)
^~~~~~~~~~~
>> arch/arm/include/asm/atomic.h:27:24: note: in expansion of macro 'READ_ONCE'
#define atomic_read(v) READ_ONCE((v)->counter)
^~~~~~~~~
drivers/message/fusion/mptsas.c:3759:7: note: in expansion of macro 'atomic_read'
atomic_read(&sdev->device_busy)));
^~~~~~~~~~~
drivers/message/fusion/mptsas.c:3759:26: error: 'struct scsi_device' has no member named 'device_busy'; did you mean 'device_blocked'?
atomic_read(&sdev->device_busy)));
^
drivers/message/fusion/mptdebug.h:72:3: note: in definition of macro 'MPT_CHECK_LOGGING'
CMD; \
^~~
drivers/message/fusion/mptsas.c:3755:7: note: in expansion of macro 'devtprintk'
devtprintk(ioc,
^~~~~~~~~~
include/linux/compiler.h:269:22: note: in expansion of macro '__READ_ONCE'
#define READ_ONCE(x) __READ_ONCE(x, 1)
^~~~~~~~~~~
>> arch/arm/include/asm/atomic.h:27:24: note: in expansion of macro 'READ_ONCE'
#define atomic_read(v) READ_ONCE((v)->counter)
^~~~~~~~~
drivers/message/fusion/mptsas.c:3759:7: note: in expansion of macro 'atomic_read'
atomic_read(&sdev->device_busy)));
^~~~~~~~~~~
drivers/message/fusion/mptsas.c:3759:26: error: 'struct scsi_device' has no member named 'device_busy'; did you mean 'device_blocked'?
atomic_read(&sdev->device_busy)));
^
drivers/message/fusion/mptdebug.h:72:3: note: in definition of macro 'MPT_CHECK_LOGGING'
CMD; \
^~~
drivers/message/fusion/mptsas.c:3755:7: note: in expansion of macro 'devtprintk'
devtprintk(ioc,
^~~~~~~~~~
include/linux/compiler.h:269:22: note: in expansion of macro '__READ_ONCE'
#define READ_ONCE(x) __READ_ONCE(x, 1)
^~~~~~~~~~~
>> arch/arm/include/asm/atomic.h:27:24: note: in expansion of macro 'READ_ONCE'
#define atomic_read(v) READ_ONCE((v)->counter)
^~~~~~~~~
drivers/message/fusion/mptsas.c:3759:7: note: in expansion of macro 'atomic_read'
atomic_read(&sdev->device_busy)));
^~~~~~~~~~~
drivers/message/fusion/mptsas.c:3759:26: error: 'struct scsi_device' has no member named 'device_busy'; did you mean 'device_blocked'?
atomic_read(&sdev->device_busy)));
^
drivers/message/fusion/mptdebug.h:72:3: note: in definition of macro 'MPT_CHECK_LOGGING'
CMD; \
^~~
drivers/message/fusion/mptsas.c:3755:7: note: in expansion of macro 'devtprintk'
devtprintk(ioc,
^~~~~~~~~~
include/linux/compiler.h:269:22: note: in expansion of macro '__READ_ONCE'
#define READ_ONCE(x) __READ_ONCE(x, 1)
^~~~~~~~~~~
>> arch/arm/include/asm/atomic.h:27:24: note: in expansion of macro 'READ_ONCE'
#define atomic_read(v) READ_ONCE((v)->counter)
^~~~~~~~~
drivers/message/fusion/mptsas.c:3759:7: note: in expansion of macro 'atomic_read'
atomic_read(&sdev->device_busy)));
^~~~~~~~~~~
--
In file included from drivers/message//fusion/mptbase.h:839:0,
from drivers/message//fusion/mptsas.c:63:
drivers/message//fusion/mptsas.c: In function 'mptsas_send_link_status_event':
drivers/message//fusion/mptsas.c:3759:26: error: 'struct scsi_device' has no member named 'device_busy'; did you mean 'device_blocked'?
atomic_read(&sdev->device_busy)));
^
drivers/message//fusion/mptdebug.h:72:3: note: in definition of macro 'MPT_CHECK_LOGGING'
CMD; \
^~~
drivers/message//fusion/mptsas.c:3755:7: note: in expansion of macro 'devtprintk'
devtprintk(ioc,
^~~~~~~~~~
include/linux/compiler.h:269:22: note: in expansion of macro '__READ_ONCE'
#define READ_ONCE(x) __READ_ONCE(x, 1)
^~~~~~~~~~~
>> arch/arm/include/asm/atomic.h:27:24: note: in expansion of macro 'READ_ONCE'
#define atomic_read(v) READ_ONCE((v)->counter)
^~~~~~~~~
drivers/message//fusion/mptsas.c:3759:7: note: in expansion of macro 'atomic_read'
atomic_read(&sdev->device_busy)));
^~~~~~~~~~~
drivers/message//fusion/mptsas.c:3759:26: error: 'struct scsi_device' has no member named 'device_busy'; did you mean 'device_blocked'?
atomic_read(&sdev->device_busy)));
^
drivers/message//fusion/mptdebug.h:72:3: note: in definition of macro 'MPT_CHECK_LOGGING'
CMD; \
^~~
drivers/message//fusion/mptsas.c:3755:7: note: in expansion of macro 'devtprintk'
devtprintk(ioc,
^~~~~~~~~~
include/linux/compiler.h:269:22: note: in expansion of macro '__READ_ONCE'
#define READ_ONCE(x) __READ_ONCE(x, 1)
^~~~~~~~~~~
>> arch/arm/include/asm/atomic.h:27:24: note: in expansion of macro 'READ_ONCE'
#define atomic_read(v) READ_ONCE((v)->counter)
^~~~~~~~~
drivers/message//fusion/mptsas.c:3759:7: note: in expansion of macro 'atomic_read'
atomic_read(&sdev->device_busy)));
^~~~~~~~~~~
drivers/message//fusion/mptsas.c:3759:26: error: 'struct scsi_device' has no member named 'device_busy'; did you mean 'device_blocked'?
atomic_read(&sdev->device_busy)));
^
drivers/message//fusion/mptdebug.h:72:3: note: in definition of macro 'MPT_CHECK_LOGGING'
CMD; \
^~~
drivers/message//fusion/mptsas.c:3755:7: note: in expansion of macro 'devtprintk'
devtprintk(ioc,
^~~~~~~~~~
include/linux/compiler.h:269:22: note: in expansion of macro '__READ_ONCE'
#define READ_ONCE(x) __READ_ONCE(x, 1)
^~~~~~~~~~~
>> arch/arm/include/asm/atomic.h:27:24: note: in expansion of macro 'READ_ONCE'
#define atomic_read(v) READ_ONCE((v)->counter)
^~~~~~~~~
drivers/message//fusion/mptsas.c:3759:7: note: in expansion of macro 'atomic_read'
atomic_read(&sdev->device_busy)));
^~~~~~~~~~~
drivers/message//fusion/mptsas.c:3759:26: error: 'struct scsi_device' has no member named 'device_busy'; did you mean 'device_blocked'?
atomic_read(&sdev->device_busy)));
^
drivers/message//fusion/mptdebug.h:72:3: note: in definition of macro 'MPT_CHECK_LOGGING'
CMD; \
^~~
drivers/message//fusion/mptsas.c:3755:7: note: in expansion of macro 'devtprintk'
devtprintk(ioc,
^~~~~~~~~~
include/linux/compiler.h:269:22: note: in expansion of macro '__READ_ONCE'
#define READ_ONCE(x) __READ_ONCE(x, 1)
^~~~~~~~~~~
>> arch/arm/include/asm/atomic.h:27:24: note: in expansion of macro 'READ_ONCE'
#define atomic_read(v) READ_ONCE((v)->counter)
^~~~~~~~~
drivers/message//fusion/mptsas.c:3759:7: note: in expansion of macro 'atomic_read'
atomic_read(&sdev->device_busy)));
^~~~~~~~~~~
drivers/message//fusion/mptsas.c:3759:26: error: 'struct scsi_device' has no member named 'device_busy'; did you mean 'device_blocked'?
atomic_read(&sdev->device_busy)));
^
drivers/message//fusion/mptdebug.h:72:3: note: in definition of macro 'MPT_CHECK_LOGGING'
CMD; \
^~~
drivers/message//fusion/mptsas.c:3755:7: note: in expansion of macro 'devtprintk'
devtprintk(ioc,
^~~~~~~~~~
include/linux/compiler.h:269:22: note: in expansion of macro '__READ_ONCE'
#define READ_ONCE(x) __READ_ONCE(x, 1)
^~~~~~~~~~~
>> arch/arm/include/asm/atomic.h:27:24: note: in expansion of macro 'READ_ONCE'
#define atomic_read(v) READ_ONCE((v)->counter)
^~~~~~~~~
drivers/message//fusion/mptsas.c:3759:7: note: in expansion of macro 'atomic_read'
atomic_read(&sdev->device_busy)));
^~~~~~~~~~~
vim +/READ_ONCE +27 arch/arm/include/asm/atomic.h
^1da177e4c3f415 include/asm-arm/atomic.h Linus Torvalds 2005-04-16 21
200b812d0084f80 arch/arm/include/asm/atomic.h Catalin Marinas 2009-09-18 22 /*
200b812d0084f80 arch/arm/include/asm/atomic.h Catalin Marinas 2009-09-18 23 * On ARM, ordinary assignment (str instruction) doesn't clear the local
200b812d0084f80 arch/arm/include/asm/atomic.h Catalin Marinas 2009-09-18 24 * strex/ldrex monitor on some implementations. The reason we can use it for
200b812d0084f80 arch/arm/include/asm/atomic.h Catalin Marinas 2009-09-18 25 * atomic_set() is the clrex or dummy strex done on every exception return.
200b812d0084f80 arch/arm/include/asm/atomic.h Catalin Marinas 2009-09-18 26 */
62e8a3258bda118 arch/arm/include/asm/atomic.h Peter Zijlstra 2015-09-18 @27 #define atomic_read(v) READ_ONCE((v)->counter)
62e8a3258bda118 arch/arm/include/asm/atomic.h Peter Zijlstra 2015-09-18 28 #define atomic_set(v,i) WRITE_ONCE(((v)->counter), (i))
^1da177e4c3f415 include/asm-arm/atomic.h Linus Torvalds 2005-04-16 29
:::::: The code at line 27 was first introduced by commit
:::::: 62e8a3258bda118f24ff462fe04cfbe75b8189b5 atomic, arch: Audit atomic_{read,set}()
:::::: TO: Peter Zijlstra <peterz@infradead.org>
:::::: CC: Ingo Molnar <mingo@kernel.org>
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org
On Fri, Feb 14, 2020 at 04:22:08AM +0800, kbuild test robot wrote: > Hi Ming, > > Thank you for the patch! Perhaps something to improve: > > [auto build test WARNING on scsi/for-next] > [also build test WARNING on mkp-scsi/for-next block/for-next v5.6-rc1 next-20200213] > [if your patch is applied to the wrong git tree, please drop us a note to help > improve the system. BTW, we also suggest to use '--base' option to specify the > base tree in git format-patch, please see https://stackoverflow.com/a/37406982] > > url: https://github.com/0day-ci/linux/commits/Ming-Lei/scsi-tracking-device-queue-depth-via-sbitmap/20200213-215727 > base: https://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi.git for-next > config: arm-allmodconfig (attached as .config) > compiler: arm-linux-gnueabi-gcc (GCC) 7.5.0 > reproduce: > wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross > chmod +x ~/bin/make.cross > # save the attached .config to linux build tree > GCC_VERSION=7.5.0 make.cross ARCH=arm > > If you fix the issue, kindly add following tag > Reported-by: kbuild test robot <lkp@intel.com> > > All warnings (new ones prefixed by >>): > > In file included from drivers/message/fusion/mptbase.h:839:0, > from drivers/message/fusion/mptsas.c:63: > drivers/message/fusion/mptsas.c: In function 'mptsas_send_link_status_event': > drivers/message/fusion/mptsas.c:3759:26: error: 'struct scsi_device' has no member named 'device_busy'; did you mean 'device_blocked'? > atomic_read(&sdev->device_busy))); > ^ > drivers/message/fusion/mptdebug.h:72:3: note: in definition of macro 'MPT_CHECK_LOGGING' > CMD; \ > ^~~ > drivers/message/fusion/mptsas.c:3755:7: note: in expansion of macro 'devtprintk' > devtprintk(ioc, > ^~~~~~~~~~ > include/linux/compiler.h:269:22: note: in expansion of macro '__READ_ONCE' > #define READ_ONCE(x) __READ_ONCE(x, 1) > ^~~~~~~~~~~ > >> arch/arm/include/asm/atomic.h:27:24: note: in expansion of macro 'READ_ONCE' > #define atomic_read(v) READ_ONCE((v)->counter) > ^~~~~~~~~ > drivers/message/fusion/mptsas.c:3759:7: note: in expansion of macro 'atomic_read' > atomic_read(&sdev->device_busy))); > ^~~~~~~~~~~ > drivers/message/fusion/mptsas.c:3759:26: error: 'struct scsi_device' has no member named 'device_busy'; did you mean 'device_blocked'? > atomic_read(&sdev->device_busy))); > ^ > drivers/message/fusion/mptdebug.h:72:3: note: in definition of macro 'MPT_CHECK_LOGGING' > CMD; \ > ^~~ > drivers/message/fusion/mptsas.c:3755:7: note: in expansion of macro 'devtprintk' > devtprintk(ioc, > ^~~~~~~~~~ > include/linux/compiler.h:269:22: note: in expansion of macro '__READ_ONCE' > #define READ_ONCE(x) __READ_ONCE(x, 1) > ^~~~~~~~~~~ > >> arch/arm/include/asm/atomic.h:27:24: note: in expansion of macro 'READ_ONCE' > #define atomic_read(v) READ_ONCE((v)->counter) > ^~~~~~~~~ > drivers/message/fusion/mptsas.c:3759:7: note: in expansion of macro 'atomic_read' > atomic_read(&sdev->device_busy))); > ^~~~~~~~~~~ > drivers/message/fusion/mptsas.c:3759:26: error: 'struct scsi_device' has no member named 'device_busy'; did you mean 'device_blocked'? > atomic_read(&sdev->device_busy))); > ^ > drivers/message/fusion/mptdebug.h:72:3: note: in definition of macro 'MPT_CHECK_LOGGING' > CMD; \ > ^~~ > drivers/message/fusion/mptsas.c:3755:7: note: in expansion of macro 'devtprintk' > devtprintk(ioc, > ^~~~~~~~~~ > include/linux/compiler.h:269:22: note: in expansion of macro '__READ_ONCE' > #define READ_ONCE(x) __READ_ONCE(x, 1) > ^~~~~~~~~~~ > >> arch/arm/include/asm/atomic.h:27:24: note: in expansion of macro 'READ_ONCE' > #define atomic_read(v) READ_ONCE((v)->counter) > ^~~~~~~~~ > drivers/message/fusion/mptsas.c:3759:7: note: in expansion of macro 'atomic_read' > atomic_read(&sdev->device_busy))); > ^~~~~~~~~~~ > drivers/message/fusion/mptsas.c:3759:26: error: 'struct scsi_device' has no member named 'device_busy'; did you mean 'device_blocked'? > atomic_read(&sdev->device_busy))); > ^ > drivers/message/fusion/mptdebug.h:72:3: note: in definition of macro 'MPT_CHECK_LOGGING' > CMD; \ > ^~~ > drivers/message/fusion/mptsas.c:3755:7: note: in expansion of macro 'devtprintk' > devtprintk(ioc, > ^~~~~~~~~~ > include/linux/compiler.h:269:22: note: in expansion of macro '__READ_ONCE' > #define READ_ONCE(x) __READ_ONCE(x, 1) > ^~~~~~~~~~~ > >> arch/arm/include/asm/atomic.h:27:24: note: in expansion of macro 'READ_ONCE' > #define atomic_read(v) READ_ONCE((v)->counter) > ^~~~~~~~~ > drivers/message/fusion/mptsas.c:3759:7: note: in expansion of macro 'atomic_read' > atomic_read(&sdev->device_busy))); > ^~~~~~~~~~~ > drivers/message/fusion/mptsas.c:3759:26: error: 'struct scsi_device' has no member named 'device_busy'; did you mean 'device_blocked'? > atomic_read(&sdev->device_busy))); > ^ > drivers/message/fusion/mptdebug.h:72:3: note: in definition of macro 'MPT_CHECK_LOGGING' > CMD; \ > ^~~ > drivers/message/fusion/mptsas.c:3755:7: note: in expansion of macro 'devtprintk' > devtprintk(ioc, > ^~~~~~~~~~ > include/linux/compiler.h:269:22: note: in expansion of macro '__READ_ONCE' > #define READ_ONCE(x) __READ_ONCE(x, 1) > ^~~~~~~~~~~ > >> arch/arm/include/asm/atomic.h:27:24: note: in expansion of macro 'READ_ONCE' > #define atomic_read(v) READ_ONCE((v)->counter) > ^~~~~~~~~ > drivers/message/fusion/mptsas.c:3759:7: note: in expansion of macro 'atomic_read' > atomic_read(&sdev->device_busy))); > ^~~~~~~~~~~ > -- > In file included from drivers/message//fusion/mptbase.h:839:0, > from drivers/message//fusion/mptsas.c:63: > drivers/message//fusion/mptsas.c: In function 'mptsas_send_link_status_event': > drivers/message//fusion/mptsas.c:3759:26: error: 'struct scsi_device' has no member named 'device_busy'; did you mean 'device_blocked'? > atomic_read(&sdev->device_busy))); > ^ > drivers/message//fusion/mptdebug.h:72:3: note: in definition of macro 'MPT_CHECK_LOGGING' > CMD; \ > ^~~ > drivers/message//fusion/mptsas.c:3755:7: note: in expansion of macro 'devtprintk' > devtprintk(ioc, > ^~~~~~~~~~ > include/linux/compiler.h:269:22: note: in expansion of macro '__READ_ONCE' > #define READ_ONCE(x) __READ_ONCE(x, 1) > ^~~~~~~~~~~ > >> arch/arm/include/asm/atomic.h:27:24: note: in expansion of macro 'READ_ONCE' > #define atomic_read(v) READ_ONCE((v)->counter) > ^~~~~~~~~ > drivers/message//fusion/mptsas.c:3759:7: note: in expansion of macro 'atomic_read' > atomic_read(&sdev->device_busy))); > ^~~~~~~~~~~ > drivers/message//fusion/mptsas.c:3759:26: error: 'struct scsi_device' has no member named 'device_busy'; did you mean 'device_blocked'? > atomic_read(&sdev->device_busy))); > ^ > drivers/message//fusion/mptdebug.h:72:3: note: in definition of macro 'MPT_CHECK_LOGGING' > CMD; \ > ^~~ > drivers/message//fusion/mptsas.c:3755:7: note: in expansion of macro 'devtprintk' > devtprintk(ioc, > ^~~~~~~~~~ > include/linux/compiler.h:269:22: note: in expansion of macro '__READ_ONCE' > #define READ_ONCE(x) __READ_ONCE(x, 1) > ^~~~~~~~~~~ > >> arch/arm/include/asm/atomic.h:27:24: note: in expansion of macro 'READ_ONCE' > #define atomic_read(v) READ_ONCE((v)->counter) > ^~~~~~~~~ > drivers/message//fusion/mptsas.c:3759:7: note: in expansion of macro 'atomic_read' > atomic_read(&sdev->device_busy))); > ^~~~~~~~~~~ > drivers/message//fusion/mptsas.c:3759:26: error: 'struct scsi_device' has no member named 'device_busy'; did you mean 'device_blocked'? > atomic_read(&sdev->device_busy))); > ^ > drivers/message//fusion/mptdebug.h:72:3: note: in definition of macro 'MPT_CHECK_LOGGING' > CMD; \ > ^~~ > drivers/message//fusion/mptsas.c:3755:7: note: in expansion of macro 'devtprintk' > devtprintk(ioc, > ^~~~~~~~~~ > include/linux/compiler.h:269:22: note: in expansion of macro '__READ_ONCE' > #define READ_ONCE(x) __READ_ONCE(x, 1) > ^~~~~~~~~~~ > >> arch/arm/include/asm/atomic.h:27:24: note: in expansion of macro 'READ_ONCE' > #define atomic_read(v) READ_ONCE((v)->counter) > ^~~~~~~~~ > drivers/message//fusion/mptsas.c:3759:7: note: in expansion of macro 'atomic_read' > atomic_read(&sdev->device_busy))); > ^~~~~~~~~~~ > drivers/message//fusion/mptsas.c:3759:26: error: 'struct scsi_device' has no member named 'device_busy'; did you mean 'device_blocked'? > atomic_read(&sdev->device_busy))); > ^ > drivers/message//fusion/mptdebug.h:72:3: note: in definition of macro 'MPT_CHECK_LOGGING' > CMD; \ > ^~~ > drivers/message//fusion/mptsas.c:3755:7: note: in expansion of macro 'devtprintk' > devtprintk(ioc, > ^~~~~~~~~~ > include/linux/compiler.h:269:22: note: in expansion of macro '__READ_ONCE' > #define READ_ONCE(x) __READ_ONCE(x, 1) > ^~~~~~~~~~~ > >> arch/arm/include/asm/atomic.h:27:24: note: in expansion of macro 'READ_ONCE' > #define atomic_read(v) READ_ONCE((v)->counter) > ^~~~~~~~~ > drivers/message//fusion/mptsas.c:3759:7: note: in expansion of macro 'atomic_read' > atomic_read(&sdev->device_busy))); > ^~~~~~~~~~~ > drivers/message//fusion/mptsas.c:3759:26: error: 'struct scsi_device' has no member named 'device_busy'; did you mean 'device_blocked'? > atomic_read(&sdev->device_busy))); > ^ > drivers/message//fusion/mptdebug.h:72:3: note: in definition of macro 'MPT_CHECK_LOGGING' > CMD; \ > ^~~ > drivers/message//fusion/mptsas.c:3755:7: note: in expansion of macro 'devtprintk' > devtprintk(ioc, > ^~~~~~~~~~ > include/linux/compiler.h:269:22: note: in expansion of macro '__READ_ONCE' > #define READ_ONCE(x) __READ_ONCE(x, 1) > ^~~~~~~~~~~ > >> arch/arm/include/asm/atomic.h:27:24: note: in expansion of macro 'READ_ONCE' > #define atomic_read(v) READ_ONCE((v)->counter) > ^~~~~~~~~ > drivers/message//fusion/mptsas.c:3759:7: note: in expansion of macro 'atomic_read' > atomic_read(&sdev->device_busy))); Hello, Thanks for the report. Looks miss this non-scsi directory, will fix it in V2 after getting some feedback. Thank, Ming
diff --git a/drivers/scsi/scsi.c b/drivers/scsi/scsi.c index 930e4803d888..5a97e0967e52 100644 --- a/drivers/scsi/scsi.c +++ b/drivers/scsi/scsi.c @@ -245,6 +245,8 @@ int scsi_change_queue_depth(struct scsi_device *sdev, int depth) if (sdev->request_queue) blk_set_queue_depth(sdev->request_queue, depth); + sbitmap_resize(&sdev->budget_map, sdev->queue_depth); + return sdev->queue_depth; } EXPORT_SYMBOL(scsi_change_queue_depth); diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c index a3474b418602..e7fbf3a9a6aa 100644 --- a/drivers/scsi/scsi_lib.c +++ b/drivers/scsi/scsi_lib.c @@ -354,7 +354,7 @@ void scsi_device_unbusy(struct scsi_device *sdev, struct scsi_cmnd *cmd) if (starget->can_queue > 0) atomic_dec(&starget->target_busy); - atomic_dec(&sdev->device_busy); + sbitmap_put(&sdev->budget_map, cmd->budget_token); } static void scsi_kick_queue(struct request_queue *q) @@ -1274,19 +1274,17 @@ scsi_prep_state_check(struct scsi_device *sdev, struct request *req) } /* - * scsi_dev_queue_ready: if we can send requests to sdev, return 1 else - * return 0. - * - * Called with the queue_lock held. + * scsi_dev_queue_ready: if we can send requests to sdev, assign one token + * and return the token else return -1. */ static inline int scsi_dev_queue_ready(struct request_queue *q, struct scsi_device *sdev) { - unsigned int busy; + int token; - busy = atomic_inc_return(&sdev->device_busy) - 1; + token = sbitmap_get(&sdev->budget_map); if (atomic_read(&sdev->device_blocked)) { - if (busy) + if (token >= 0) goto out_dec; /* @@ -1298,13 +1296,11 @@ static inline int scsi_dev_queue_ready(struct request_queue *q, "unblocking device at zero depth\n")); } - if (busy >= sdev->queue_depth) - goto out_dec; - - return 1; + return token; out_dec: - atomic_dec(&sdev->device_busy); - return 0; + if (token >= 0) + sbitmap_put(&sdev->budget_map, token); + return -1; } /* @@ -1624,16 +1620,17 @@ static void scsi_mq_put_budget(struct blk_mq_hw_ctx *hctx, int budget_token) struct request_queue *q = hctx->queue; struct scsi_device *sdev = q->queuedata; - atomic_dec(&sdev->device_busy); + sbitmap_put(&sdev->budget_map, budget_token); } static int scsi_mq_get_budget(struct blk_mq_hw_ctx *hctx) { struct request_queue *q = hctx->queue; struct scsi_device *sdev = q->queuedata; + int token = scsi_dev_queue_ready(q, sdev); - if (scsi_dev_queue_ready(q, sdev)) - return 0; + if (token >= 0) + return token; if (scsi_device_busy(sdev) == 0 && !scsi_device_blocked(sdev)) blk_mq_delay_run_hw_queue(hctx, SCSI_QUEUE_DELAY); @@ -1677,6 +1674,8 @@ static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx, blk_mq_start_request(req); } + cmd->budget_token = bd->budget_token; + cmd->flags &= SCMD_PRESERVED_FLAGS; if (sdev->simple_tags) cmd->flags |= SCMD_TAGGED; @@ -1701,12 +1700,12 @@ static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx, if (scsi_target(sdev)->can_queue > 0) atomic_dec(&scsi_target(sdev)->target_busy); out_put_budget: - scsi_mq_put_budget(hctx, 0); + scsi_mq_put_budget(hctx, bd->budget_token); switch (ret) { case BLK_STS_OK: break; case BLK_STS_RESOURCE: - if (atomic_read(&sdev->device_busy) || + if (sbitmap_any_bit_set(&sdev->budget_map) || scsi_device_blocked(sdev)) ret = BLK_STS_DEV_RESOURCE; break; diff --git a/drivers/scsi/scsi_priv.h b/drivers/scsi/scsi_priv.h index 3bff9f7aa684..b934686aee32 100644 --- a/drivers/scsi/scsi_priv.h +++ b/drivers/scsi/scsi_priv.h @@ -5,6 +5,7 @@ #include <linux/device.h> #include <linux/async.h> #include <scsi/scsi_device.h> +#include <linux/sbitmap.h> struct request_queue; struct request; diff --git a/drivers/scsi/scsi_scan.c b/drivers/scsi/scsi_scan.c index 058079f915f1..27b64c82d26b 100644 --- a/drivers/scsi/scsi_scan.c +++ b/drivers/scsi/scsi_scan.c @@ -215,6 +215,7 @@ static void scsi_unlock_floptical(struct scsi_device *sdev, static struct scsi_device *scsi_alloc_sdev(struct scsi_target *starget, u64 lun, void *hostdata) { + unsigned int depth; struct scsi_device *sdev; int display_failure_msg = 1, ret; struct Scsi_Host *shost = dev_to_shost(starget->dev.parent); @@ -277,8 +278,23 @@ static struct scsi_device *scsi_alloc_sdev(struct scsi_target *starget, WARN_ON_ONCE(!blk_get_queue(sdev->request_queue)); sdev->request_queue->queuedata = sdev; - scsi_change_queue_depth(sdev, sdev->host->cmd_per_lun ? - sdev->host->cmd_per_lun : 1); + depth = sdev->host->cmd_per_lun ?: 1; + + /* + * Use .can_queue as budget map's depth because we have to + * support adjusting queue depth from sysfs. Meantime use + * default device queue depth to figure out sbitmap shift + * since we use this queue depth most of times. + */ + if (sbitmap_init_node(&sdev->budget_map, sdev->host->can_queue, + sbitmap_calculate_shift(depth), + GFP_KERNEL, sdev->request_queue->node, + false, true)) { + put_device(&starget->dev); + kfree(sdev); + } + + scsi_change_queue_depth(sdev, depth); scsi_sysfs_device_initialize(sdev); @@ -980,6 +996,7 @@ static int scsi_add_lun(struct scsi_device *sdev, unsigned char *inq_result, scsi_attach_vpd(sdev); sdev->max_queue_depth = sdev->queue_depth; + WARN_ON_ONCE(sdev->max_queue_depth > sdev->budget_map.depth); sdev->sdev_bflags = *bflags; /* diff --git a/drivers/scsi/scsi_sysfs.c b/drivers/scsi/scsi_sysfs.c index d6cb5a0a03f2..835566b805fb 100644 --- a/drivers/scsi/scsi_sysfs.c +++ b/drivers/scsi/scsi_sysfs.c @@ -466,6 +466,8 @@ static void scsi_device_dev_release_usercontext(struct work_struct *work) /* NULL queue means the device can't be used */ sdev->request_queue = NULL; + sbitmap_free(&sdev->budget_map); + mutex_lock(&sdev->inquiry_mutex); vpd_pg0 = rcu_replace_pointer(sdev->vpd_pg0, vpd_pg0, lockdep_is_held(&sdev->inquiry_mutex)); diff --git a/include/scsi/scsi_cmnd.h b/include/scsi/scsi_cmnd.h index a2849bb9cd19..e6f750f43889 100644 --- a/include/scsi/scsi_cmnd.h +++ b/include/scsi/scsi_cmnd.h @@ -76,6 +76,8 @@ struct scsi_cmnd { int eh_eflags; /* Used by error handlr */ + int budget_token; + /* * This is set to jiffies as it was when the command was first * allocated. It is used to time how long the command has diff --git a/include/scsi/scsi_device.h b/include/scsi/scsi_device.h index 09caf6f2f528..cdc1f28ec57f 100644 --- a/include/scsi/scsi_device.h +++ b/include/scsi/scsi_device.h @@ -8,6 +8,7 @@ #include <linux/blkdev.h> #include <scsi/scsi.h> #include <linux/atomic.h> +#include <linux/sbitmap.h> struct device; struct request_queue; @@ -106,7 +107,7 @@ struct scsi_device { struct list_head siblings; /* list of all devices on this host */ struct list_head same_target_siblings; /* just the devices sharing same target id */ - atomic_t device_busy; /* commands actually active on LLDD */ + struct sbitmap budget_map; atomic_t device_blocked; /* Device returned QUEUE_FULL. */ spinlock_t list_lock; @@ -586,7 +587,7 @@ static inline int scsi_device_supports_vpd(struct scsi_device *sdev) static inline int scsi_device_busy(struct scsi_device *sdev) { - return atomic_read(&sdev->device_busy); + return sbitmap_weight(&sdev->budget_map); } #define MODULE_ALIAS_SCSI_DEVICE(type) \
scsi uses one global atomic variable to track queue depth for each LUN/request queue. This way doesn't scale well when there is lots of CPU cores and the disk is very fast. It has been observed that IOPS is affected a lot by tracking queue depth via sdev->device_busy in IO path. So replace the atomic variable sdev->device_busy with sbitmap for tracking scsi device queue depth. Test shows that IOPS difference is just ~1% compared with bypassing device queue depth on scsi_debug by applying patches [1]. Meantime IOPS is improved > 20% compared with linus tree. Follows test steps: 1) test machine(32 logical CPU cores) Thread(s) per core: 2 Core(s) per socket: 8 Socket(s): 2 NUMA node(s): 2 Model name: Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz 2) setup scsi_debug: modprobe scsi_debug virtual_gb=128 max_luns=1 submit_queues=32 delay=0 max_queue=256 3) fio script: fio --rw=randread --size=128G --direct=1 --ioengine=libaio --iodepth=2048 \ --numjobs=32 --bs=4k --group_reporting=1 --group_reporting=1 --runtime=60 \ --loops=10000 --name=job1 --filename=/dev/sdN [1] https://lore.kernel.org/linux-block/20200119071432.18558-6-ming.lei@redhat.com/ Cc: Omar Sandoval <osandov@fb.com> Cc: Sathya Prakash <sathya.prakash@broadcom.com> Cc: Chaitra P B <chaitra.basappa@broadcom.com> Cc: Suganath Prabu Subramani <suganath-prabu.subramani@broadcom.com> Cc: Kashyap Desai <kashyap.desai@broadcom.com> Cc: Sumit Saxena <sumit.saxena@broadcom.com> Cc: Shivasharan S <shivasharan.srikanteshwara@broadcom.com> Cc: Ewan D. Milne <emilne@redhat.com> Cc: Hannes Reinecke <hare@suse.de> Cc: Bart Van Assche <bart.vanassche@wdc.com> Signed-off-by: Ming Lei <ming.lei@redhat.com> --- drivers/scsi/scsi.c | 2 ++ drivers/scsi/scsi_lib.c | 37 ++++++++++++++++++------------------- drivers/scsi/scsi_priv.h | 1 + drivers/scsi/scsi_scan.c | 21 +++++++++++++++++++-- drivers/scsi/scsi_sysfs.c | 2 ++ include/scsi/scsi_cmnd.h | 2 ++ include/scsi/scsi_device.h | 5 +++-- 7 files changed, 47 insertions(+), 23 deletions(-)