diff mbox

[4/5] IB: Switch from struct ib_dma_mapping_ops to struct dma_mapping_ops

Message ID 25d066c2-59d7-2be7-dd56-e29e99b43620@sandisk.com (mailing list archive)
State RFC
Headers show

Commit Message

Bart Van Assche Dec. 8, 2016, 1:11 a.m. UTC 
Additionally, use dma_noop_ops instead of duplicating it.

This patch eliminates one branch from every ib_dma_map_*() call.
---
 drivers/infiniband/hw/hfi1/dma.c      | 183 -------------------------------
 drivers/infiniband/hw/qib/qib_dma.c   | 169 -----------------------------
 drivers/infiniband/hw/qib/qib_keys.c  |   2 +-
 drivers/infiniband/sw/rdmavt/Makefile |   2 +-
 drivers/infiniband/sw/rdmavt/dma.c    | 198 ----------------------------------
 drivers/infiniband/sw/rdmavt/dma.h    |  53 ---------
 drivers/infiniband/sw/rdmavt/mr.c     |   4 +-
 drivers/infiniband/sw/rdmavt/vt.c     |   4 +-
 drivers/infiniband/sw/rdmavt/vt.h     |   1 -
 drivers/infiniband/sw/rxe/Makefile    |   1 -
 drivers/infiniband/sw/rxe/rxe_dma.c   | 183 -------------------------------
 drivers/infiniband/sw/rxe/rxe_loc.h   |   2 -
 drivers/infiniband/sw/rxe/rxe_verbs.c |   2 +-
 include/rdma/ib_verbs.h               | 117 +++-----------------
 14 files changed, 22 insertions(+), 899 deletions(-)
 delete mode 100644 drivers/infiniband/hw/hfi1/dma.c
 delete mode 100644 drivers/infiniband/hw/qib/qib_dma.c
 delete mode 100644 drivers/infiniband/sw/rdmavt/dma.c
 delete mode 100644 drivers/infiniband/sw/rdmavt/dma.h
 delete mode 100644 drivers/infiniband/sw/rxe/rxe_dma.c

Comments

Christoph Hellwig Dec. 9, 2016, 6:23 p.m. UTC | #1 
Looks fine:

Reviewed-by: Christoph Hellwig <hch@lst.de>

Even better with a follow up patch to kill the ib_* wrappers..
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Christoph Hellwig Dec. 9, 2016, 6:24 p.m. UTC | #2 
> Additionally, use dma_noop_ops instead of duplicating it.

Oops, I don't think that part actually is correct.  Both rdmavt
and the weird intel drivers want the _virtual_ address in their dma
address instead of the physical one set by dma_noop_ops.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Dennis Dalessandro Dec. 19, 2016, 4:42 p.m. UTC | #3 
On 12/09/2016 01:24 PM, Christoph Hellwig wrote:
>> Additionally, use dma_noop_ops instead of duplicating it.
>
> Oops, I don't think that part actually is correct.  Both rdmavt
> and the weird intel drivers want the _virtual_ address in their dma
> address instead of the physical one set by dma_noop_ops.

Yes, Christoph is correct in that our drivers use the virtual address. 
It would require more changes in the rest of the driver to make it use 
the physical address. Possible I think, but more work.

-Denny

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Bart Van Assche Dec. 19, 2016, 4:55 p.m. UTC | #4 
On Mon, 2016-12-19 at 11:42 -0500, Dennis Dalessandro wrote:
> On 12/09/2016 01:24 PM, Christoph Hellwig wrote:

> > > Additionally, use dma_noop_ops instead of duplicating it.

> > 

> > Oops, I don't think that part actually is correct.  Both rdmavt

> > and the weird intel drivers want the _virtual_ address in their dma

> > address instead of the physical one set by dma_noop_ops.

> 

> Yes, Christoph is correct in that our drivers use the virtual address. 

> It would require more changes in the rest of the driver to make it use 

> the physical address. Possible I think, but more work.


Hello Denny,

Once the v4.10 merge window has closed and v4.10-rc1 is out I will
repost this patch series. I will make sure when I repost this patch
series that the DMA code for the RDMA drivers touched by this patch
series uses virtual addresses instead of physical.

Bart.
Laurence Oberman Dec. 20, 2016, 7:33 p.m. UTC | #5 
Hello Bart

I pulled the latest linux-next and built kernels for both server and client to rerun all my EDR tests for srp. 

For some reason the I/O size is being capped again to 1MB in my testing.
Using my same testbed.
Remember we spent a lot of time making sure we could do 4MB I/O :)

Its working fine in the RHEL 7.3 kernel so before I start going back testing upstream kernels decided to ask.

Have you tested large I/O with latest linux-next

Server Configuration
---------------------
Linux fedstorage.bos.redhat.com 4.9.0+ 

[root@fedstorage modprobe.d]# cat ib_srp.conf
options ib_srp cmd_sg_entries=64 indirect_sg_entries=2048

[root@fedstorage modprobe.d]# cat ib_srpt.conf
options ib_srpt srp_max_req_size=8296

Also Using 

# Set the srp_sq_size
for i in /sys/kernel/config/target/srpt/0xfe800000000000007cfe900300726e4e /sys/kernel/config/target/srpt/0xfe800000000000007cfe900300726e4f
do 
	echo 16384 > $i/tpgt_1/attrib/srp_sq_size
done

Client Configuration
--------------------
Linux ibclient 4.9.0+

[root@ibclient modprobe.d]# cat ib_srp.conf
options ib_srp cmd_sg_entries=255 indirect_sg_entries=2048 

dd if=/dev/sdw bs=4096k of=/dev/null iflag=direct

### RECORD    4 >>> ibclient <<< (1482261733.001) (Tue Dec 20 14:22:13 2016) ###
# DISK STATISTICS (/sec)
#                   <---------reads---------------><---------writes--------------><--------averages--------> Pct
#Time     Name       KBytes Merged  IOs Size  Wait  KBytes Merged  IOs Size  Wait  RWSize  QLen  Wait SvcTim Util
14:22:13 sdw         1373184      0 1341 1024     2       0      0    0    0     0    1024     3     2      0   97


If I reboot into my 7.3 kernel its back to what I expect

 dd if=/dev/sdw bs=4096k of=/dev/null iflag=direct


### RECORD    3 >>> ibclient <<< (1482262254.001) (Tue Dec 20 14:30:54 2016) ###
# DISK STATISTICS (/sec)
#                   <---------reads---------------><---------writes--------------><--------averages--------> Pct
#Time     Name       KBytes Merged  IOs Size  Wait  KBytes Merged  IOs Size  Wait  RWSize  QLen  Wait SvcTim Util
14:30:54 sdw         172032    129   42 4096     3       0      0    0    0     0    4096     1     3      3   130

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Laurence Oberman Dec. 20, 2016, 7:43 p.m. UTC | #6 
----- Original Message -----
> From: "Laurence Oberman" <loberman@redhat.com>
> To: "Bart Van Assche" <Bart.VanAssche@sandisk.com>
> Cc: linux-rdma@vger.kernel.org
> Sent: Tuesday, December 20, 2016 2:33:26 PM
> Subject: Re: Testing latest linux-next 4.9 ib_srp and ib_srpt
> 
> Hello Bart
> 
> I pulled the latest linux-next and built kernels for both server and client
> to rerun all my EDR tests for srp.
> 
> For some reason the I/O size is being capped again to 1MB in my testing.
> Using my same testbed.
> Remember we spent a lot of time making sure we could do 4MB I/O :)
> 
> Its working fine in the RHEL 7.3 kernel so before I start going back testing
> upstream kernels decided to ask.
> 
> Have you tested large I/O with latest linux-next
> 
> Server Configuration
> ---------------------
> Linux fedstorage.bos.redhat.com 4.9.0+
> 
> [root@fedstorage modprobe.d]# cat ib_srp.conf
> options ib_srp cmd_sg_entries=64 indirect_sg_entries=2048
> 
> [root@fedstorage modprobe.d]# cat ib_srpt.conf
> options ib_srpt srp_max_req_size=8296
> 
> Also Using
> 
> # Set the srp_sq_size
> for i in /sys/kernel/config/target/srpt/0xfe800000000000007cfe900300726e4e
> /sys/kernel/config/target/srpt/0xfe800000000000007cfe900300726e4f
> do
> 	echo 16384 > $i/tpgt_1/attrib/srp_sq_size
> done
> 
> Client Configuration
> --------------------
> Linux ibclient 4.9.0+
> 
> [root@ibclient modprobe.d]# cat ib_srp.conf
> options ib_srp cmd_sg_entries=255 indirect_sg_entries=2048
> 
> dd if=/dev/sdw bs=4096k of=/dev/null iflag=direct
> 
> ### RECORD    4 >>> ibclient <<< (1482261733.001) (Tue Dec 20 14:22:13 2016)
> ###
> # DISK STATISTICS (/sec)
> #
> <---------reads---------------><---------writes--------------><--------averages-------->
> Pct
> #Time     Name       KBytes Merged  IOs Size  Wait  KBytes Merged  IOs Size
> Wait  RWSize  QLen  Wait SvcTim Util
> 14:22:13 sdw         1373184      0 1341 1024     2       0      0    0    0
> 0    1024     3     2      0   97
> 
> 
> If I reboot into my 7.3 kernel its back to what I expect
> 
>  dd if=/dev/sdw bs=4096k of=/dev/null iflag=direct
> 
> 
> ### RECORD    3 >>> ibclient <<< (1482262254.001) (Tue Dec 20 14:30:54 2016)
> ###
> # DISK STATISTICS (/sec)
> #
> <---------reads---------------><---------writes--------------><--------averages-------->
> Pct
> #Time     Name       KBytes Merged  IOs Size  Wait  KBytes Merged  IOs Size
> Wait  RWSize  QLen  Wait SvcTim Util
> 14:30:54 sdw         172032    129   42 4096     3       0      0    0    0
> 0    4096     1     3      3   130
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Hi Bart,

Just FYI

That dd snap was just as I had stopped the dd.

Here is a stable dd snap with the RHEL kernel, I just noticed the merging, need to reboot back into upstream to compare again.
No merging seen in upstream.

Thanks
Laurence

### RECORD    6 >>> ibclient <<< (1482262723.001) (Tue Dec 20 14:38:43 2016) ###
# DISK STATISTICS (/sec)
#                   <---------reads---------------><---------writes--------------><--------averages--------> Pct
#Time     Name       KBytes Merged  IOs Size  Wait  KBytes Merged  IOs Size  Wait  RWSize  QLen  Wait SvcTim Util
14:38:43 sdw         1200128    879  293 4096     3       0      0    0    0     0    4096     1     3      3   95
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Laurence Oberman Dec. 20, 2016, 8:44 p.m. UTC | #7 
----- Original Message -----
> From: "Laurence Oberman" <loberman@redhat.com>
> To: "Bart Van Assche" <Bart.VanAssche@sandisk.com>
> Cc: linux-rdma@vger.kernel.org
> Sent: Tuesday, December 20, 2016 2:43:48 PM
> Subject: Re: Testing latest linux-next 4.9 ib_srp and ib_srpt
> 
> 
> 
> ----- Original Message -----
> > From: "Laurence Oberman" <loberman@redhat.com>
> > To: "Bart Van Assche" <Bart.VanAssche@sandisk.com>
> > Cc: linux-rdma@vger.kernel.org
> > Sent: Tuesday, December 20, 2016 2:33:26 PM
> > Subject: Re: Testing latest linux-next 4.9 ib_srp and ib_srpt
> > 
> > Hello Bart
> > 
> > I pulled the latest linux-next and built kernels for both server and client
> > to rerun all my EDR tests for srp.
> > 
> > For some reason the I/O size is being capped again to 1MB in my testing.
> > Using my same testbed.
> > Remember we spent a lot of time making sure we could do 4MB I/O :)
> > 
> > Its working fine in the RHEL 7.3 kernel so before I start going back
> > testing
> > upstream kernels decided to ask.
> > 
> > Have you tested large I/O with latest linux-next
> > 
> > Server Configuration
> > ---------------------
> > Linux fedstorage.bos.redhat.com 4.9.0+
> > 
> > [root@fedstorage modprobe.d]# cat ib_srp.conf
> > options ib_srp cmd_sg_entries=64 indirect_sg_entries=2048
> > 
> > [root@fedstorage modprobe.d]# cat ib_srpt.conf
> > options ib_srpt srp_max_req_size=8296
> > 
> > Also Using
> > 
> > # Set the srp_sq_size
> > for i in /sys/kernel/config/target/srpt/0xfe800000000000007cfe900300726e4e
> > /sys/kernel/config/target/srpt/0xfe800000000000007cfe900300726e4f
> > do
> > 	echo 16384 > $i/tpgt_1/attrib/srp_sq_size
> > done
> > 
> > Client Configuration
> > --------------------
> > Linux ibclient 4.9.0+
> > 
> > [root@ibclient modprobe.d]# cat ib_srp.conf
> > options ib_srp cmd_sg_entries=255 indirect_sg_entries=2048
> > 
> > dd if=/dev/sdw bs=4096k of=/dev/null iflag=direct
> > 
> > ### RECORD    4 >>> ibclient <<< (1482261733.001) (Tue Dec 20 14:22:13
> > 2016)
> > ###
> > # DISK STATISTICS (/sec)
> > #
> > <---------reads---------------><---------writes--------------><--------averages-------->
> > Pct
> > #Time     Name       KBytes Merged  IOs Size  Wait  KBytes Merged  IOs Size
> > Wait  RWSize  QLen  Wait SvcTim Util
> > 14:22:13 sdw         1373184      0 1341 1024     2       0      0    0
> > 0
> > 0    1024     3     2      0   97
> > 
> > 
> > If I reboot into my 7.3 kernel its back to what I expect
> > 
> >  dd if=/dev/sdw bs=4096k of=/dev/null iflag=direct
> > 
> > 
> > ### RECORD    3 >>> ibclient <<< (1482262254.001) (Tue Dec 20 14:30:54
> > 2016)
> > ###
> > # DISK STATISTICS (/sec)
> > #
> > <---------reads---------------><---------writes--------------><--------averages-------->
> > Pct
> > #Time     Name       KBytes Merged  IOs Size  Wait  KBytes Merged  IOs Size
> > Wait  RWSize  QLen  Wait SvcTim Util
> > 14:30:54 sdw         172032    129   42 4096     3       0      0    0    0
> > 0    4096     1     3      3   130
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> 
> Hi Bart,
> 
> Just FYI
> 
> That dd snap was just as I had stopped the dd.
> 
> Here is a stable dd snap with the RHEL kernel, I just noticed the merging,
> need to reboot back into upstream to compare again.
> No merging seen in upstream.
> 
> Thanks
> Laurence
> 
> ### RECORD    6 >>> ibclient <<< (1482262723.001) (Tue Dec 20 14:38:43 2016)
> ###
> # DISK STATISTICS (/sec)
> #
> <---------reads---------------><---------writes--------------><--------averages-------->
> Pct
> #Time     Name       KBytes Merged  IOs Size  Wait  KBytes Merged  IOs Size
> Wait  RWSize  QLen  Wait SvcTim Util
> 14:38:43 sdw         1200128    879  293 4096     3       0      0    0    0
> 0    4096     1     3      3   95
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Replying to my own message to keep the thread going


This is Linux ibclient 4.8.0-rc4 

Behaves like I expected, and I see the 4MB I/O sizes. as I had already tested this.

dd if=/dev/sdw bs=4096k of=/dev/null iflag=direct


### RECORD    6 >>> ibclient <<< (1482266543.001) (Tue Dec 20 15:42:23 2016) ###
# DISK STATISTICS (/sec)
#                   <---------reads---------------><---------writes--------------><--------averages--------> Pct
#Time     Name       KBytes Merged  IOs Size  Wait  KBytes Merged  IOs Size  Wait  RWSize  QLen  Wait SvcTim Util
15:42:23 sdw         278528    201   68 4096     2       0      0    0    0     0    4096     1     2      2   206

Rebooting back into 4.9 and will bisect leading to it once I confirm

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Laurence Oberman Dec. 20, 2016, 9:09 p.m. UTC | #8 
----- Original Message -----
> From: "Laurence Oberman" <loberman@redhat.com>
> To: "Bart Van Assche" <Bart.VanAssche@sandisk.com>
> Cc: linux-rdma@vger.kernel.org
> Sent: Tuesday, December 20, 2016 3:44:42 PM
> Subject: Re: Testing latest linux-next 4.9 ib_srp and ib_srpt
> 
> 
> 
> ----- Original Message -----
> > From: "Laurence Oberman" <loberman@redhat.com>
> > To: "Bart Van Assche" <Bart.VanAssche@sandisk.com>
> > Cc: linux-rdma@vger.kernel.org
> > Sent: Tuesday, December 20, 2016 2:43:48 PM
> > Subject: Re: Testing latest linux-next 4.9 ib_srp and ib_srpt
> > 
> > 
> > 
> > ----- Original Message -----
> > > From: "Laurence Oberman" <loberman@redhat.com>
> > > To: "Bart Van Assche" <Bart.VanAssche@sandisk.com>
> > > Cc: linux-rdma@vger.kernel.org
> > > Sent: Tuesday, December 20, 2016 2:33:26 PM
> > > Subject: Re: Testing latest linux-next 4.9 ib_srp and ib_srpt
> > > 
> > > Hello Bart
> > > 
> > > I pulled the latest linux-next and built kernels for both server and
> > > client
> > > to rerun all my EDR tests for srp.
> > > 
> > > For some reason the I/O size is being capped again to 1MB in my testing.
> > > Using my same testbed.
> > > Remember we spent a lot of time making sure we could do 4MB I/O :)
> > > 
> > > Its working fine in the RHEL 7.3 kernel so before I start going back
> > > testing
> > > upstream kernels decided to ask.
> > > 
> > > Have you tested large I/O with latest linux-next
> > > 
> > > Server Configuration
> > > ---------------------
> > > Linux fedstorage.bos.redhat.com 4.9.0+
> > > 
> > > [root@fedstorage modprobe.d]# cat ib_srp.conf
> > > options ib_srp cmd_sg_entries=64 indirect_sg_entries=2048
> > > 
> > > [root@fedstorage modprobe.d]# cat ib_srpt.conf
> > > options ib_srpt srp_max_req_size=8296
> > > 
> > > Also Using
> > > 
> > > # Set the srp_sq_size
> > > for i in
> > > /sys/kernel/config/target/srpt/0xfe800000000000007cfe900300726e4e
> > > /sys/kernel/config/target/srpt/0xfe800000000000007cfe900300726e4f
> > > do
> > > 	echo 16384 > $i/tpgt_1/attrib/srp_sq_size
> > > done
> > > 
> > > Client Configuration
> > > --------------------
> > > Linux ibclient 4.9.0+
> > > 
> > > [root@ibclient modprobe.d]# cat ib_srp.conf
> > > options ib_srp cmd_sg_entries=255 indirect_sg_entries=2048
> > > 
> > > dd if=/dev/sdw bs=4096k of=/dev/null iflag=direct
> > > 
> > > ### RECORD    4 >>> ibclient <<< (1482261733.001) (Tue Dec 20 14:22:13
> > > 2016)
> > > ###
> > > # DISK STATISTICS (/sec)
> > > #
> > > <---------reads---------------><---------writes--------------><--------averages-------->
> > > Pct
> > > #Time     Name       KBytes Merged  IOs Size  Wait  KBytes Merged  IOs
> > > Size
> > > Wait  RWSize  QLen  Wait SvcTim Util
> > > 14:22:13 sdw         1373184      0 1341 1024     2       0      0    0
> > > 0
> > > 0    1024     3     2      0   97
> > > 
> > > 
> > > If I reboot into my 7.3 kernel its back to what I expect
> > > 
> > >  dd if=/dev/sdw bs=4096k of=/dev/null iflag=direct
> > > 
> > > 
> > > ### RECORD    3 >>> ibclient <<< (1482262254.001) (Tue Dec 20 14:30:54
> > > 2016)
> > > ###
> > > # DISK STATISTICS (/sec)
> > > #
> > > <---------reads---------------><---------writes--------------><--------averages-------->
> > > Pct
> > > #Time     Name       KBytes Merged  IOs Size  Wait  KBytes Merged  IOs
> > > Size
> > > Wait  RWSize  QLen  Wait SvcTim Util
> > > 14:30:54 sdw         172032    129   42 4096     3       0      0    0
> > > 0
> > > 0    4096     1     3      3   130
> > > 
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > 
> > 
> > Hi Bart,
> > 
> > Just FYI
> > 
> > That dd snap was just as I had stopped the dd.
> > 
> > Here is a stable dd snap with the RHEL kernel, I just noticed the merging,
> > need to reboot back into upstream to compare again.
> > No merging seen in upstream.
> > 
> > Thanks
> > Laurence
> > 
> > ### RECORD    6 >>> ibclient <<< (1482262723.001) (Tue Dec 20 14:38:43
> > 2016)
> > ###
> > # DISK STATISTICS (/sec)
> > #
> > <---------reads---------------><---------writes--------------><--------averages-------->
> > Pct
> > #Time     Name       KBytes Merged  IOs Size  Wait  KBytes Merged  IOs Size
> > Wait  RWSize  QLen  Wait SvcTim Util
> > 14:38:43 sdw         1200128    879  293 4096     3       0      0    0
> > 0
> > 0    4096     1     3      3   95
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> 
> Replying to my own message to keep the thread going
> 
> 
> This is Linux ibclient 4.8.0-rc4
> 
> Behaves like I expected, and I see the 4MB I/O sizes. as I had already tested
> this.
> 
> dd if=/dev/sdw bs=4096k of=/dev/null iflag=direct
> 
> 
> ### RECORD    6 >>> ibclient <<< (1482266543.001) (Tue Dec 20 15:42:23 2016)
> ###
> # DISK STATISTICS (/sec)
> #
> <---------reads---------------><---------writes--------------><--------averages-------->
> Pct
> #Time     Name       KBytes Merged  IOs Size  Wait  KBytes Merged  IOs Size
> Wait  RWSize  QLen  Wait SvcTim Util
> 15:42:23 sdw         278528    201   68 4096     2       0      0    0    0
> 0    4096     1     2      2   206
> 
> Rebooting back into 4.9 and will bisect leading to it once I confirm
> 
> 

[root@ibclient ~]# uname -a
Linux ibclient 4.9.0+ #1 SMP Tue Dec 20 10:06:26 EST 2016 x86_64 x86_64 x86_64 GNU/Linux

Yep, back into 4.9 and we are at 1MB I/O

Confirmed regression or behavior change in 4.9

while true; do dd if=/dev/sdw bs=4096k of=/dev/null iflag=direct 1>/dev/null 2>&1 ; done &

# DISK STATISTICS (/sec)
#                   <---------reads---------------><---------writes--------------><--------averages--------> Pct
#Time     Name       KBytes Merged  IOs Size  Wait  KBytes Merged  IOs Size  Wait  RWSize  QLen  Wait SvcTim Util
16:06:33 sdw         1405952      0 1373 1024     2       0      0    0    0     0    1024     3     2      0   98
16:06:34 sdw         1424384      0 1391 1024     2       0      0    0    0     0    1024     3     2      0   97
16:06:35 sdw         1462272      0 1428 1024     2       0      0    0    0     0    1024     3     2      0   97
16:06:36 sdw         1417216      0 1384 1024     2       0      0    0    0     0    1024     3     2      0   97
16:06:37 sdw         1380352      0 1348 1024     2       0      0    0    0     0    1024     3     2      0   98
16:06:38 sdw         1404928      0 1372 1024     2       0      0    0    0     0    1024     3     2      0   98
16:06:39 sdw         1458176      0 1424 1024     2       0      0    0    0     0    1024     3     2      0   97

I will try chase it down and be back with an update

Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Laurence Oberman Dec. 21, 2016, 3:31 a.m. UTC | #9 
----- Original Message -----
> From: "Laurence Oberman" <loberman@redhat.com>
> To: "Bart Van Assche" <Bart.VanAssche@sandisk.com>
> Cc: linux-rdma@vger.kernel.org
> Sent: Tuesday, December 20, 2016 3:44:42 PM
> Subject: Re: Testing latest linux-next 4.9 ib_srp and ib_srpt
> 
> 
> 
> ----- Original Message -----
> > From: "Laurence Oberman" <loberman@redhat.com>
> > To: "Bart Van Assche" <Bart.VanAssche@sandisk.com>
> > Cc: linux-rdma@vger.kernel.org
> > Sent: Tuesday, December 20, 2016 2:43:48 PM
> > Subject: Re: Testing latest linux-next 4.9 ib_srp and ib_srpt
> > 
> > 
> > 
> > ----- Original Message -----
> > > From: "Laurence Oberman" <loberman@redhat.com>
> > > To: "Bart Van Assche" <Bart.VanAssche@sandisk.com>
> > > Cc: linux-rdma@vger.kernel.org
> > > Sent: Tuesday, December 20, 2016 2:33:26 PM
> > > Subject: Re: Testing latest linux-next 4.9 ib_srp and ib_srpt
> > > 
> > > Hello Bart
> > > 
> > > I pulled the latest linux-next and built kernels for both server and
> > > client
> > > to rerun all my EDR tests for srp.
> > > 
> > > For some reason the I/O size is being capped again to 1MB in my testing.
> > > Using my same testbed.
> > > Remember we spent a lot of time making sure we could do 4MB I/O :)
> > > 
> > > Its working fine in the RHEL 7.3 kernel so before I start going back
> > > testing
> > > upstream kernels decided to ask.
> > > 
> > > Have you tested large I/O with latest linux-next
> > > 
> > > Server Configuration
> > > ---------------------
> > > Linux fedstorage.bos.redhat.com 4.9.0+
> > > 
> > > [root@fedstorage modprobe.d]# cat ib_srp.conf
> > > options ib_srp cmd_sg_entries=64 indirect_sg_entries=2048
> > > 
> > > [root@fedstorage modprobe.d]# cat ib_srpt.conf
> > > options ib_srpt srp_max_req_size=8296
> > > 
> > > Also Using
> > > 
> > > # Set the srp_sq_size
> > > for i in
> > > /sys/kernel/config/target/srpt/0xfe800000000000007cfe900300726e4e
> > > /sys/kernel/config/target/srpt/0xfe800000000000007cfe900300726e4f
> > > do
> > > 	echo 16384 > $i/tpgt_1/attrib/srp_sq_size
> > > done
> > > 
> > > Client Configuration
> > > --------------------
> > > Linux ibclient 4.9.0+
> > > 
> > > [root@ibclient modprobe.d]# cat ib_srp.conf
> > > options ib_srp cmd_sg_entries=255 indirect_sg_entries=2048
> > > 
> > > dd if=/dev/sdw bs=4096k of=/dev/null iflag=direct
> > > 
> > > ### RECORD    4 >>> ibclient <<< (1482261733.001) (Tue Dec 20 14:22:13
> > > 2016)
> > > ###
> > > # DISK STATISTICS (/sec)
> > > #
> > > <---------reads---------------><---------writes--------------><--------averages-------->
> > > Pct
> > > #Time     Name       KBytes Merged  IOs Size  Wait  KBytes Merged  IOs
> > > Size
> > > Wait  RWSize  QLen  Wait SvcTim Util
> > > 14:22:13 sdw         1373184      0 1341 1024     2       0      0    0
> > > 0
> > > 0    1024     3     2      0   97
> > > 
> > > 
> > > If I reboot into my 7.3 kernel its back to what I expect
> > > 
> > >  dd if=/dev/sdw bs=4096k of=/dev/null iflag=direct
> > > 
> > > 
> > > ### RECORD    3 >>> ibclient <<< (1482262254.001) (Tue Dec 20 14:30:54
> > > 2016)
> > > ###
> > > # DISK STATISTICS (/sec)
> > > #
> > > <---------reads---------------><---------writes--------------><--------averages-------->
> > > Pct
> > > #Time     Name       KBytes Merged  IOs Size  Wait  KBytes Merged  IOs
> > > Size
> > > Wait  RWSize  QLen  Wait SvcTim Util
> > > 14:30:54 sdw         172032    129   42 4096     3       0      0    0
> > > 0
> > > 0    4096     1     3      3   130
> > > 
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > 
> > 
> > Hi Bart,
> > 
> > Just FYI
> > 
> > That dd snap was just as I had stopped the dd.
> > 
> > Here is a stable dd snap with the RHEL kernel, I just noticed the merging,
> > need to reboot back into upstream to compare again.
> > No merging seen in upstream.
> > 
> > Thanks
> > Laurence
> > 
> > ### RECORD    6 >>> ibclient <<< (1482262723.001) (Tue Dec 20 14:38:43
> > 2016)
> > ###
> > # DISK STATISTICS (/sec)
> > #
> > <---------reads---------------><---------writes--------------><--------averages-------->
> > Pct
> > #Time     Name       KBytes Merged  IOs Size  Wait  KBytes Merged  IOs Size
> > Wait  RWSize  QLen  Wait SvcTim Util
> > 14:38:43 sdw         1200128    879  293 4096     3       0      0    0
> > 0
> > 0    4096     1     3      3   95
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> 
> Replying to my own message to keep the thread going
> 
> 
> This is Linux ibclient 4.8.0-rc4
> 
> Behaves like I expected, and I see the 4MB I/O sizes. as I had already tested
> this.
> 
> dd if=/dev/sdw bs=4096k of=/dev/null iflag=direct
> 
> 
> ### RECORD    6 >>> ibclient <<< (1482266543.001) (Tue Dec 20 15:42:23 2016)
> ###
> # DISK STATISTICS (/sec)
> #
> <---------reads---------------><---------writes--------------><--------averages-------->
> Pct
> #Time     Name       KBytes Merged  IOs Size  Wait  KBytes Merged  IOs Size
> Wait  RWSize  QLen  Wait SvcTim Util
> 15:42:23 sdw         278528    201   68 4096     2       0      0    0    0
> 0    4096     1     2      2   206
> 
> Rebooting back into 4.9 and will bisect leading to it once I confirm
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

So this is where I got to here:

These all worked and I get the 4MB I/O size with direct I/O

v4.8-rc4
v4.8-rc5
v4.8-rc6
v4.8-rc7
v4.8-rc8
v4.9
v4.9-rc1
v4.9-rc2
v4.9-rc3
v4.9-rc4
v4.9-rc5
v4.9-rc6
v4.9-rc7
v4.9-rc8

Then git checkout master and build final test kernel
4.9.0+

This one fails

# DISK STATISTICS (/sec)
#                   <---------reads---------------><---------writes--------------><--------averages--------> Pct
#Time     Name       KBytes Merged  IOs Size  Wait  KBytes Merged  IOs Size  Wait  RWSize  QLen  Wait SvcTim Util
22:12:48 sdw         1413120      0 1380 1024     2       0      0    0    0     0    1024     3     2      0   99
22:12:49 sdw         1409024      0 1376 1024     2       0      0    0    0     0    1024     3     2      0   98
22:12:50 sdw         1445888      0 1412 1024     2       0      0    0    0     0    1024     3     2      0   98
22:12:51 sdw         1429504      0 1396 1024     2       0      0    0    0     0    1024     3     2      0   98
22:12:52 sdw         1426432      0 1393 1024     2       0      0    0    0     0    1024     3     2      0   98
22:12:53 sdw         1408000      0 1375 1024     2       0      0    0    0     0    1024     3     2      0   98
                                ***      ****                                         ****

The last commit for 4.9-rc8 was 3e5de27

Between this and the master checkout HEAD there are 2313 lines of commits
This was the giant merge.

Something broke here. 
I guess I will have to bisect the hard way, unless somebody realizes which of the commits broke it.

Commits for SRP, none of these look like candidates to break the I/O size and merge

9032ad7 Merge branches 'misc', 'qedr', 'reject-helpers', 'rxe' and 'srp' into merge-test
4fa354c IB/srp: Make writing the add_target sysfs attr interruptible
290081b IB/srp: Make mapping failures easier to debug
3787d99 IB/srp: Make login failures easier to debug
042dd76 IB/srp: Introduce a local variable in srp_add_one()
1a1faf7 IB/srp: Fix CONFIG_DYNAMIC_DEBUG=n build
0d38c24 IB/srpt: Report login failures only once

dm related, however I am testing against an sd, not dm device here

ef548c5 dm flakey: introduce "error_writes" feature
e99dda8f dm cache policy smq: use hash_32() instead of hash_32_generic()
027c431 dm crypt: reject key strings containing whitespace chars
b446396 dm space map: always set ev if sm_ll_mutate() succeeds
0c79ce0 dm space map metadata: skip useless memcpy in metadata_ll_init_index()
314c25c dm space map metadata: fix 'struct sm_metadata' leak on failed create
58fc4fe Documentation: dm raid: define data_offset status field
11e2968 dm raid: fix discard support regression
affa9d2 dm raid: don't allow "write behind" with raid4/5/6
54cd640 dm mpath: use hw_handler_params if attached hw_handler is same as requested
c538f6e dm crypt: add ability to use keys from the kernel key retention service
0637018 dm array: remove a dead assignment in populate_ablock_with_values()
6080758 dm ioctl: use offsetof() instead of open-coding it
b23df0d dm rq: simplify use_blk_mq initialization
41c73a4 dm bufio: drop the lock when doing GFP_NOIO allocation
d12067f dm bufio: don't take the lock in dm_bufio_shrink_count
9ea61ca dm bufio: avoid sleeping while holding the dm_bufio lock
5b8c01f dm table: simplify dm_table_determine_type()
301fc3f dm table: an 'all_blk_mq' table must be loaded for a blk-mq DM device
6936c12 dm table: fix 'all_blk_mq' inconsistency when an empty table is loaded

I have time next week so will try narrow it down via bisect by commit

Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Laurence Oberman Dec. 21, 2016, 6:34 a.m. UTC | #10 
----- Original Message -----
> From: "Laurence Oberman" <loberman@redhat.com>
> To: "Bart Van Assche" <Bart.VanAssche@sandisk.com>
> Cc: linux-rdma@vger.kernel.org
> Sent: Tuesday, December 20, 2016 10:31:34 PM
> Subject: Re: Testing latest linux-next 4.9 ib_srp and ib_srpt
> 
> 
> 
> ----- Original Message -----
> > From: "Laurence Oberman" <loberman@redhat.com>
> > To: "Bart Van Assche" <Bart.VanAssche@sandisk.com>
> > Cc: linux-rdma@vger.kernel.org
> > Sent: Tuesday, December 20, 2016 3:44:42 PM
> > Subject: Re: Testing latest linux-next 4.9 ib_srp and ib_srpt
> > 
> > 
> > 
> > ----- Original Message -----
> > > From: "Laurence Oberman" <loberman@redhat.com>
> > > To: "Bart Van Assche" <Bart.VanAssche@sandisk.com>
> > > Cc: linux-rdma@vger.kernel.org
> > > Sent: Tuesday, December 20, 2016 2:43:48 PM
> > > Subject: Re: Testing latest linux-next 4.9 ib_srp and ib_srpt
> > > 
> > > 
> > > 
> > > ----- Original Message -----
> > > > From: "Laurence Oberman" <loberman@redhat.com>
> > > > To: "Bart Van Assche" <Bart.VanAssche@sandisk.com>
> > > > Cc: linux-rdma@vger.kernel.org
> > > > Sent: Tuesday, December 20, 2016 2:33:26 PM
> > > > Subject: Re: Testing latest linux-next 4.9 ib_srp and ib_srpt
> > > > 
> > > > Hello Bart
> > > > 
> > > > I pulled the latest linux-next and built kernels for both server and
> > > > client
> > > > to rerun all my EDR tests for srp.
> > > > 
> > > > For some reason the I/O size is being capped again to 1MB in my
> > > > testing.
> > > > Using my same testbed.
> > > > Remember we spent a lot of time making sure we could do 4MB I/O :)
> > > > 
> > > > Its working fine in the RHEL 7.3 kernel so before I start going back
> > > > testing
> > > > upstream kernels decided to ask.
> > > > 
> > > > Have you tested large I/O with latest linux-next
> > > > 
> > > > Server Configuration
> > > > ---------------------
> > > > Linux fedstorage.bos.redhat.com 4.9.0+
> > > > 
> > > > [root@fedstorage modprobe.d]# cat ib_srp.conf
> > > > options ib_srp cmd_sg_entries=64 indirect_sg_entries=2048
> > > > 
> > > > [root@fedstorage modprobe.d]# cat ib_srpt.conf
> > > > options ib_srpt srp_max_req_size=8296
> > > > 
> > > > Also Using
> > > > 
> > > > # Set the srp_sq_size
> > > > for i in
> > > > /sys/kernel/config/target/srpt/0xfe800000000000007cfe900300726e4e
> > > > /sys/kernel/config/target/srpt/0xfe800000000000007cfe900300726e4f
> > > > do
> > > > 	echo 16384 > $i/tpgt_1/attrib/srp_sq_size
> > > > done
> > > > 
> > > > Client Configuration
> > > > --------------------
> > > > Linux ibclient 4.9.0+
> > > > 
> > > > [root@ibclient modprobe.d]# cat ib_srp.conf
> > > > options ib_srp cmd_sg_entries=255 indirect_sg_entries=2048
> > > > 
> > > > dd if=/dev/sdw bs=4096k of=/dev/null iflag=direct
> > > > 
> > > > ### RECORD    4 >>> ibclient <<< (1482261733.001) (Tue Dec 20 14:22:13
> > > > 2016)
> > > > ###
> > > > # DISK STATISTICS (/sec)
> > > > #
> > > > <---------reads---------------><---------writes--------------><--------averages-------->
> > > > Pct
> > > > #Time     Name       KBytes Merged  IOs Size  Wait  KBytes Merged  IOs
> > > > Size
> > > > Wait  RWSize  QLen  Wait SvcTim Util
> > > > 14:22:13 sdw         1373184      0 1341 1024     2       0      0    0
> > > > 0
> > > > 0    1024     3     2      0   97
> > > > 
> > > > 
> > > > If I reboot into my 7.3 kernel its back to what I expect
> > > > 
> > > >  dd if=/dev/sdw bs=4096k of=/dev/null iflag=direct
> > > > 
> > > > 
> > > > ### RECORD    3 >>> ibclient <<< (1482262254.001) (Tue Dec 20 14:30:54
> > > > 2016)
> > > > ###
> > > > # DISK STATISTICS (/sec)
> > > > #
> > > > <---------reads---------------><---------writes--------------><--------averages-------->
> > > > Pct
> > > > #Time     Name       KBytes Merged  IOs Size  Wait  KBytes Merged  IOs
> > > > Size
> > > > Wait  RWSize  QLen  Wait SvcTim Util
> > > > 14:30:54 sdw         172032    129   42 4096     3       0      0    0
> > > > 0
> > > > 0    4096     1     3      3   130
> > > > 
> > > > --
> > > > To unsubscribe from this list: send the line "unsubscribe linux-rdma"
> > > > in
> > > > the body of a message to majordomo@vger.kernel.org
> > > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > > 
> > > 
> > > Hi Bart,
> > > 
> > > Just FYI
> > > 
> > > That dd snap was just as I had stopped the dd.
> > > 
> > > Here is a stable dd snap with the RHEL kernel, I just noticed the
> > > merging,
> > > need to reboot back into upstream to compare again.
> > > No merging seen in upstream.
> > > 
> > > Thanks
> > > Laurence
> > > 
> > > ### RECORD    6 >>> ibclient <<< (1482262723.001) (Tue Dec 20 14:38:43
> > > 2016)
> > > ###
> > > # DISK STATISTICS (/sec)
> > > #
> > > <---------reads---------------><---------writes--------------><--------averages-------->
> > > Pct
> > > #Time     Name       KBytes Merged  IOs Size  Wait  KBytes Merged  IOs
> > > Size
> > > Wait  RWSize  QLen  Wait SvcTim Util
> > > 14:38:43 sdw         1200128    879  293 4096     3       0      0    0
> > > 0
> > > 0    4096     1     3      3   95
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > 
> > 
> > Replying to my own message to keep the thread going
> > 
> > 
> > This is Linux ibclient 4.8.0-rc4
> > 
> > Behaves like I expected, and I see the 4MB I/O sizes. as I had already
> > tested
> > this.
> > 
> > dd if=/dev/sdw bs=4096k of=/dev/null iflag=direct
> > 
> > 
> > ### RECORD    6 >>> ibclient <<< (1482266543.001) (Tue Dec 20 15:42:23
> > 2016)
> > ###
> > # DISK STATISTICS (/sec)
> > #
> > <---------reads---------------><---------writes--------------><--------averages-------->
> > Pct
> > #Time     Name       KBytes Merged  IOs Size  Wait  KBytes Merged  IOs Size
> > Wait  RWSize  QLen  Wait SvcTim Util
> > 15:42:23 sdw         278528    201   68 4096     2       0      0    0    0
> > 0    4096     1     2      2   206
> > 
> > Rebooting back into 4.9 and will bisect leading to it once I confirm
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> 
> So this is where I got to here:
> 
> These all worked and I get the 4MB I/O size with direct I/O
> 
> v4.8-rc4
> v4.8-rc5
> v4.8-rc6
> v4.8-rc7
> v4.8-rc8
> v4.9
> v4.9-rc1
> v4.9-rc2
> v4.9-rc3
> v4.9-rc4
> v4.9-rc5
> v4.9-rc6
> v4.9-rc7
> v4.9-rc8
> 
> Then git checkout master and build final test kernel
> 4.9.0+
> 
> This one fails
> 
> # DISK STATISTICS (/sec)
> #
> <---------reads---------------><---------writes--------------><--------averages-------->
> Pct
> #Time     Name       KBytes Merged  IOs Size  Wait  KBytes Merged  IOs Size
> Wait  RWSize  QLen  Wait SvcTim Util
> 22:12:48 sdw         1413120      0 1380 1024     2       0      0    0    0
> 0    1024     3     2      0   99
> 22:12:49 sdw         1409024      0 1376 1024     2       0      0    0    0
> 0    1024     3     2      0   98
> 22:12:50 sdw         1445888      0 1412 1024     2       0      0    0    0
> 0    1024     3     2      0   98
> 22:12:51 sdw         1429504      0 1396 1024     2       0      0    0    0
> 0    1024     3     2      0   98
> 22:12:52 sdw         1426432      0 1393 1024     2       0      0    0    0
> 0    1024     3     2      0   98
> 22:12:53 sdw         1408000      0 1375 1024     2       0      0    0    0
> 0    1024     3     2      0   98
>                                 ***      ****
>                                 ****
> 
> The last commit for 4.9-rc8 was 3e5de27
> 
> Between this and the master checkout HEAD there are 2313 lines of commits
> This was the giant merge.
> 
> Something broke here.
> I guess I will have to bisect the hard way, unless somebody realizes which of
> the commits broke it.
> 
> Commits for SRP, none of these look like candidates to break the I/O size and
> merge
> 
> 9032ad7 Merge branches 'misc', 'qedr', 'reject-helpers', 'rxe' and 'srp' into
> merge-test
> 4fa354c IB/srp: Make writing the add_target sysfs attr interruptible
> 290081b IB/srp: Make mapping failures easier to debug
> 3787d99 IB/srp: Make login failures easier to debug
> 042dd76 IB/srp: Introduce a local variable in srp_add_one()
> 1a1faf7 IB/srp: Fix CONFIG_DYNAMIC_DEBUG=n build
> 0d38c24 IB/srpt: Report login failures only once
> 
> dm related, however I am testing against an sd, not dm device here
> 
> ef548c5 dm flakey: introduce "error_writes" feature
> e99dda8f dm cache policy smq: use hash_32() instead of hash_32_generic()
> 027c431 dm crypt: reject key strings containing whitespace chars
> b446396 dm space map: always set ev if sm_ll_mutate() succeeds
> 0c79ce0 dm space map metadata: skip useless memcpy in
> metadata_ll_init_index()
> 314c25c dm space map metadata: fix 'struct sm_metadata' leak on failed create
> 58fc4fe Documentation: dm raid: define data_offset status field
> 11e2968 dm raid: fix discard support regression
> affa9d2 dm raid: don't allow "write behind" with raid4/5/6
> 54cd640 dm mpath: use hw_handler_params if attached hw_handler is same as
> requested
> c538f6e dm crypt: add ability to use keys from the kernel key retention
> service
> 0637018 dm array: remove a dead assignment in populate_ablock_with_values()
> 6080758 dm ioctl: use offsetof() instead of open-coding it
> b23df0d dm rq: simplify use_blk_mq initialization
> 41c73a4 dm bufio: drop the lock when doing GFP_NOIO allocation
> d12067f dm bufio: don't take the lock in dm_bufio_shrink_count
> 9ea61ca dm bufio: avoid sleeping while holding the dm_bufio lock
> 5b8c01f dm table: simplify dm_table_determine_type()
> 301fc3f dm table: an 'all_blk_mq' table must be loaded for a blk-mq DM device
> 6936c12 dm table: fix 'all_blk_mq' inconsistency when an empty table is
> loaded
> 
> I have time next week so will try narrow it down via bisect by commit
> 
> Thanks
> Laurence
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Started with bisect at > 6000 revisions and 13 cycles between v4.9-rc8 and master.
bisect good for v4.9-rc8, bisect bad for master HEAD

Down to around 700 now, but its late, going to bed now so will finish tomorrow  

Thanks
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Bart Van Assche Dec. 21, 2016, 8:08 a.m. UTC | #11 
T24gV2VkLCAyMDE2LTEyLTIxIGF0IDAxOjM0IC0wNTAwLCBMYXVyZW5jZSBPYmVybWFuIHdyb3Rl
Og0KPiBTdGFydGVkIHdpdGggYmlzZWN0IGF0ID4gNjAwMCByZXZpc2lvbnMgYW5kIDEzIGN5Y2xl
cyBiZXR3ZWVuIHY0LjktcmM4IGFuZCBtYXN0ZXIuDQo+IGJpc2VjdCBnb29kIGZvciB2NC45LXJj
OCwgYmlzZWN0IGJhZCBmb3IgbWFzdGVyIEhFQUQNCg0KSGVsbG8gTGF1cmVuY2UsDQoNClRoZSBu
dW1iZXIgb2YgYmxvY2sgbGF5ZXIgY2hhbmdlcyB0aGF0IHdlbnQgaW50byB3aGF0IHdpbGwgYmVj
b21lIDQuMTAtcmMxIGlzDQpodWdlLiBSdW5uaW5nIGEgYmlzZWN0IHNlZW1zIGxpa2UgYSBnb29k
IGlkZWEgdG8gbWUuDQoNCkJhcnQu
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
diff mbox

Patch

diff --git a/drivers/infiniband/hw/hfi1/dma.c b/drivers/infiniband/hw/hfi1/dma.c
deleted file mode 100644
index 7e8dab892848..000000000000
--- a/drivers/infiniband/hw/hfi1/dma.c
+++ /dev/null
@@ -1,183 +0,0 @@ 
-/*
- * Copyright(c) 2015, 2016 Intel Corporation.
- *
- * This file is provided under a dual BSD/GPLv2 license.  When using or
- * redistributing this file, you may do so under either license.
- *
- * GPL LICENSE SUMMARY
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of version 2 of the GNU General Public License as
- * published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful, but
- * WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
- * General Public License for more details.
- *
- * BSD LICENSE
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- *
- *  - Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- *  - Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in
- *    the documentation and/or other materials provided with the
- *    distribution.
- *  - Neither the name of Intel Corporation nor the names of its
- *    contributors may be used to endorse or promote products derived
- *    from this software without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
- * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
- * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
- * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
- * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
- * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
- * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
- * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
- * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
- * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- *
- */
-#include <linux/types.h>
-#include <linux/scatterlist.h>
-
-#include "verbs.h"
-
-#define BAD_DMA_ADDRESS ((u64)0)
-
-/*
- * The following functions implement driver specific replacements
- * for the ib_dma_*() functions.
- *
- * These functions return kernel virtual addresses instead of
- * device bus addresses since the driver uses the CPU to copy
- * data instead of using hardware DMA.
- */
-
-static int hfi1_mapping_error(struct ib_device *dev, u64 dma_addr)
-{
-	return dma_addr == BAD_DMA_ADDRESS;
-}
-
-static u64 hfi1_dma_map_single(struct ib_device *dev, void *cpu_addr,
-			       size_t size, enum dma_data_direction direction)
-{
-	if (WARN_ON(!valid_dma_direction(direction)))
-		return BAD_DMA_ADDRESS;
-
-	return (u64)cpu_addr;
-}
-
-static void hfi1_dma_unmap_single(struct ib_device *dev, u64 addr, size_t size,
-				  enum dma_data_direction direction)
-{
-	/* This is a stub, nothing to be done here */
-}
-
-static u64 hfi1_dma_map_page(struct ib_device *dev, struct page *page,
-			     unsigned long offset, size_t size,
-			    enum dma_data_direction direction)
-{
-	u64 addr;
-
-	if (WARN_ON(!valid_dma_direction(direction)))
-		return BAD_DMA_ADDRESS;
-
-	if (offset + size > PAGE_SIZE)
-		return BAD_DMA_ADDRESS;
-
-	addr = (u64)page_address(page);
-	if (addr)
-		addr += offset;
-
-	return addr;
-}
-
-static void hfi1_dma_unmap_page(struct ib_device *dev, u64 addr, size_t size,
-				enum dma_data_direction direction)
-{
-	/* This is a stub, nothing to be done here */
-}
-
-static int hfi1_map_sg(struct ib_device *dev, struct scatterlist *sgl,
-		       int nents, enum dma_data_direction direction)
-{
-	struct scatterlist *sg;
-	u64 addr;
-	int i;
-	int ret = nents;
-
-	if (WARN_ON(!valid_dma_direction(direction)))
-		return BAD_DMA_ADDRESS;
-
-	for_each_sg(sgl, sg, nents, i) {
-		addr = (u64)page_address(sg_page(sg));
-		if (!addr) {
-			ret = 0;
-			break;
-		}
-		sg->dma_address = addr + sg->offset;
-#ifdef CONFIG_NEED_SG_DMA_LENGTH
-		sg->dma_length = sg->length;
-#endif
-	}
-	return ret;
-}
-
-static void hfi1_unmap_sg(struct ib_device *dev,
-			  struct scatterlist *sg, int nents,
-			 enum dma_data_direction direction)
-{
-	/* This is a stub, nothing to be done here */
-}
-
-static void hfi1_sync_single_for_cpu(struct ib_device *dev, u64 addr,
-				     size_t size, enum dma_data_direction dir)
-{
-}
-
-static void hfi1_sync_single_for_device(struct ib_device *dev, u64 addr,
-					size_t size,
-					enum dma_data_direction dir)
-{
-}
-
-static void *hfi1_dma_alloc_coherent(struct ib_device *dev, size_t size,
-				     u64 *dma_handle, gfp_t flag)
-{
-	struct page *p;
-	void *addr = NULL;
-
-	p = alloc_pages(flag, get_order(size));
-	if (p)
-		addr = page_address(p);
-	if (dma_handle)
-		*dma_handle = (u64)addr;
-	return addr;
-}
-
-static void hfi1_dma_free_coherent(struct ib_device *dev, size_t size,
-				   void *cpu_addr, u64 dma_handle)
-{
-	free_pages((unsigned long)cpu_addr, get_order(size));
-}
-
-struct ib_dma_mapping_ops hfi1_dma_mapping_ops = {
-	.mapping_error = hfi1_mapping_error,
-	.map_single = hfi1_dma_map_single,
-	.unmap_single = hfi1_dma_unmap_single,
-	.map_page = hfi1_dma_map_page,
-	.unmap_page = hfi1_dma_unmap_page,
-	.map_sg = hfi1_map_sg,
-	.unmap_sg = hfi1_unmap_sg,
-	.sync_single_for_cpu = hfi1_sync_single_for_cpu,
-	.sync_single_for_device = hfi1_sync_single_for_device,
-	.alloc_coherent = hfi1_dma_alloc_coherent,
-	.free_coherent = hfi1_dma_free_coherent
-};
diff --git a/drivers/infiniband/hw/qib/qib_dma.c b/drivers/infiniband/hw/qib/qib_dma.c
deleted file mode 100644
index 59fe092b4b0f..000000000000
--- a/drivers/infiniband/hw/qib/qib_dma.c
+++ /dev/null
@@ -1,169 +0,0 @@ 
-/*
- * Copyright (c) 2006, 2009, 2010 QLogic, Corporation. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses.  You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- *     Redistribution and use in source and binary forms, with or
- *     without modification, are permitted provided that the following
- *     conditions are met:
- *
- *      - Redistributions of source code must retain the above
- *        copyright notice, this list of conditions and the following
- *        disclaimer.
- *
- *      - Redistributions in binary form must reproduce the above
- *        copyright notice, this list of conditions and the following
- *        disclaimer in the documentation and/or other materials
- *        provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-#include <linux/types.h>
-#include <linux/scatterlist.h>
-
-#include "qib_verbs.h"
-
-#define BAD_DMA_ADDRESS ((u64) 0)
-
-/*
- * The following functions implement driver specific replacements
- * for the ib_dma_*() functions.
- *
- * These functions return kernel virtual addresses instead of
- * device bus addresses since the driver uses the CPU to copy
- * data instead of using hardware DMA.
- */
-
-static int qib_mapping_error(struct ib_device *dev, u64 dma_addr)
-{
-	return dma_addr == BAD_DMA_ADDRESS;
-}
-
-static u64 qib_dma_map_single(struct ib_device *dev, void *cpu_addr,
-			      size_t size, enum dma_data_direction direction)
-{
-	BUG_ON(!valid_dma_direction(direction));
-	return (u64) cpu_addr;
-}
-
-static void qib_dma_unmap_single(struct ib_device *dev, u64 addr, size_t size,
-				 enum dma_data_direction direction)
-{
-	BUG_ON(!valid_dma_direction(direction));
-}
-
-static u64 qib_dma_map_page(struct ib_device *dev, struct page *page,
-			    unsigned long offset, size_t size,
-			    enum dma_data_direction direction)
-{
-	u64 addr;
-
-	BUG_ON(!valid_dma_direction(direction));
-
-	if (offset + size > PAGE_SIZE) {
-		addr = BAD_DMA_ADDRESS;
-		goto done;
-	}
-
-	addr = (u64) page_address(page);
-	if (addr)
-		addr += offset;
-	/* TODO: handle highmem pages */
-
-done:
-	return addr;
-}
-
-static void qib_dma_unmap_page(struct ib_device *dev, u64 addr, size_t size,
-			       enum dma_data_direction direction)
-{
-	BUG_ON(!valid_dma_direction(direction));
-}
-
-static int qib_map_sg(struct ib_device *dev, struct scatterlist *sgl,
-		      int nents, enum dma_data_direction direction)
-{
-	struct scatterlist *sg;
-	u64 addr;
-	int i;
-	int ret = nents;
-
-	BUG_ON(!valid_dma_direction(direction));
-
-	for_each_sg(sgl, sg, nents, i) {
-		addr = (u64) page_address(sg_page(sg));
-		/* TODO: handle highmem pages */
-		if (!addr) {
-			ret = 0;
-			break;
-		}
-		sg->dma_address = addr + sg->offset;
-#ifdef CONFIG_NEED_SG_DMA_LENGTH
-		sg->dma_length = sg->length;
-#endif
-	}
-	return ret;
-}
-
-static void qib_unmap_sg(struct ib_device *dev,
-			 struct scatterlist *sg, int nents,
-			 enum dma_data_direction direction)
-{
-	BUG_ON(!valid_dma_direction(direction));
-}
-
-static void qib_sync_single_for_cpu(struct ib_device *dev, u64 addr,
-				    size_t size, enum dma_data_direction dir)
-{
-}
-
-static void qib_sync_single_for_device(struct ib_device *dev, u64 addr,
-				       size_t size,
-				       enum dma_data_direction dir)
-{
-}
-
-static void *qib_dma_alloc_coherent(struct ib_device *dev, size_t size,
-				    u64 *dma_handle, gfp_t flag)
-{
-	struct page *p;
-	void *addr = NULL;
-
-	p = alloc_pages(flag, get_order(size));
-	if (p)
-		addr = page_address(p);
-	if (dma_handle)
-		*dma_handle = (u64) addr;
-	return addr;
-}
-
-static void qib_dma_free_coherent(struct ib_device *dev, size_t size,
-				  void *cpu_addr, u64 dma_handle)
-{
-	free_pages((unsigned long) cpu_addr, get_order(size));
-}
-
-struct ib_dma_mapping_ops qib_dma_mapping_ops = {
-	.mapping_error = qib_mapping_error,
-	.map_single = qib_dma_map_single,
-	.unmap_single = qib_dma_unmap_single,
-	.map_page = qib_dma_map_page,
-	.unmap_page = qib_dma_unmap_page,
-	.map_sg = qib_map_sg,
-	.unmap_sg = qib_unmap_sg,
-	.sync_single_for_cpu = qib_sync_single_for_cpu,
-	.sync_single_for_device = qib_sync_single_for_device,
-	.alloc_coherent = qib_dma_alloc_coherent,
-	.free_coherent = qib_dma_free_coherent
-};
diff --git a/drivers/infiniband/hw/qib/qib_keys.c b/drivers/infiniband/hw/qib/qib_keys.c
index 2c3c93572c17..d2f64a03733e 100644
--- a/drivers/infiniband/hw/qib/qib_keys.c
+++ b/drivers/infiniband/hw/qib/qib_keys.c
@@ -160,7 +160,7 @@  int qib_rkey_ok(struct rvt_qp *qp, struct rvt_sge *sge,
 
 	/*
 	 * We use RKEY == zero for kernel virtual addresses
-	 * (see qib_get_dma_mr and qib_dma.c).
+	 * (see qib_get_dma_mr).
 	 */
 	rcu_read_lock();
 	if (rkey == 0) {
diff --git a/drivers/infiniband/sw/rdmavt/Makefile b/drivers/infiniband/sw/rdmavt/Makefile
index ccaa7992ac97..2a821d2fb569 100644
--- a/drivers/infiniband/sw/rdmavt/Makefile
+++ b/drivers/infiniband/sw/rdmavt/Makefile
@@ -7,7 +7,7 @@ 
 #
 obj-$(CONFIG_INFINIBAND_RDMAVT) += rdmavt.o
 
-rdmavt-y := vt.o ah.o cq.o dma.o mad.o mcast.o mmap.o mr.o pd.o qp.o srq.o \
+rdmavt-y := vt.o ah.o cq.o mad.o mcast.o mmap.o mr.o pd.o qp.o srq.o \
 	trace.o
 
 CFLAGS_trace.o = -I$(src)
diff --git a/drivers/infiniband/sw/rdmavt/dma.c b/drivers/infiniband/sw/rdmavt/dma.c
deleted file mode 100644
index f2cefb0d9180..000000000000
--- a/drivers/infiniband/sw/rdmavt/dma.c
+++ /dev/null
@@ -1,198 +0,0 @@ 
-/*
- * Copyright(c) 2016 Intel Corporation.
- *
- * This file is provided under a dual BSD/GPLv2 license.  When using or
- * redistributing this file, you may do so under either license.
- *
- * GPL LICENSE SUMMARY
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of version 2 of the GNU General Public License as
- * published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful, but
- * WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
- * General Public License for more details.
- *
- * BSD LICENSE
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- *
- *  - Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- *  - Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in
- *    the documentation and/or other materials provided with the
- *    distribution.
- *  - Neither the name of Intel Corporation nor the names of its
- *    contributors may be used to endorse or promote products derived
- *    from this software without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
- * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
- * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
- * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
- * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
- * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
- * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
- * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
- * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
- * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- *
- */
-#include <linux/types.h>
-#include <linux/scatterlist.h>
-#include <rdma/ib_verbs.h>
-
-#include "dma.h"
-
-#define BAD_DMA_ADDRESS ((u64)0)
-
-/*
- * The following functions implement driver specific replacements
- * for the ib_dma_*() functions.
- *
- * These functions return kernel virtual addresses instead of
- * device bus addresses since the driver uses the CPU to copy
- * data instead of using hardware DMA.
- */
-
-static int rvt_mapping_error(struct ib_device *dev, u64 dma_addr)
-{
-	return dma_addr == BAD_DMA_ADDRESS;
-}
-
-static u64 rvt_dma_map_single(struct ib_device *dev, void *cpu_addr,
-			      size_t size, enum dma_data_direction direction)
-{
-	if (WARN_ON(!valid_dma_direction(direction)))
-		return BAD_DMA_ADDRESS;
-
-	return (u64)cpu_addr;
-}
-
-static void rvt_dma_unmap_single(struct ib_device *dev, u64 addr, size_t size,
-				 enum dma_data_direction direction)
-{
-	/* This is a stub, nothing to be done here */
-}
-
-static u64 rvt_dma_map_page(struct ib_device *dev, struct page *page,
-			    unsigned long offset, size_t size,
-			    enum dma_data_direction direction)
-{
-	u64 addr;
-
-	if (WARN_ON(!valid_dma_direction(direction)))
-		return BAD_DMA_ADDRESS;
-
-	addr = (u64)page_address(page);
-	if (addr)
-		addr += offset;
-
-	return addr;
-}
-
-static void rvt_dma_unmap_page(struct ib_device *dev, u64 addr, size_t size,
-			       enum dma_data_direction direction)
-{
-	/* This is a stub, nothing to be done here */
-}
-
-static int rvt_map_sg(struct ib_device *dev, struct scatterlist *sgl,
-		      int nents, enum dma_data_direction direction)
-{
-	struct scatterlist *sg;
-	u64 addr;
-	int i;
-	int ret = nents;
-
-	if (WARN_ON(!valid_dma_direction(direction)))
-		return 0;
-
-	for_each_sg(sgl, sg, nents, i) {
-		addr = (u64)page_address(sg_page(sg));
-		if (!addr) {
-			ret = 0;
-			break;
-		}
-		sg->dma_address = addr + sg->offset;
-#ifdef CONFIG_NEED_SG_DMA_LENGTH
-		sg->dma_length = sg->length;
-#endif
-	}
-	return ret;
-}
-
-static void rvt_unmap_sg(struct ib_device *dev,
-			 struct scatterlist *sg, int nents,
-			 enum dma_data_direction direction)
-{
-	/* This is a stub, nothing to be done here */
-}
-
-static int rvt_map_sg_attrs(struct ib_device *dev, struct scatterlist *sgl,
-			    int nents, enum dma_data_direction direction,
-			    unsigned long attrs)
-{
-	return rvt_map_sg(dev, sgl, nents, direction);
-}
-
-static void rvt_unmap_sg_attrs(struct ib_device *dev,
-			       struct scatterlist *sg, int nents,
-			       enum dma_data_direction direction,
-			       unsigned long attrs)
-{
-	return rvt_unmap_sg(dev, sg, nents, direction);
-}
-
-static void rvt_sync_single_for_cpu(struct ib_device *dev, u64 addr,
-				    size_t size, enum dma_data_direction dir)
-{
-}
-
-static void rvt_sync_single_for_device(struct ib_device *dev, u64 addr,
-				       size_t size,
-				       enum dma_data_direction dir)
-{
-}
-
-static void *rvt_dma_alloc_coherent(struct ib_device *dev, size_t size,
-				    u64 *dma_handle, gfp_t flag)
-{
-	struct page *p;
-	void *addr = NULL;
-
-	p = alloc_pages(flag, get_order(size));
-	if (p)
-		addr = page_address(p);
-	if (dma_handle)
-		*dma_handle = (u64)addr;
-	return addr;
-}
-
-static void rvt_dma_free_coherent(struct ib_device *dev, size_t size,
-				  void *cpu_addr, u64 dma_handle)
-{
-	free_pages((unsigned long)cpu_addr, get_order(size));
-}
-
-struct ib_dma_mapping_ops rvt_default_dma_mapping_ops = {
-	.mapping_error = rvt_mapping_error,
-	.map_single = rvt_dma_map_single,
-	.unmap_single = rvt_dma_unmap_single,
-	.map_page = rvt_dma_map_page,
-	.unmap_page = rvt_dma_unmap_page,
-	.map_sg = rvt_map_sg,
-	.unmap_sg = rvt_unmap_sg,
-	.map_sg_attrs = rvt_map_sg_attrs,
-	.unmap_sg_attrs = rvt_unmap_sg_attrs,
-	.sync_single_for_cpu = rvt_sync_single_for_cpu,
-	.sync_single_for_device = rvt_sync_single_for_device,
-	.alloc_coherent = rvt_dma_alloc_coherent,
-	.free_coherent = rvt_dma_free_coherent
-};
diff --git a/drivers/infiniband/sw/rdmavt/dma.h b/drivers/infiniband/sw/rdmavt/dma.h
deleted file mode 100644
index 979f07e09195..000000000000
--- a/drivers/infiniband/sw/rdmavt/dma.h
+++ /dev/null
@@ -1,53 +0,0 @@ 
-#ifndef DEF_RDMAVTDMA_H
-#define DEF_RDMAVTDMA_H
-
-/*
- * Copyright(c) 2016 Intel Corporation.
- *
- * This file is provided under a dual BSD/GPLv2 license.  When using or
- * redistributing this file, you may do so under either license.
- *
- * GPL LICENSE SUMMARY
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of version 2 of the GNU General Public License as
- * published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful, but
- * WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
- * General Public License for more details.
- *
- * BSD LICENSE
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- *
- *  - Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- *  - Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in
- *    the documentation and/or other materials provided with the
- *    distribution.
- *  - Neither the name of Intel Corporation nor the names of its
- *    contributors may be used to endorse or promote products derived
- *    from this software without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
- * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
- * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
- * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
- * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
- * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
- * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
- * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
- * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
- * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- *
- */
-
-extern struct ib_dma_mapping_ops rvt_default_dma_mapping_ops;
-
-#endif          /* DEF_RDMAVTDMA_H */
diff --git a/drivers/infiniband/sw/rdmavt/mr.c b/drivers/infiniband/sw/rdmavt/mr.c
index 46b64970058e..e12f1345bbab 100644
--- a/drivers/infiniband/sw/rdmavt/mr.c
+++ b/drivers/infiniband/sw/rdmavt/mr.c
@@ -303,8 +303,8 @@  static void __rvt_free_mr(struct rvt_mr *mr)
  * @acc: access flags
  *
  * Return: the memory region on success, otherwise returns an errno.
- * Note that all DMA addresses should be created via the
- * struct ib_dma_mapping_ops functions (see dma.c).
+ * Note that all DMA addresses should be created via the functions in
+ * struct dma_noop_ops.
  */
 struct ib_mr *rvt_get_dma_mr(struct ib_pd *pd, int acc)
 {
diff --git a/drivers/infiniband/sw/rdmavt/vt.c b/drivers/infiniband/sw/rdmavt/vt.c
index d430c2f7cec4..1014a813a942 100644
--- a/drivers/infiniband/sw/rdmavt/vt.c
+++ b/drivers/infiniband/sw/rdmavt/vt.c
@@ -777,8 +777,8 @@  int rvt_register_device(struct rvt_dev_info *rdi)
 	}
 
 	/* DMA Operations */
-	rdi->ibdev.dma_ops =
-		rdi->ibdev.dma_ops ? : &rvt_default_dma_mapping_ops;
+	if (rdi->ibdev.dma_device->dma_ops == NULL)
+		set_dma_ops(rdi->ibdev.dma_device, &dma_noop_ops);
 
 	/* Protection Domain */
 	spin_lock_init(&rdi->n_pds_lock);
diff --git a/drivers/infiniband/sw/rdmavt/vt.h b/drivers/infiniband/sw/rdmavt/vt.h
index 6b01eaa4461b..f363505312be 100644
--- a/drivers/infiniband/sw/rdmavt/vt.h
+++ b/drivers/infiniband/sw/rdmavt/vt.h
@@ -50,7 +50,6 @@ 
 
 #include <rdma/rdma_vt.h>
 #include <linux/pci.h>
-#include "dma.h"
 #include "pd.h"
 #include "qp.h"
 #include "ah.h"
diff --git a/drivers/infiniband/sw/rxe/Makefile b/drivers/infiniband/sw/rxe/Makefile
index 3b3fb9d1c470..ec35ff022a42 100644
--- a/drivers/infiniband/sw/rxe/Makefile
+++ b/drivers/infiniband/sw/rxe/Makefile
@@ -14,7 +14,6 @@  rdma_rxe-y := \
 	rxe_qp.o \
 	rxe_cq.o \
 	rxe_mr.o \
-	rxe_dma.o \
 	rxe_opcode.o \
 	rxe_mmap.o \
 	rxe_icrc.o \
diff --git a/drivers/infiniband/sw/rxe/rxe_dma.c b/drivers/infiniband/sw/rxe/rxe_dma.c
deleted file mode 100644
index a0f8af5851ae..000000000000
--- a/drivers/infiniband/sw/rxe/rxe_dma.c
+++ /dev/null
@@ -1,183 +0,0 @@ 
-/*
- * Copyright (c) 2016 Mellanox Technologies Ltd. All rights reserved.
- * Copyright (c) 2015 System Fabric Works, Inc. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses.  You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- *     Redistribution and use in source and binary forms, with or
- *     without modification, are permitted provided that the following
- *     conditions are met:
- *
- *	- Redistributions of source code must retain the above
- *	  copyright notice, this list of conditions and the following
- *	  disclaimer.
- *
- *	- Redistributions in binary form must reproduce the above
- *	  copyright notice, this list of conditions and the following
- *	  disclaimer in the documentation and/or other materials
- *	  provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#include "rxe.h"
-#include "rxe_loc.h"
-
-#define DMA_BAD_ADDER ((u64)0)
-
-static int rxe_mapping_error(struct ib_device *dev, u64 dma_addr)
-{
-	return dma_addr == DMA_BAD_ADDER;
-}
-
-static u64 rxe_dma_map_single(struct ib_device *dev,
-			      void *cpu_addr, size_t size,
-			      enum dma_data_direction direction)
-{
-	WARN_ON(!valid_dma_direction(direction));
-	return (uintptr_t)cpu_addr;
-}
-
-static void rxe_dma_unmap_single(struct ib_device *dev,
-				 u64 addr, size_t size,
-				 enum dma_data_direction direction)
-{
-	WARN_ON(!valid_dma_direction(direction));
-}
-
-static u64 rxe_dma_map_page(struct ib_device *dev,
-			    struct page *page,
-			    unsigned long offset,
-			    size_t size, enum dma_data_direction direction)
-{
-	u64 addr;
-
-	WARN_ON(!valid_dma_direction(direction));
-
-	if (offset + size > PAGE_SIZE) {
-		addr = DMA_BAD_ADDER;
-		goto done;
-	}
-
-	addr = (uintptr_t)page_address(page);
-	if (addr)
-		addr += offset;
-
-done:
-	return addr;
-}
-
-static void rxe_dma_unmap_page(struct ib_device *dev,
-			       u64 addr, size_t size,
-			       enum dma_data_direction direction)
-{
-	WARN_ON(!valid_dma_direction(direction));
-}
-
-static int rxe_map_sg(struct ib_device *dev, struct scatterlist *sgl,
-		      int nents, enum dma_data_direction direction)
-{
-	struct scatterlist *sg;
-	u64 addr;
-	int i;
-	int ret = nents;
-
-	WARN_ON(!valid_dma_direction(direction));
-
-	for_each_sg(sgl, sg, nents, i) {
-		addr = (uintptr_t)page_address(sg_page(sg));
-		if (!addr) {
-			ret = 0;
-			break;
-		}
-		sg->dma_address = addr + sg->offset;
-#ifdef CONFIG_NEED_SG_DMA_LENGTH
-		sg->dma_length = sg->length;
-#endif
-	}
-
-	return ret;
-}
-
-static void rxe_unmap_sg(struct ib_device *dev,
-			 struct scatterlist *sg, int nents,
-			 enum dma_data_direction direction)
-{
-	WARN_ON(!valid_dma_direction(direction));
-}
-
-static int rxe_map_sg_attrs(struct ib_device *dev, struct scatterlist *sgl,
-			    int nents, enum dma_data_direction direction,
-			    unsigned long attrs)
-{
-	return rxe_map_sg(dev, sgl, nents, direction);
-}
-
-static void rxe_unmap_sg_attrs(struct ib_device *dev,
-			       struct scatterlist *sg, int nents,
-			       enum dma_data_direction direction,
-			       unsigned long attrs)
-{
-	rxe_unmap_sg(dev, sg, nents, direction);
-}
-
-static void rxe_sync_single_for_cpu(struct ib_device *dev,
-				    u64 addr,
-				    size_t size, enum dma_data_direction dir)
-{
-}
-
-static void rxe_sync_single_for_device(struct ib_device *dev,
-				       u64 addr,
-				       size_t size, enum dma_data_direction dir)
-{
-}
-
-static void *rxe_dma_alloc_coherent(struct ib_device *dev, size_t size,
-				    u64 *dma_handle, gfp_t flag)
-{
-	struct page *p;
-	void *addr = NULL;
-
-	p = alloc_pages(flag, get_order(size));
-	if (p)
-		addr = page_address(p);
-
-	if (dma_handle)
-		*dma_handle = (uintptr_t)addr;
-
-	return addr;
-}
-
-static void rxe_dma_free_coherent(struct ib_device *dev, size_t size,
-				  void *cpu_addr, u64 dma_handle)
-{
-	free_pages((unsigned long)cpu_addr, get_order(size));
-}
-
-struct ib_dma_mapping_ops rxe_dma_mapping_ops = {
-	.mapping_error		= rxe_mapping_error,
-	.map_single		= rxe_dma_map_single,
-	.unmap_single		= rxe_dma_unmap_single,
-	.map_page		= rxe_dma_map_page,
-	.unmap_page		= rxe_dma_unmap_page,
-	.map_sg			= rxe_map_sg,
-	.unmap_sg		= rxe_unmap_sg,
-	.map_sg_attrs		= rxe_map_sg_attrs,
-	.unmap_sg_attrs		= rxe_unmap_sg_attrs,
-	.sync_single_for_cpu	= rxe_sync_single_for_cpu,
-	.sync_single_for_device	= rxe_sync_single_for_device,
-	.alloc_coherent		= rxe_dma_alloc_coherent,
-	.free_coherent		= rxe_dma_free_coherent
-};
diff --git a/drivers/infiniband/sw/rxe/rxe_loc.h b/drivers/infiniband/sw/rxe/rxe_loc.h
index 73849a5a91b3..a075023332dc 100644
--- a/drivers/infiniband/sw/rxe/rxe_loc.h
+++ b/drivers/infiniband/sw/rxe/rxe_loc.h
@@ -221,8 +221,6 @@  int rxe_srq_from_attr(struct rxe_dev *rxe, struct rxe_srq *srq,
 		      struct ib_srq_attr *attr, enum ib_srq_attr_mask mask,
 		      struct ib_udata *udata);
 
-extern struct ib_dma_mapping_ops rxe_dma_mapping_ops;
-
 void rxe_release(struct kref *kref);
 
 int rxe_completer(void *arg);
diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c
index 19841c863daf..c4a5154928b7 100644
--- a/drivers/infiniband/sw/rxe/rxe_verbs.c
+++ b/drivers/infiniband/sw/rxe/rxe_verbs.c
@@ -1226,7 +1226,7 @@  int rxe_register_device(struct rxe_dev *rxe)
 	dev->dma_device = rxe->ifc_ops->dma_device(rxe);
 	dev->local_dma_lkey = 0;
 	dev->node_guid = rxe->ifc_ops->node_guid(rxe);
-	dev->dma_ops = &rxe_dma_mapping_ops;
+	set_dma_ops(dev->dma_device, &dma_noop_ops);
 
 	dev->uverbs_abi_ver = RXE_UVERBS_ABI_VERSION;
 	dev->uverbs_cmd_mask = BIT_ULL(IB_USER_VERBS_CMD_GET_CONTEXT)
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 5ad43a487745..663a28f37570 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -1763,53 +1763,6 @@  struct ib_cache {
 	u8                     *lmc_cache;
 };
 
-struct ib_dma_mapping_ops {
-	int		(*mapping_error)(struct ib_device *dev,
-					 u64 dma_addr);
-	u64		(*map_single)(struct ib_device *dev,
-				      void *ptr, size_t size,
-				      enum dma_data_direction direction);
-	void		(*unmap_single)(struct ib_device *dev,
-					u64 addr, size_t size,
-					enum dma_data_direction direction);
-	u64		(*map_page)(struct ib_device *dev,
-				    struct page *page, unsigned long offset,
-				    size_t size,
-				    enum dma_data_direction direction);
-	void		(*unmap_page)(struct ib_device *dev,
-				      u64 addr, size_t size,
-				      enum dma_data_direction direction);
-	int		(*map_sg)(struct ib_device *dev,
-				  struct scatterlist *sg, int nents,
-				  enum dma_data_direction direction);
-	void		(*unmap_sg)(struct ib_device *dev,
-				    struct scatterlist *sg, int nents,
-				    enum dma_data_direction direction);
-	int		(*map_sg_attrs)(struct ib_device *dev,
-					struct scatterlist *sg, int nents,
-					enum dma_data_direction direction,
-					unsigned long attrs);
-	void		(*unmap_sg_attrs)(struct ib_device *dev,
-					  struct scatterlist *sg, int nents,
-					  enum dma_data_direction direction,
-					  unsigned long attrs);
-	void		(*sync_single_for_cpu)(struct ib_device *dev,
-					       u64 dma_handle,
-					       size_t size,
-					       enum dma_data_direction dir);
-	void		(*sync_single_for_device)(struct ib_device *dev,
-						  u64 dma_handle,
-						  size_t size,
-						  enum dma_data_direction dir);
-	void		*(*alloc_coherent)(struct ib_device *dev,
-					   size_t size,
-					   u64 *dma_handle,
-					   gfp_t flag);
-	void		(*free_coherent)(struct ib_device *dev,
-					 size_t size, void *cpu_addr,
-					 u64 dma_handle);
-};
-
 struct iw_cm_verbs;
 
 struct ib_port_immutable {
@@ -2070,7 +2023,6 @@  struct ib_device {
 							   struct ib_rwq_ind_table_init_attr *init_attr,
 							   struct ib_udata *udata);
 	int                        (*destroy_rwq_ind_table)(struct ib_rwq_ind_table *wq_ind_table);
-	struct ib_dma_mapping_ops   *dma_ops;
 
 	struct module               *owner;
 	struct device                dev;
@@ -2927,8 +2879,6 @@  static inline int ib_req_ncomp_notif(struct ib_cq *cq, int wc_cnt)
  */
 static inline int ib_dma_mapping_error(struct ib_device *dev, u64 dma_addr)
 {
-	if (dev->dma_ops)
-		return dev->dma_ops->mapping_error(dev, dma_addr);
 	return dma_mapping_error(dev->dma_device, dma_addr);
 }
 
@@ -2943,8 +2893,6 @@  static inline u64 ib_dma_map_single(struct ib_device *dev,
 				    void *cpu_addr, size_t size,
 				    enum dma_data_direction direction)
 {
-	if (dev->dma_ops)
-		return dev->dma_ops->map_single(dev, cpu_addr, size, direction);
 	return dma_map_single(dev->dma_device, cpu_addr, size, direction);
 }
 
@@ -2959,10 +2907,7 @@  static inline void ib_dma_unmap_single(struct ib_device *dev,
 				       u64 addr, size_t size,
 				       enum dma_data_direction direction)
 {
-	if (dev->dma_ops)
-		dev->dma_ops->unmap_single(dev, addr, size, direction);
-	else
-		dma_unmap_single(dev->dma_device, addr, size, direction);
+	dma_unmap_single(dev->dma_device, addr, size, direction);
 }
 
 static inline u64 ib_dma_map_single_attrs(struct ib_device *dev,
@@ -2997,8 +2942,6 @@  static inline u64 ib_dma_map_page(struct ib_device *dev,
 				  size_t size,
 					 enum dma_data_direction direction)
 {
-	if (dev->dma_ops)
-		return dev->dma_ops->map_page(dev, page, offset, size, direction);
 	return dma_map_page(dev->dma_device, page, offset, size, direction);
 }
 
@@ -3013,10 +2956,7 @@  static inline void ib_dma_unmap_page(struct ib_device *dev,
 				     u64 addr, size_t size,
 				     enum dma_data_direction direction)
 {
-	if (dev->dma_ops)
-		dev->dma_ops->unmap_page(dev, addr, size, direction);
-	else
-		dma_unmap_page(dev->dma_device, addr, size, direction);
+	dma_unmap_page(dev->dma_device, addr, size, direction);
 }
 
 /**
@@ -3030,8 +2970,6 @@  static inline int ib_dma_map_sg(struct ib_device *dev,
 				struct scatterlist *sg, int nents,
 				enum dma_data_direction direction)
 {
-	if (dev->dma_ops)
-		return dev->dma_ops->map_sg(dev, sg, nents, direction);
 	return dma_map_sg(dev->dma_device, sg, nents, direction);
 }
 
@@ -3046,10 +2984,7 @@  static inline void ib_dma_unmap_sg(struct ib_device *dev,
 				   struct scatterlist *sg, int nents,
 				   enum dma_data_direction direction)
 {
-	if (dev->dma_ops)
-		dev->dma_ops->unmap_sg(dev, sg, nents, direction);
-	else
-		dma_unmap_sg(dev->dma_device, sg, nents, direction);
+	dma_unmap_sg(dev->dma_device, sg, nents, direction);
 }
 
 static inline int ib_dma_map_sg_attrs(struct ib_device *dev,
@@ -3057,12 +2992,8 @@  static inline int ib_dma_map_sg_attrs(struct ib_device *dev,
 				      enum dma_data_direction direction,
 				      unsigned long dma_attrs)
 {
-	if (dev->dma_ops)
-		return dev->dma_ops->map_sg_attrs(dev, sg, nents, direction,
-						  dma_attrs);
-	else
-		return dma_map_sg_attrs(dev->dma_device, sg, nents, direction,
-					dma_attrs);
+	return dma_map_sg_attrs(dev->dma_device, sg, nents, direction,
+				dma_attrs);
 }
 
 static inline void ib_dma_unmap_sg_attrs(struct ib_device *dev,
@@ -3070,12 +3001,7 @@  static inline void ib_dma_unmap_sg_attrs(struct ib_device *dev,
 					 enum dma_data_direction direction,
 					 unsigned long dma_attrs)
 {
-	if (dev->dma_ops)
-		return dev->dma_ops->unmap_sg_attrs(dev, sg, nents, direction,
-						  dma_attrs);
-	else
-		dma_unmap_sg_attrs(dev->dma_device, sg, nents, direction,
-				   dma_attrs);
+	dma_unmap_sg_attrs(dev->dma_device, sg, nents, direction, dma_attrs);
 }
 /**
  * ib_sg_dma_address - Return the DMA address from a scatter/gather entry
@@ -3117,10 +3043,7 @@  static inline void ib_dma_sync_single_for_cpu(struct ib_device *dev,
 					      size_t size,
 					      enum dma_data_direction dir)
 {
-	if (dev->dma_ops)
-		dev->dma_ops->sync_single_for_cpu(dev, addr, size, dir);
-	else
-		dma_sync_single_for_cpu(dev->dma_device, addr, size, dir);
+	dma_sync_single_for_cpu(dev->dma_device, addr, size, dir);
 }
 
 /**
@@ -3135,10 +3058,7 @@  static inline void ib_dma_sync_single_for_device(struct ib_device *dev,
 						 size_t size,
 						 enum dma_data_direction dir)
 {
-	if (dev->dma_ops)
-		dev->dma_ops->sync_single_for_device(dev, addr, size, dir);
-	else
-		dma_sync_single_for_device(dev->dma_device, addr, size, dir);
+	dma_sync_single_for_device(dev->dma_device, addr, size, dir);
 }
 
 /**
@@ -3153,16 +3073,12 @@  static inline void *ib_dma_alloc_coherent(struct ib_device *dev,
 					   u64 *dma_handle,
 					   gfp_t flag)
 {
-	if (dev->dma_ops)
-		return dev->dma_ops->alloc_coherent(dev, size, dma_handle, flag);
-	else {
-		dma_addr_t handle;
-		void *ret;
-
-		ret = dma_alloc_coherent(dev->dma_device, size, &handle, flag);
-		*dma_handle = handle;
-		return ret;
-	}
+	dma_addr_t handle;
+	void *ret;
+
+	ret = dma_alloc_coherent(dev->dma_device, size, &handle, flag);
+	*dma_handle = handle;
+	return ret;
 }
 
 /**
@@ -3176,10 +3092,7 @@  static inline void ib_dma_free_coherent(struct ib_device *dev,
 					size_t size, void *cpu_addr,
 					u64 dma_handle)
 {
-	if (dev->dma_ops)
-		dev->dma_ops->free_coherent(dev, size, cpu_addr, dma_handle);
-	else
-		dma_free_coherent(dev->dma_device, size, cpu_addr, dma_handle);
+	dma_free_coherent(dev->dma_device, size, cpu_addr, dma_handle);
 }
 
 /**