[OmniOS-discuss] questions

Ian Kaufman ikaufman at eng.ucsd.edu
Thu Sep 14 15:07:06 UTC 2017


Some other things you need to take into account:

QDR Infiniband is 40Gbps, not 40GB/s. That is a factor of 8 difference.
That is also a theoretical maximum throughput, there is some overhead. In
reality, you will never see 40Gbps.

My system tested out at 6Gbps - 8Gbps using NFS over IPoIB, with DDR
(20Gbps) nodes and a QDR (40Gbps) storage server. IPoIB drops the
theoretical max rates to 18Gbps and 36Gbps respectively.

If you are getting 185MB/s, you are seeing 1.48Gbps.

Keep your B's and b's straight. Did you play with your frame size at all?

Ian

On Thu, Sep 14, 2017 at 7:10 AM, Jim Klimov <jimklimov at cos.ru> wrote:

> On September 14, 2017 2:26:13 PM GMT+02:00, Dirk Willems <
> dirk.willems at exitas.be> wrote:
> >Hello,
> >
> >
> >I'm trying to understand something let me explain.
> >
> >
> >Oracle always told to me that if you create a etherstub switch it has
> >infiniband speed 40GB/s.
> >
> >But I have a customer running on Solaris (Yeah I know but let me
> >explain) who is copy from 1 NGZ to another NGZ on the same GZ over Lan
> >(I know told him to to use etherstub).
> >
> >The copy witch is performed for a Oracle database with sql command, the
> >
> >DBA witch have 5 streams say it's waiting on the disk, the disk are 50
> >-
> >60 % busy the speed is 30 mb/s.
> >
> >
> >So I did some test just to see and understand if it's the database or
> >the system, but with doing my tests I get very confused ???
> >
> >
> >On another Solaris at my work copy over etherstub switch => copy speed
> >is 185MB/s expected much more of infiniband speed ???
> >
> >
> >root at test1:/export/home/Admin# scp test10G
> >Admin at 192.168.1.2:/export/home/Admin/
> >Password:
> >test10G              100%
> >|****************************************************************|
> >10240
> >MB    00:59
> >
> >
> >root at test2:~# dlstat -i 2
> >
> >  LINK    IPKTS   RBYTES    OPKTS   OBYTES
> >            net1   25.76K 185.14M 10.08K    2.62M
> >            net1   27.04K  187.16M   11.23K    3.22M
> >            net1   26.97K  186.37M   11.24K    3.23M
> >            net1   26.63K  187.67M   10.82K    2.99M
> >            net1   27.94K  186.65M   12.17K    3.75M
> >            net1   27.45K  187.46M   11.70K    3.47M
> >            net1   26.01K  181.95M   10.63K    2.99M
> >            net1   27.95K  188.19M   12.14K    3.69M
> >            net1   27.91K  188.36M   12.08K    3.64M
> >
> >The disks are all separate luns with all separated pools => disk are 20
> >
> >- 30% busy
> >
> >
> >On my OmniOSce at my lab over etherstub
> >
> >
> >root at GNUHealth:~# scp test10G witte at 192.168.20.3:/export/home/witte/
> >Password:
> >test10G 76% 7853MB 116.4MB/s
> >
> >
> >=> copy is 116.4 MB/s => expected much more from infiniband speed is
> >just the same as Lan ???
> >
> >
> >Is not that my disk can not follow 17% busy there sleeping ...
> >
> >    extended device statistics
> >     r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
> >     0,0  248,4    0,0    2,1  0,0  1,3    0,0    5,3   0 102 c1
> >    0,0   37,5    0,0    0,7  0,0  0,2    0,0    4,7   0  17 c1t0d0 =>
> >rpool
> >    0,0   38,5    0,0    0,7  0,0  0,2    0,0    4,9   0  17 c1t1d0 =>
> >rpool
> >    0,0   40,5    0,0    0,1  0,0  0,2    0,0    5,6   0  17 c1t2d0 =>
> >data pool
> >    0,0   43,5    0,0    0,2  0,0  0,2    0,0    5,4   0  17 c1t3d0 =>
> >data pool
> >    0,0   44,5    0,0    0,2  0,0  0,2    0,0    5,5   0  18 c1t4d0 =>
> >data pool
> >    0,0   44,0    0,0    0,2  0,0  0,2    0,0    5,4   0  17 c1t5d0 =>
> >data pool
> >     0,0   76,0    0,0    1,5  7,4  0,4   97,2    4,9  14  18 rpool
> >     0,0  172,4    0,0    0,6  2,0  0,9   11,4    5,5  12  20 DATA
> >
> >
> >
> >root at NGINX:/root# dlstat show-link NGINX1 -i 2
> >
> >  LINK  TYPE      ID  INDEX     PKTS    BYTES
> >          NGINX1    rx   bcast     --        0        0
> >          NGINX1    rx      sw     --        0        0
> >          NGINX1    tx   bcast     --        0        0
> >          NGINX1    tx      sw     --    9.26K  692.00K
> >          NGINX1    rx   local     --   26.00K 216.32M
> >          NGINX1    rx   bcast     --        0        0
> >          NGINX1    rx      sw     --        0        0
> >          NGINX1    tx   bcast     --        0        0
> >          NGINX1    tx      sw     --    7.01K  531.38K
> >          NGINX1    rx   local     --   30.65K 253.73M
> >          NGINX1    rx   bcast     --        0        0
> >          NGINX1    rx      sw     --        0        0
> >          NGINX1    tx   bcast     --        0        0
> >          NGINX1    tx      sw     --    8.95K  669.32K
> >          NGINX1    rx   local     --   29.10K 241.15M
> >
> >
> >On the other NGZ I receive 250MB/s ????
> >
> >
> >- So my question is how comes that the speed is equal to Lan 100MB/s on
> >
> >OmniOSce but i receive 250MB/s ?
> >
> >- Why is etherstub so slow if infiniband speed is 40GB/s ???
> >
> >
> >I'm very confused right now ...
> >
> >
> >And want to know for sure how to understand and see this in the right
> >way, because this customer will be the first customer from my who gonna
> >
> >switch complety over to OmniOSce on production and because this
> >customer
> >is one or the biggest company's in Belgium I really don't want to mess
> >up !!!
> >
> >
> >So any help and clarification will be highly appreciate !!!
> >
> >
> >Thank you very much.
> >
> >
> >Kind Regards,
> >
> >
> >Dirk
>
> I am not sure where the infiniband claim comes from, but copying data disk
> to disk, you involve the slow layers like disk, skewed by faster layers
> like cache of already-read data and delayed writes :)
>
> If you have a wide pipe that you may fill, it doesn't mean you do have the
> means to fill it with a few disks.
>
> To estimate the speeds, try pure UDP streams from process to process (no
> disk), large-packet floodping, etc.
>
> I believe etherstub is not constrained artificially, and defaults to jumbo
> frames. Going to LAN and back can in fact use external hardware (IIRC there
> may be a system option to disable that, not sure) and so is constrained by
> that.
>
> Jim
> --
> Typos courtesy of K-9 Mail on my Android
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>



-- 
Ian Kaufman
Research Systems Administrator
UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170914/0b30fd66/attachment.html>


More information about the OmniOS-discuss mailing list