bullen
Posts: 283
Joined: Sun Apr 28, 2013 2:52 pm

Toshiba 256GB SD have bad latency

Thu Jul 18, 2019 10:20 pm

Has anyone noticed how laggy it feels in the ssh session. Many commands block for several seconds.
Last edited by bullen on Thu Jul 25, 2019 9:41 am, edited 2 times in total.
https://github.com/tinspin/rupy - A tiny Java async HTTP application server.

ejolson
Posts: 3575
Joined: Tue Mar 18, 2014 11:47 am

Re: Performance Buster on Pi 2

Thu Jul 18, 2019 10:36 pm

bullen wrote:
Thu Jul 18, 2019 10:20 pm
Has anyone noticed how laggy it feels in the ssh session. Many commands block for several seconds.
Procede with caution: It may be possible that your SD card is about to die.

I'll be trying the buster image in a few days on a Pi 2B and will report back of anything seems out of the ordinary.

echmain
Posts: 238
Joined: Fri Mar 04, 2016 8:26 pm

Re: Performance Buster on Pi 2

Thu Jul 18, 2019 11:10 pm

I have Buster (Lite) running on a Pi Zero and it seems totally normal when I SSH into it.

bullen
Posts: 283
Joined: Sun Apr 28, 2013 2:52 pm

Re: Performance Buster on Pi 2

Fri Jul 19, 2019 12:13 am

Hm, ok thanks for feedback, I guess the SD cards I bought are not that good then. I bought 6x TOSHIBA 256GB for 40$ each so I guess that might make sense that they are crap... :(

But it's only sometimes, so I'll let things run for a couple of weeks and see how things pan out.
https://github.com/tinspin/rupy - A tiny Java async HTTP application server.

echmain
Posts: 238
Joined: Fri Mar 04, 2016 8:26 pm

Re: Performance Buster on Pi 2

Fri Jul 19, 2019 12:20 am

On my Zero I’m using a Sandisk Ultra A1 (64gb).

bullen
Posts: 283
Joined: Sun Apr 28, 2013 2:52 pm

Re: Performance Buster on Pi 2

Fri Jul 19, 2019 11:17 pm

Ok, I think my router is starting to give way... or something else on the network but I don't know yet... will post back when I replace it.
https://github.com/tinspin/rupy - A tiny Java async HTTP application server.

ejolson
Posts: 3575
Joined: Tue Mar 18, 2014 11:47 am

Re: Performance Buster on Pi 2

Sat Jul 20, 2019 12:18 am

bullen wrote:
Fri Jul 19, 2019 11:17 pm
Ok, I think my router is starting to give way... or something else on the network but I don't know yet... will post back when I replace it.
Maybe it's network cable. Have you tried turning the switch or router off and on again?

bullen
Posts: 283
Joined: Sun Apr 28, 2013 2:52 pm

Re: Performance Buster on Pi 2

Sat Jul 20, 2019 5:38 am

Funny you would say that, I replaced the Ethernet cable yesterday from my main Windows PC because PUBG was lagging out too! But it seems it's actually Windows and/or the Network card on my main PC that is the culprit! Narrowing this down...
https://github.com/tinspin/rupy - A tiny Java async HTTP application server.

andrum99
Posts: 811
Joined: Fri Jul 20, 2012 2:41 pm

Re: Performance Buster on Pi 2

Sat Jul 20, 2019 5:12 pm

The key thing with SD cards is whether they are fake or not. Fake ones may work fine for a while, but won't last as long as genuine cards. Buy from reputable suppliers, and remember that if the price looks too good to be true, then it probably is. There are a lot of fake cards out there. Steer clear of Ebay for SD cards.

User avatar
FTrevorGowen
Forum Moderator
Forum Moderator
Posts: 5086
Joined: Mon Mar 04, 2013 6:12 pm
Location: Bristol, U.K.
Contact: Website

Re: Performance Buster on Pi 2

Sat Jul 20, 2019 5:20 pm

@bullen OOC can you post the output of

Code: Select all

cat /sys/block/mmcblk0/device/cid
(ie. the uSDHC card's CID info.)
From that we can see what it claims to be** (and, maybe, if it's similar to other "problem" cards).
Trev.
** more info. here: http://www.cpmspectrepi.uk/raspberry_pi ... cking.html
Still running Raspbian Jessie on some older Pi's (an A, B1, B2, B+, P2B, 3xP0, P0W) but Stretch on my 2xP3A+, P3B+, P3B, B+, A+ and a B2. See: https://www.cpmspectrepi.uk/raspberry_pi/raspiidx.htm

bullen
Posts: 283
Joined: Sun Apr 28, 2013 2:52 pm

Re: Performance Buster on Pi 2

Sat Jul 20, 2019 6:40 pm

The CID seems legit TOSHIBA 256 manufactured 11/18 and I think the SD cards work fine because if I SSH from another computer it works fine.

So hopefully I just sniped a good deal since the seller is one of the largest in Europe and the price dipped the same day and went up 50% the day after I bought them. Robot price setting?

Edit: Ok, maybe there is something fishy with the SD cards and/or my router because the distributed database is reporting some high latencies too. I think we have to be realistic about flash memory; SLC is way to expensive and sparse, TLC is complete garbage and these MLC (sometimes aMLC offering redundant bits per bit) is the peak of what we will ever get.

So 256GB on a die small enough to fit a microSDXC for 40$ is very impressive (1TB is peak today but they cost 10x for 4x the memory), if the write latencies are all over the place with this density that is less of a problem in my completely async. architecture (24 core parallel cluster) than the bits failing after a short duration and/or after a few writes!

So we'll have to learn this final spec by experience, no manufacturer will come out of the trees and declare defects before the cat is in the bag (if you look at the deflationary pressure the prices are describing the manufacturers will have a hard time surviving as it is!). So far the only problem I have noticed are IO-waits, these might become a problem at high concurrency down the line, but if they are caused by the manufacturing process we don't really have an option but to work around them in software by making the file system async. or increase the threads.
https://github.com/tinspin/rupy - A tiny Java async HTTP application server.

bullen
Posts: 283
Joined: Sun Apr 28, 2013 2:52 pm

Re: Performance Buster on Pi 2

Mon Jul 22, 2019 3:21 pm

Ok, I can now almost 100% say the Toshiba cards are to blame. Don't buy these cards, spend 2x the money on SanDisk instead, unless it's a process problem and then we're out of luck for SD cards larger than 64/128GB.

I will have to go through the very painful process of custom partitioning another 6x batch of more expensive cards so this lesson was rough but; you live, you learn. I just purchased one of each SanDisk 256/128/64GB High Endurance so that we can see if this is a process issue and if smaller cards have less issues.
Last edited by bullen on Mon Jul 22, 2019 7:14 pm, edited 2 times in total.
https://github.com/tinspin/rupy - A tiny Java async HTTP application server.

dickon
Posts: 477
Joined: Sun Dec 09, 2012 3:54 pm
Location: Home, just outside Reading

Re: Performance Buster on Pi 2

Mon Jul 22, 2019 3:59 pm

The other possibility is wifi: if you're doing anything over wifi and there's interference around, you'll often see oddities.

If you ping the device from whatever you're using to access it, check that the times are all reasonably small, and in the 1-20ms range (or so). If you're seeing wild swings from 1ms up to 3000ms or more, your poor TCP stack may be having to reorder all the packets as they arrive, and you'll get a very lumpy experience from anything interactive.

bullen
Posts: 283
Joined: Sun Apr 28, 2013 2:52 pm

Re: Performance Buster on Pi 2

Mon Jul 22, 2019 4:00 pm

I'm building a distributed cluster database, no wifi involved.

Also the reason I know the SD cards are to blame (unless there is some really weird thing with the Buster release) is that I have my old test cluster with 3x SanDisk 32GB cards performing flawlessly on the same network. The Toshiba cluster has double the nodes but performs VERY poorly, one in 5-6 writes across the cluster there is 1 or more nodes with latency above 1000ms, sometimes 5000ms!

The SanDisk clusters I have ALWAYS had 30-35ms write on all nodes without a single hitch for many years.

The latency is so bad on these Toshiba cards that they are simply unusable and should not be on the market:

https://classic.prisjakt.nu/produkt.php?p=4675642
Last edited by bullen on Tue Jul 23, 2019 12:38 am, edited 1 time in total.
https://github.com/tinspin/rupy - A tiny Java async HTTP application server.

User avatar
Imperf3kt
Posts: 2775
Joined: Tue Jun 20, 2017 12:16 am
Location: Australia

Re: Performance Buster on Pi 2

Mon Jul 22, 2019 11:29 pm

bullen wrote:
Fri Jul 19, 2019 12:13 am
Hm, ok thanks for feedback, I guess the SD cards I bought are not that good then. I bought 6x TOSHIBA 256GB for 40$ each so I guess that might make sense that they are crap... :(

But it's only sometimes, so I'll let things run for a couple of weeks and see how things pan out.
This sends alarm bells ringing in my head.
The cards are most likely fake.
Then again though, assuming USD, the retail price is only about $60, so maybe just on sale?
55:55:44:44:4C
52:4C:52:42:41

bullen
Posts: 283
Joined: Sun Apr 28, 2013 2:52 pm

Re: Performance Buster on Pi 2 Edit: Toshiba 256GB SD are bad

Thu Jul 25, 2019 9:38 am

Just a finishing comment on this issue:

The Toshiba cards are legit.
They are 2 orders of magnitude slower in terms of latency than a old 32GB SanDisc.
They are 1 order of magnitude slower in terms of latency than brand new 256/128/64GB SanDisc.

I have contacted Toshiba, and they are not replying after I faced them with the truth, you draw your own conclusions.
The price dump is likely due to economic pressure since they have a flawed product.

Here is the data:

These cards have serious read/write latency sometimes over 5 seconds and very often like every 10th operation at least:

Code: Select all

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
mmcblk0          0.01    0.14      0.39      1.34     0.01     0.06  54.43  29.30    6.81   79.12   0.00    33.73     9.80  15.92   0.24
mmcblk0          0.01    0.24      0.45     15.61     0.02     0.08  54.47  25.76   12.94  113.27   0.02    33.76    66.10  60.26   1.50
mmcblk0          0.01    0.18      0.39      7.33     0.01     0.07  54.39  27.26    8.19  118.54   0.01    33.80    41.63  51.81   0.97

This is the same command from an old 32GB SanDisc:

Code: Select all

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
mmcblk0           0.00     0.05    0.00    0.15     0.01     1.20    15.79     0.00    1.10    3.52    1.10   0.34   0.01

This is from the new 256/128/64GB SanDisc:

Code: Select all

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
mmcblk0          5.33    9.56    204.75   2140.14     7.64     2.90  58.90  23.25    4.29   24.12   0.12    38.43   223.79   7.11  10.59 // 128GB
mmcblk0         11.62    4.95    384.60   1056.78    32.56     0.99  73.71  16.73    3.97   14.14   0.08    33.11   213.67   4.13   6.84 // 256GB
https://github.com/tinspin/rupy - A tiny Java async HTTP application server.

Return to “General discussion”