User avatar
makem2
Posts: 231
Joined: Tue Jul 02, 2013 9:12 am
Location: Essex
Contact: Website

Keep 'losing' one of two HD's

Mon Aug 05, 2013 4:31 pm

I have 2 x 1Tb HD's attached to my Pi, one to each USB port. They are both powered from a 2 amp 4 port adapter which can supply the full 2 amp across all ports or to one port. I feed the drives from 2 ports and feel sure the question of lack of power is not the problem (both 'low power' drives), but stand to be corrected. Each drive has one partition only and it is primary.

Everything works fine for quite a long period, backups are done automatically and cron jobs work overnight backing up changes to the second HD.

However, on random occasions, I 'lose' one HD. It is always the same drive which does not appear in file manager and of course, at that time AllwaySync reports errors. That drive (HDD1) is the primary drive of the pair and it is a little older although very little used, than the other drive.

I have had similar occurrences in Windows and similarly, a reboot brings the drive back.

I was wondering if anyone else had this happen and if a reason was found.

User avatar
RaTTuS
Posts: 10412
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK

Re: Keep 'losing' one of two HD's

Mon Aug 05, 2013 4:32 pm

does dmesg report anythign useful when it has gone awol ?
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

User avatar
makem2
Posts: 231
Joined: Tue Jul 02, 2013 9:12 am
Location: Essex
Contact: Website

Re: Keep 'losing' one of two HD's

Mon Aug 05, 2013 4:52 pm

RaTTuS wrote:does dmesg report anythign useful when it has gone awol ?
I was unaware of that facility but will look next time it happens thanks.

Wow, it does produce a lot of information.

Are you able to tell me how to save a copy to compare when the drive disappears again?

It would be helpful also to know how to save all the information produced on a reboot. I did notice this time after having lost a drive, on reboot I saw a Red entry which was one or two words 'not mounted' I think but I was to slow across the room to read what it referred to. However, both drives are showing normally and working correctly at the moment.

User avatar
RaTTuS
Posts: 10412
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK

Re: Keep 'losing' one of two HD's

Tue Aug 06, 2013 7:28 am

sudo apt-get update && sudo apt-get upgrade
sudo apt-get install pastebinit
dmesg | pastbinit
...
it's possible you have a failing HD
or the connection may drop to it for some reason - you may be able to get away with a re-mount of the drive
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

User avatar
makem2
Posts: 231
Joined: Tue Jul 02, 2013 9:12 am
Location: Essex
Contact: Website

Re: Keep 'losing' one of two HD's

Tue Aug 06, 2013 10:06 am

Thank you that pastebinit suggestion is most useful.

There is a typo in "dmesg | pastbinit", it should read dmesg | pastebinit"

For followers: https://help.ubuntu.com/community/Pastebinit

Is the dmesg the contents of the text I see at boot time?

jdb
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2073
Joined: Thu Jul 11, 2013 2:37 pm

Re: Keep 'losing' one of two HD's

Tue Aug 06, 2013 12:10 pm

dmesg prints out the contents of the kernel log buffer.

The kernel log starts at boot time and will print to the screen (so you can see if anything broke or the machine crashed) until the normal boot process stops the output from printing. The log will still fill up, but only in memory.

The buffer will contain up to 128k of the latest information - old information is dropped.
Rockets are loud.
https://astro-pi.org

User avatar
makem2
Posts: 231
Joined: Tue Jul 02, 2013 9:12 am
Location: Essex
Contact: Website

Re: Keep 'losing' one of two HD's

Tue Aug 06, 2013 12:33 pm

Thank you

fredoll
Posts: 227
Joined: Thu Jan 10, 2013 10:30 am

Re: Keep 'losing' one of two HD's

Tue Aug 06, 2013 12:34 pm

you could install smartmon tools

Code: Select all

sudo apt-get update
sudo apt-get install smartmontools

User avatar
makem2
Posts: 231
Joined: Tue Jul 02, 2013 9:12 am
Location: Essex
Contact: Website

Re: Keep 'losing' one of two HD's

Tue Aug 06, 2013 12:45 pm

Thank you, that looks a very interesting suggestion.

jdb
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2073
Joined: Thu Jul 11, 2013 2:37 pm

Re: Keep 'losing' one of two HD's

Tue Aug 06, 2013 12:49 pm

fredoll wrote:you could install smartmon tools

Code: Select all

sudo apt-get update
sudo apt-get install smartmontools
Smartmontools most often doesn't work on USB hard disks.

The USB to sata / pata bridges inside external HDD caddies typically do not pass through the necessary commands for SMART because they abstract the drive as a basic scsi device. There are a few exceptions - some integrated true USB chips may support it.

http://sourceforge.net/apps/trac/smartmontools/wiki/USB
Rockets are loud.
https://astro-pi.org

fredoll
Posts: 227
Joined: Thu Jan 10, 2013 10:30 am

Re: Keep 'losing' one of two HD's

Tue Aug 06, 2013 12:53 pm

Thanks, I didn't know that :-) and sorry for the false hope

User avatar
makem2
Posts: 231
Joined: Tue Jul 02, 2013 9:12 am
Location: Essex
Contact: Website

Re: Keep 'losing' one of two HD's

Tue Aug 06, 2013 4:00 pm

Installed and worked out what 'drive name' is!
Output of -i says drive does not support SMART

So I need to investigate further to see if I can use the software.

User avatar
makem2
Posts: 231
Joined: Tue Jul 02, 2013 9:12 am
Location: Essex
Contact: Website

Re: Keep 'losing' one of two HD's

Mon Aug 12, 2013 9:47 am

RaTTuS wrote:sudo apt-get update && sudo apt-get upgrade
sudo apt-get install pastebinit
dmesg | pastbinit
...
it's possible you have a failing HD
or the connection may drop to it for some reason - you may be able to get away with a re-mount of the drive
Following a reboot I have use the Pi for a while now without problem, however last night it transferred a large amount of data from HDD1 to HDD2 with cron jobs.

I have 'lost' HDD2 once again. File manager says, "The specified directory is not valid" when I try to open that drive and it does not show in filemanager except when I navigate to /media/ where I have the option to choose either HD but I get that error when choosing HDD2.

As you suggested I have done a dmesg before rebot and sent it to pastebinit:

http://paste.debian.net/24282/

I would appreciate any comment.

I am away in Germany for a week so may not be able to look here until after 23rd.

User avatar
RaTTuS
Posts: 10412
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK

Re: Keep 'losing' one of two HD's

Mon Aug 12, 2013 10:04 am

looks like something is running out of memory
df -h
free
sudo apt-get update && sudo apt-get upgrade -y
what other things are you running ?
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

User avatar
makem2
Posts: 231
Joined: Tue Jul 02, 2013 9:12 am
Location: Essex
Contact: Website

Re: Keep 'losing' one of two HD's

Mon Aug 12, 2013 1:30 pm

df -h:

Filesystem Size Used Avail Use% Mounted on
rootfs 30G 1.7G 27G 6% /
/dev/root 30G 1.7G 27G 6% /
devtmpfs 116M 0 116M 0% /dev
tmpfs 25M 384K 25M 2% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 50M 0 50M 0% /run/shm
/dev/mmcblk0p1 56M 19M 38M 33% /boot
/dev/sda1 932G 100G 833G 11% /media/HDD1
/dev/sdb1 932G 157G 776G 17% /media/HDD2

free:

total used free shared buffers cached
Mem: 253904 145120 108784 0 15532 77924
-/+ buffers/cache: 51664 202240
Swap: 102396 0 102396

sudo apt-get update && sudo apt-get upgrade -y - done

What does the -y do?

I am running nothing until the cron jobs start:

0 3 * * 7 rsync -av --delete /media/HDD1/shares/archive /media/HDD2/shares/ >> /home/pi/log/archive.log 2>&1

0 4 * * 7 rsync -av --delete /media/HDD1/shares/mailwasher /media/HDD2/shares/ >> /home/pi/log/mailwasher.log 2>&1

0 5 * * * rsync -av --delete /media/HDD1/shares/hanprofile /media/HDD2/shares/ >> /home/pi/log/hanprofile.log 2>&1

0 6 * * * rsync -av --delete /media/HDD1/shares/myprofile /media/HDD2/shares/ >> /home/pi/log/myprofile.log 2>&1

Then again it is idle.

EDIT: No changes made when sudo apt-get update && sudo apt-get upgrade -y

User avatar
RaTTuS
Posts: 10412
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK

Re: Keep 'losing' one of two HD's

Mon Aug 12, 2013 1:56 pm

the -y just installs without asking - I usually don't type that in my replied but I do it all the time [sorry]

all I can think of is that you are running out of mem on the rsync but it should not do that ...
as long as you don't have any liinx between the files systems that could case a loop...
try the rsyc with
-x on the line [don't traverse filesystems]
nothing else I can suggest
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

User avatar
makem2
Posts: 231
Joined: Tue Jul 02, 2013 9:12 am
Location: Essex
Contact: Website

Re: Keep 'losing' one of two HD's

Mon Aug 12, 2013 2:14 pm

0 6 * * * rsync -avx --delete /media............?

I have noticed in file properties that the file sizes are not correct:

Total Size of Files: 12.2 GB
Size on Disk: 98.2 GB

For one folder on HDD1.

For the whole share on HDD1:

Total Size of Files: 106.4 GB
Size on Disk: 852.1 GB

They are both out by the same %age and I wonder what happens when it reaches 1TB.

Maybe that is a consequence of a memory problem.

I will run the cron jobs manually with the '-x' and see what happens.

Thanks for you guidance.

User avatar
makem2
Posts: 231
Joined: Tue Jul 02, 2013 9:12 am
Location: Essex
Contact: Website

Re: Keep 'losing' one of two HD's

Mon Aug 12, 2013 2:36 pm

ok, the two jobs ran ok and the HD is still showing.

I have done another dmesg | pastebinit in case that throws any light on the subject.

http://paste.debian.net/24354/

I am wondering what would happen if the second cron job started before the first had ended although in tests the first cron job is small and never normally takes near an hour to complete. Maybe I need to run them on differing days.

User avatar
RaTTuS
Posts: 10412
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK

Re: Keep 'losing' one of two HD's

Mon Aug 12, 2013 2:41 pm

ummm
mount.ntfs: page allocation failure: order:0, mode:0x20
are the drives
ntfs formatted
are they clean?
are they mounted with ntfs-3g

can ext4 be used instead ?

just wondering?
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

User avatar
makem2
Posts: 231
Joined: Tue Jul 02, 2013 9:12 am
Location: Essex
Contact: Website

Re: Keep 'losing' one of two HD's

Mon Aug 12, 2013 3:38 pm

Clean? as in?

Formatted NTFS and never defragged since install.

When I first set this up I connected each drive to a Win7 computer and copied over the data as it was quite large, therefore NTFS.

Now I have no real need to do the same again as all I do I copy over changes (some quite large though).

They are mounted with ntfs-3g.

As for ext4, yes they could be but thenthey can no longer read by Win7? That means that should I need to copy all data again for some reason it must be done via the pi. That would be a large load for it?

What would you do in my position? I understand that ext4 would be a better proposition but as I have never used it in conjunction with Windows, I don't know all the implications. If i did change to ext4 then I have some 250 GB to transfer via the pi.

EDIT: Having read about ext4 v NTFS it would appear that in my circumstances nothing will be gained. However, I may be wrong. I suppose, to defrag the disks I could delete one and copy over from the other and then repeat in the opposite direction. Would I gain anything though? According to what I read, even if there was a gain, I would be unlikely to notice it as they are USB drives.

User avatar
makem2
Posts: 231
Joined: Tue Jul 02, 2013 9:12 am
Location: Essex
Contact: Website

Re: Keep 'losing' one of two HD's

Wed Aug 28, 2013 1:45 pm

Still getting the 'losing' a drive. Very annoying as it is occurring quite often and spoiling backups.

I notice on boot-up a 'failed' in red relating to a drive:

special device /dev/sda2 does not exist

I have never noticed this before and wonder if it is related to the drive which keeps going AWOL.

User avatar
abishur
Posts: 4477
Joined: Thu Jul 28, 2011 4:10 am
Location: USA
Contact: Website

Re: Keep 'losing' one of two HD's

Wed Aug 28, 2013 2:56 pm

makem2 wrote:Still getting the 'losing' a drive. Very annoying as it is occurring quite often and spoiling backups.

I notice on boot-up a 'failed' in red relating to a drive:

special device /dev/sda2 does not exist

I have never noticed this before and wonder if it is related to the drive which keeps going AWOL.
is that how you reference it in your fstab? If so you might find it helps to reference it by it's uuid instead
Dear forum: Play nice ;-)

User avatar
makem2
Posts: 231
Joined: Tue Jul 02, 2013 9:12 am
Location: Essex
Contact: Website

Re: Keep 'losing' one of two HD's

Wed Aug 28, 2013 5:00 pm

Yes that is how it is reference in the fstab.

/dev/sda1 /media/HDD1 auto noatime 0 0
/dev/sda2 /media/HDD2 auto noatime 0 0

(no idea what auto noatime 0 0 relate to - about time I found out)

Actually, I had forgotten I set that up!

I didn't realise sda2 was one of the drives I was using because when the reboot had ended, both drives were present again. A drive just became 'lost' during use.

I have seen somewhere mention of the the solution you suggest. I will find it again, thanks.

EDIT: ls -l /dev/disk/by-uuid/ shows only sdb1 and sda1 but both drives show in file manager. It seems I have gained an sdb1 and not a sda2 as it should be.

User avatar
abishur
Posts: 4477
Joined: Thu Jul 28, 2011 4:10 am
Location: USA
Contact: Website

Re: Keep 'losing' one of two HD's

Wed Aug 28, 2013 5:57 pm

makem2 wrote:EDIT: ls -l /dev/disk/by-uuid/ shows only sdb1 and sda1 but both drives show in file manager. It seems I have gained an sdb1 and not a sda2 as it should be.
Ah, that's something I also learned not all that long ago (relatively speaking). When mounting drives its recommended to use their uuid as the sd## can change when you have multiple drives plugged in. It's possible that this is the root cause of your troubles and change fstab to mount by uuid could fix your problems. Even if it doesn't fix it, it's still a better way of doing things :lol:
Dear forum: Play nice ;-)

User avatar
makem2
Posts: 231
Joined: Tue Jul 02, 2013 9:12 am
Location: Essex
Contact: Website

Re: Keep 'losing' one of two HD's

Wed Aug 28, 2013 6:00 pm

/dev/sda1 /media/HDD1 auto noatime 0 0
/dev/sda2 /media/HDD2 auto noatime 0 0

was the original fstab and I changed this to:

#/dev/sda1 /media/HDD1 auto noatime 0 0
#/dev/sda2 /media/HDD2 auto noatime 0 0

UUID=86D8F0FED8F0ED75 /media/HDD1 auto noatime 0 0
UUID=4A7E9DD27E9DB6E3 /media/HDD2 auto noatime 0 0

However ls -l /dev/disk/by-uuid/ still produces:

total 0
lrwxrwxrwx 1 root root 15 Aug 28 18:45 41cd5baa-7a62-4706-b8e8-02c43ccee8d9 -> ../../mmcblk0p2
lrwxrwxrwx 1 root root 10 Aug 28 18:45 4A7E9DD27E9DB6E3 -> ../../sdb1
lrwxrwxrwx 1 root root 15 Jan 1 1970 5D2D-B09A -> ../../mmcblk0p1
lrwxrwxrwx 1 root root 10 Jan 1 1970 86D8F0FED8F0ED75 -> ../../sda1

So, nothing has actually changed. However, the 'failed' message was not present on reboot.

I suppose I must investigate 'auto noatime 0 0'

Return to “Troubleshooting”