Project

General

Profile

Bug #2023

Greyhole, drives go offline frequently

Added by kikkegek over 5 years ago. Updated over 4 years ago.

Status:
Closed
Priority:
Normal
Assignee:
kikkegek
Target version:
apps - amahi-8
Start date:
Due date:
% Done:

0%

Description

he guys,

foe some reason I can find, my drives keep going offline about once a month.

somestimes drive2 sometimes drive3

its very annoying, and and can track down the readon why.

url to the topic with all the info:

https://forums.amahi.org/viewtopic.php?f=39&t=6431&p=36497#p36497

e9a98adc-0f7f-41c9-a37a-4164bd1a8d11.jpg View - no complete boot (128 KB) kikkegek, 02/06/2017 03:03 PM

31ae9673-8cb6-4b7d-b98d-ba4f7ba80b55.jpg View - boot errors (198 KB) kikkegek, 02/06/2017 03:04 PM

free blocks count wrong.jpg View - free blocks counts wrong (220 KB) kikkegek, 02/06/2017 03:06 PM

free inodes counts wrong.jpg View - free inodes counts wrong (124 KB) kikkegek, 02/06/2017 03:07 PM

History

#1 Updated by kikkegek over 5 years ago

  • Due date set to 04/15/2016
  • Estimated time set to 2.00 h

#2 Updated by bigfoot65 over 5 years ago

  • Due date deleted (04/15/2016)
  • Assignee deleted (bigfoot65)
  • Start date deleted (04/05/2016)
  • Estimated time deleted (2.00 h)

#3 Updated by bigfoot65 over 5 years ago

Please provide the URL for the following:

fpaste --sysinfo

#4 Updated by kikkegek over 5 years ago

bigfoot65 wrote:

Please provide the URL for the following:
[...]

[[http://paste.fedoraproject.org/355335/06268831/]]

#5 Updated by cpg over 5 years ago

  • Description updated (diff)

All this information, but the question is -- what does it mean for the drives to be offline?

When "offline", does it mean a drive as not mounted on the system?

This is definitely something we have not seen before. There are no known parts of the system that would unmount a drive.

This looks like a hardware.

What do the system logs say? If the system is still booted since it happened, you can output the system journal from the current boot with this:

journalctl -b | fpaste

#6 Updated by kikkegek over 5 years ago

well, I get an error there. should i reboot and do it again?

[jochen@localhost ~]$ sudo journalctl -b | fpaste
Error was encountered while opening journal files: Invoer-/uitvoerfout
WARNING: your paste size (2971.3KiB) is very large and may be rejected by the server. A pastebin is NOT a file hosting service!
Uploading (2971.3KiB)...
Error: Server did not return a correct JSON response

http://paste.fedoraproject.org/356449/60789938/

#7 Updated by cpg over 5 years ago

You can try

journalctl -b | tail -500 | fpaste

(if it's still too large, reduce the 500 to 400 or 300)

#8 Updated by cpg over 5 years ago

also, please respond to the questions -- does the drive get unmounted?
what does "offline" mean for you, exactly?

#9 Updated by kikkegek over 5 years ago

cpg wrote:

also, please respond to the questions -- does the drive get unmounted?
what does "offline" mean for you, exactly?

[jochen@localhost ~]$ sudo journalctl -b | tail -500 | fpaste

http://paste.fedoraproject.org/356452/14607907/

#10 Updated by kikkegek over 5 years ago

kikkegek wrote:

cpg wrote:

also, please respond to the questions -- does the drive get unmounted?
what does "offline" mean for you, exactly?

[jochen@localhost ~]$ sudo journalctl b | tail -500 | fpaste
Error was encountered while opening journal files: Invoer/uitvoerfout
Uploading (66.4KiB)...
http://paste.fedoraproject.org/356452/14607907/ -> [[http://paste.fedoraproject.org/356452/14607907]]
[jochen@localhost ~]$

the "offline" comes from the status it has in greyhole -s

[jochen@localhost ~]$ greyhole -s

Greyhole Statistics
===================

Storage Pool
                                    Total -   Used =   Free +  Trash = Possible
  /var/hda/files/drives/drive2/gh:   917G -   790G =    80G +    15G =    95G
  /var/hda/files/drives/drive3/gh:                  Offline                  
  /var/hda/files/drives/drive1/gh:   688G -   653G =     0G +     3G =     3G
                                   ==========================================
  Total:                            1604G -  1444G =    80G +    18G =    97G

#11 Updated by kikkegek over 5 years ago

kikkegek wrote:

kikkegek wrote:

cpg wrote:

also, please respond to the questions -- does the drive get unmounted?
what does "offline" mean for you, exactly?

[jochen@localhost ~]$ sudo journalctl b | tail -500 | fpaste
Error was encountered while opening journal files: Invoer/uitvoerfout
Uploading (66.4KiB)...
http://paste.fedoraproject.org/356452/14607907/ -> [[http://paste.fedoraproject.org/356452/14607907]]
[jochen@localhost ~]$

the "offline" comes from the status it has in greyhole -s

[...]

and the drive is still mounted, because i can visit that location on on my filesystem

/var/hda/files/drives/drive3/gh
[jochen@localhost ~]$ cd  /var/hda/files/drives/drive3/gh
[jochen@localhost gh]$ la
-bash: la: opdracht niet gevonden
[jochen@localhost gh]$ ls
Docs  Movies  Music  Newsgroups  Pictures  Software  Torrents
[jochen@localhost gh]$ cd /
[jochen@localhost /]$ cd  /var/hda/files/drives/drive3/gh
[jochen@localhost gh]$ ls -al
totaal 68
drwxr-xr-x 12 root   root   4096  7 apr 14:36 .
drwxr-xr-x  4 root   root   4096 17 apr  2014 ..
drwxrwxr-x 10 jochen users  4096 24 jan 21:09 Docs
drwxrwxrwx  9 root   root   4096  1 sep  2014 .gh_metastore
drwxrwxrwx  8 root   root   4096  3 sep  2014 .gh_metastore_backup
-rw-rw-rw-  1 root   root   1340 16 jan 08:39 .gh_settings.bak
drwxrwxrwx  3 root   root   4096  7 apr 14:36 .gh_trash
drwxrwxrwx  8 root   root   4096 21 dec 19:03 Movies
drwxrwxr-x  3 jochen users  4096  1 okt  2014 Music
drwxrwxrwx  7 jochen users  4096 18 sep  2015 Newsgroups
drwxrwxr-x 39 jochen users  4096  5 jan 07:40 Pictures
drwxrwxr-x 57 jochen users  4096  5 dec 09:33 Software
drwxrwxr-x 18 jochen users 20480 21 jan 04:04 Torrents
[jochen@localhost gh]$

#12 Updated by kikkegek over 5 years ago

I checked the greyhole log and ound this:

Apr 18 08:37:17 INFO fsck:   Drive /var/hda/files/drives/drive3/gh is not part of the Greyhole storage pool anymore. The above file will not be counted as a valid file copy, but can be used to create a new valid copy.
Apr 18 13:45:40 WARN daemon: Warning! It seems the partition UUID of /var/hda/files/drives/drive3/gh changed. This probably means this mount is currently unmounted, or that you replaced this drive and didn't use 'greyhole --replace'. Because of that, Greyhole will NOT use this drive at this time.
Apr 18 13:46:03 WARN daemon: Warning! It seems the partition UUID of /var/hda/files/drives/drive3/gh changed. This probably means this mount is currently unmounted, or that you replaced this drive and didn't use 'greyhole --replace'. Because of that, Greyhole will NOT use this drive at this time.
[jochen@localhost log]$

I didnt change anything to the drive or filesystem though.

#13 Updated by kikkegek about 5 years ago

hello. is there any news?

#14 Updated by bigfoot65 about 5 years ago

Please share the /etc/greyhole.conf file along with any errors you see in /var/log/greyhole.log file.

Best to attach them as files.

I noticed there are partition UUIDs on 2 of your 3 drives:

/dev/sdb1: UUID="7493b2b3-9ec4-4711-92ef-388390822826" TYPE="ext4" PARTUUID="7f04484c-01" 
/dev/sdc1: LABEL="SAMSUNG" UUID="667bd417-70f4-4524-ab03-501113edf499" TYPE="ext3" PARTUUID="9310de50-01"

Are these 2 drives configured differently than the sdd drive? That might be a clue as to your issue.

#15 Updated by kikkegek about 5 years ago

  • Priority changed from Normal to Low

bigfoot65 wrote:

Please share the /etc/greyhole.conf file along with any errors you see in /var/log/greyhole.log file.

Best to attach them as files.

I noticed there are partition UUIDs on 2 of your 3 drives:
[...]
Are these 2 drives configured differently than the sdd drive? That might be a clue as to your issue.

Hi Bigfoot65

I configured the drives according to the wiki. The only thing I have changed is that I use a different landing zone as the wiki.

is having a UUID of a drive good or bad?

[jochen@localhost log]$ less /etc/greyhole.conf | fpaste
Uploading (1.5KiB)...
http://paste.fedoraproject.org/370726/64188047/ -> http://paste.fedoraproject.org/370726/64188047

the last greyhole.log has NO ERRORS

and all available log files contain no errors. The logs from the time the drive went offline dont exist anymore. sorry

#16 Updated by bigfoot65 about 5 years ago

  • Assignee set to kikkegek
  • Priority changed from Low to Normal

Seems odd that your /etc/fstab contains partition UUIDs in it. Typically a Greyhole drive has a single partition and the UUID for the drive is all that is used.

Wonder if that has anything to do with the issue.

Have you checked permissions on the drives?

#17 Updated by kikkegek about 5 years ago

what permissions would you like me to check? Folder permissions of certain folders? or read/write persmissions on the greyhole shares in the dashboard?

I have always followed the wiki from this link:

[[https://wiki.amahi.org/index.php/Adding_a_second_hard_drive_to_your_HDA]]

#18 Updated by kikkegek over 4 years ago

kikkegek wrote:

what permissions would you like me to check? Folder permissions of certain folders? or read/write persmissions on the greyhole shares in the dashboard?

I have always followed the wiki from this link:

[[https://wiki.amahi.org/index.php/Adding_a_second_hard_drive_to_your_HDA]]

bigfoot?

#19 Updated by kikkegek over 4 years ago

kikkegek wrote:

kikkegek wrote:

what permissions would you like me to check? Folder permissions of certain folders? or read/write persmissions on the greyhole shares in the dashboard?

I have always followed the wiki from this link:

[[https://wiki.amahi.org/index.php/Adding_a_second_hard_drive_to_your_HDA]]

bigfoot?

when I do
systemctl restart amahi-greyhole.service

and check the gh log:
Dec 22 15:32:32 WARN daemon: Warning! It seems the partition UUID of /var/hda/files/drives/drive3/gh changed. This probably means this mount is currently unmounted, or that you replaced this drive and didn't use 'greyhole --replace'. Because of that, Greyhole will NOT use this drive at this time.

#20 Updated by bigfoot65 over 4 years ago

Please post the additional drive mount text from /etc/fstab.

Something is not configured correctly that is causing this issue.

Also please provide the result of:

ls -al /var/hda/files/drives/drive3

Need the contents of /etc/greyhole.conf file (please attach it) as the fpaste is no longer available.

#21 Updated by kikkegek over 4 years ago

bigfoot65 wrote:

Please post the additional drive mount text from /etc/fstab.

Something is not configured correctly that is causing this issue.

Also please provide the result of:
[...]

Need the contents of /etc/greyhole.conf file (please attach it) as the fpaste is no longer available.

fstab drive mount info

# 2TB Western Digital drive added 20140417
UUID=514a5219-1017-4a91-acc5-8259c09c6ae4 /var/hda/files/drives/drive3 ext4 defaults 1 2

[jochen@localhost ~]$ ls -al /var/hda/files/drives/drive3
total 28
drwxr-xr-x  4 root   root   4096 Apr 17  2014 .
drwxrwxr-x  6 jochen users  4096 Apr 17  2014 ..
drwxr-xr-x 11 root   root   4096 Oct 29 13:53 gh
drwx------  2 root   root  16384 Apr 17  2014 lost+found
[jochen@localhost ~]$

[[http://paste.fedoraproject.org/511271/14824423]]

#23 Updated by kikkegek over 4 years ago

bigfoot65 wrote:

Which format method did you follow from the wiki?
https://wiki.amahi.org/index.php/Adding_a_second_hard_drive_to_your_HDA#Partition.2FFormat

dont know exactly what you mean, but tried this

login as: jochen
jochen@192.168.178.12's password:
Last login: Thu Dec 22 22:30:34 2016 from 192.168.178.152
[jochen@localhost ~]$ diff before.txt after.txt
diff: before.txt: No such file or directory
diff: after.txt: No such file or directory
[jochen@localhost ~]$

#24 Updated by bigfoot65 over 4 years ago

I meant the partitioning guidance. Did you do the msdos or gpt method?

#25 Updated by kikkegek over 4 years ago

I dont remember sorry

does this list help?

[jochen@localhost ~]$ ls -l /dev/disk/by-id/
total 0
lrwxrwxrwx 1 root root  9 Sep 21 20:47 ata-OCZ-VERTEX2_OCZ-SG01P8JB56URE6LO -> ../../sda
lrwxrwxrwx 1 root root 10 Sep 21 20:47 ata-OCZ-VERTEX2_OCZ-SG01P8JB56URE6LO-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Sep 21 20:47 ata-OCZ-VERTEX2_OCZ-SG01P8JB56URE6LO-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Sep 21 20:47 ata-OCZ-VERTEX2_OCZ-SG01P8JB56URE6LO-part3 -> ../../sda3
lrwxrwxrwx 1 root root 10 Sep 21 20:47 ata-OCZ-VERTEX2_OCZ-SG01P8JB56URE6LO-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Sep 21 20:47 ata-OCZ-VERTEX2_OCZ-SG01P8JB56URE6LO-part5 -> ../../sda5
lrwxrwxrwx 1 root root  9 Sep 21 20:47 ata-SAMSUNG_HD753LJ_S13UJ1KQ331160 -> ../../sdc
lrwxrwxrwx 1 root root 10 Sep 21 20:47 ata-SAMSUNG_HD753LJ_S13UJ1KQ331160-part1 -> ../../sdc1
lrwxrwxrwx 1 root root  9 Sep 21 20:47 ata-WDC_WD10EARS-00MVWB0_WD-WCAZA7199244 -> ../../sdb
lrwxrwxrwx 1 root root 10 Sep 21 20:47 ata-WDC_WD10EARS-00MVWB0_WD-WCAZA7199244-part1 -> ../../sdb1
lrwxrwxrwx 1 root root  9 Sep 21 20:47 ata-WDC_WD20EZRX-00D8PB0_WD-WMC4M2275947 -> ../../sdd
lrwxrwxrwx 1 root root 10 Sep 21 20:47 ata-WDC_WD20EZRX-00D8PB0_WD-WMC4M2275947-part1 -> ../../sdd1
lrwxrwxrwx 1 root root  9 Sep 21 20:47 wwn-0x50000f000b331106 -> ../../sdc
lrwxrwxrwx 1 root root 10 Sep 21 20:47 wwn-0x50000f000b331106-part1 -> ../../sdc1
lrwxrwxrwx 1 root root  9 Sep 21 20:47 wwn-0x50014ee205d8594b -> ../../sdb
lrwxrwxrwx 1 root root 10 Sep 21 20:47 wwn-0x50014ee205d8594b-part1 -> ../../sdb1
lrwxrwxrwx 1 root root  9 Sep 21 20:47 wwn-0x50014ee604207d8e -> ../../sdd
lrwxrwxrwx 1 root root 10 Sep 21 20:47 wwn-0x50014ee604207d8e-part1 -> ../../sdd1
lrwxrwxrwx 1 root root  9 Sep 21 20:47 wwn-0x5e83a97f24b1d0c0 -> ../../sda
lrwxrwxrwx 1 root root 10 Sep 21 20:47 wwn-0x5e83a97f24b1d0c0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Sep 21 20:47 wwn-0x5e83a97f24b1d0c0-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Sep 21 20:47 wwn-0x5e83a97f24b1d0c0-part3 -> ../../sda3
lrwxrwxrwx 1 root root 10 Sep 21 20:47 wwn-0x5e83a97f24b1d0c0-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Sep 21 20:47 wwn-0x5e83a97f24b1d0c0-part5 -> ../../sda5

#26 Updated by bigfoot65 over 4 years ago

No worries. If you did not partition and format it correctly, that could cause issues.

Odd that is works sometimes. Almost seems like a drive or driver issue.

Not sure what else we can do to assist here.

#27 Updated by kikkegek over 4 years ago

bigfoot65 wrote:

No worries. If you did not partition and format it correctly, that could cause issues.

Odd that is works sometimes. Almost seems like a drive or driver issue.

Not sure what else we can do to assist here.

what I dont understand is that the drive is mounted. I can see files in the folders. But in greyhole its offline?

what could cause that?

#28 Updated by bigfoot65 over 4 years ago

Not sure, but maybe try this:

sudo greyhole --replace=/var/hda/files/drives/drive3/gh

Change drive3 to the drive that is offline.

#29 Updated by kikkegek over 4 years ago

well...couple days ago the HDA didnt even completely boot anymore. giving me all kinds of mounting errors. See attached images.

trying to mount the drive3 gave errors

what fixed it in the end was logging in with the root password and

#e2fsck -y /dev/sdd1

it found a lot of free blocks and inodes counts wrong

fixed it and now it reboots without errors and drive3 gets mounted again withut problems

Greyhole is fixing all the missing files and links

so case closed, although I still dont have a clue what could have caused it. I am glad its fixed (Y)

#30 Updated by bigfoot65 over 4 years ago

  • Status changed from Resolved to Closed

That's great news that you got it sorted out.

Hard to say how the drive got that way, but obviously it was the problem all along. Would have never thought to check for any of those issues.

Will close the bug report.

Also available in: Atom