Project

General

Profile

Bug #766

Files disappear from greyhole shares

Added by andynj over 8 years ago. Updated about 7 years ago.

Status:
Assigned
Priority:
Normal
Assignee:
Target version:
-
Start date:
02/16/2011
Due date:
% Done:

0%


Description

When transferring files to a greyhole (v0.8.99-1) share, they disappear after a few seconds and never reappear. This only happens with the files, the folders (created by client) remain. Additionally, after the files have disappeared, the folder becomes locked and I cannot delete it or write to it anymore without doing a chmod. I have noticed that when this happens, the folder's owner is set to root.

I have experienced this with both Windows and Mac clients. I am using the latest version of Samba.

History

#1 Updated by andynj over 8 years ago

Also, this issue does not happen for shares that are not in the greyhole pool.

#3 Updated by andynj over 8 years ago

1.

2.6.31.5-127.fc12.x86_64
samba-3.4.9-60.fc12.x86_64
hda-greyhole-0.8.99-1.x86_64

2.

http://fpaste.org/tQzB/
http://fpaste.org/wjGP/

3.

/dev/sda1 on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda2 on /var/hda/files/drives/drive1 type ext4 (rw,_netdev)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
//127.0.0.1/Docs on /mnt/samba/Docs type cifs (rw,mand)
//127.0.0.1/Music on /mnt/samba/Music type cifs (rw,mand)
//127.0.0.1/Photos on /mnt/samba/Photos type cifs (rw,mand)
//127.0.0.1/Software on /mnt/samba/Software type cifs (rw,mand)
//127.0.0.1/TimeMachine on /mnt/samba/TimeMachine type cifs (rw,mand)
//127.0.0.1/Videos on /mnt/samba/Videos type cifs (rw,mand)
gvfs-fuse-daemon on /home/andy/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev,user=andy)
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              16G  3.2G   12G  22% /
tmpfs                1005M  548K 1005M   1% /dev/shm
/dev/sda2             211G   22G  178G  11% /var/hda/files/drives/drive1
//127.0.0.1/Docs      211G   33G  178G  16% /mnt/samba/Docs
//127.0.0.1/Music     211G   33G  178G  16% /mnt/samba/Music
//127.0.0.1/Photos    211G   33G  178G  16% /mnt/samba/Photos
//127.0.0.1/Software  211G   33G  178G  16% /mnt/samba/Software
//127.0.0.1/TimeMachine
                      211G   33G  178G  16% /mnt/samba/TimeMachine
//127.0.0.1/Videos    211G   33G  178G  16% /mnt/samba/Videos

Greyhole Statistics
===================

Storage Pool
                                   Total -  Used =  Free + Attic = Possible
  /var/hda/files/drives/drive1/gh:   N/A -   N/A =   N/A +   N/A =   N/A

4.


+----+------------------------------+--------------+---------------------+---------------------+
| id | path                         | minimum_free | created_at          | updated_at          |
+----+------------------------------+--------------+---------------------+---------------------+
|  2 | /var/hda/files/drives/drive1 |           10 | 2011-02-14 02:44:20 | 2011-02-14 02:44:20 |
+----+------------------------------+--------------+---------------------+---------------------+

5.

ls: cannot access /var/hda/files/drives/drive1/gh: Permission denied
No text to send.

6.

Greyhole Work Queue Statistics
==============================

This table gives you the number of pending operations queued for the Greyhole daemon, per share.

             Write   Delete  Rename
Docs         0       0       0
Music        0       0       0
Photos       0       0       0
Software     0       0       0
TimeMachine  0       0       0
Videos       0       0       0
===========
Total        0     + 0     + 0     = 0

The following is the number of pending operations that the Greyhole daemon still needs to parse.
Until it does, the nature of those operations is unknown.
Spooled operations that have been parsed will be listed above and disappear from the count below.

Spooled      0

7. The problem happens with every file, but here's an example.

Debugging file operations for file named "/ste-java-interface.pdf" 

From DB
=======

From logs
=========
Feb 16 09:54:39 7 simplify_tasks: Simplifying pending tasks.
Feb 16 09:54:39 7 write: Now working on task ID 15: write Docs/ste-java-interface.pdf
Feb 16 09:54:39 6 write: File created: Docs/ste-java-interface.pdf - 1.56MB
Feb 16 09:54:39 7 write: Loading tombstones for Docs/./ste-java-interface.pdf... Got 0 tombstones.
Feb 16 09:54:39 7 write:   File /var/hda/files/docs/ste-java-interface.pdf is locked by another process. Will wait until it's unlocked to work on it.
Feb 16 09:54:39 7 sleep: Only locked files operations pending... Sleeping.
Feb 16 09:54:49 7 write: Now working on task ID 16: write Docs/ste-java-interface.pdf
Feb 16 09:54:49 6 write: File created: Docs/ste-java-interface.pdf - 1.56MB
Feb 16 09:54:49 7 write: Loading tombstones for Docs/./ste-java-interface.pdf... Got 0 tombstones.
Feb 16 09:54:50 7 write: Drives with available space: /var/hda/files/drives/drive1/gh (168GB avail)
Feb 16 09:54:50 7 write:   Saving 1 tombstones for Docs/./ste-java-interface.pdf
Feb 16 09:54:50 4 write: PHP Warning [2]: mkdir(): File exists in /usr/bin/greyhole on line 1980
--
Feb 16 09:54:50 4 write:   Failed to create directory /var/hda/files/drives/drive1/gh/.gh_graveyard/Docs/.
Feb 16 09:54:50 7 write:     Saving tombstones in /var/hda/files/drives/drive1/gh/.gh_graveyard/Docs/./ste-java-interface.pdf
Feb 16 09:54:50 7 write:   Copying file to /var/hda/files/drives/drive1/gh/Docs/ste-java-interface.pdf
Feb 16 09:54:50 7 write:     Creating symlink in share pointing to the above file copy.
Feb 16 09:54:50 7 write:   Saving 1 tombstones for Docs/./ste-java-interface.pdf
Feb 16 09:54:50 7 write:     Saving tombstones in /var/hda/files/drives/drive1/gh/.gh_graveyard/Docs/./ste-java-interface.pdf
Feb 16 09:54:50 7 read_smb_spool: Processing Samba spool...

From filesystem
===============
Landing Zone:
  ls: cannot access /ste-java-interface.pdf: No such file or directory

Graveyard:

File copies:

#4 Updated by gboudreau over 8 years ago

Did you really run all those commands as root ?
I'd be surprised that root wouldn't have access to list this directory (step 5):
ls: cannot access /var/hda/files/drives/drive1/gh: Permission denied

If you didn't, please re-run the following, as root:

greyhole --stats
mysql -u root -phda -e "select concat(path, '/gh') from disk_pool_partitions" hda_production | grep -v 'concat(' | xargs ls -la | fpaste
greyhole --debug ste-java-interface.pdf

I find it weird that --stats gives all N/As.
That that --debug, "From filesystem" section, can't find where this file's landing zone is. Which probably is your main problem.

#5 Updated by andynj over 8 years ago

If I do sudo, I get a permission denied, but if I do su, it works.

greyhole --stats

Greyhole Statistics
===================

Storage Pool
                                   Total -  Used =  Free + Attic = Possible
  /var/hda/files/drives/drive1/gh:  210G -   22G =  178G +    1G =  178G

mysql -u root -phda -e "select concat(path, '/gh') from disk_pool_partitions" hda_production | grep -v 'concat(' | xargs ls -la | fpaste

http://fpaste.org/MSVI/

greyhole --debug ste-java-interface.pdf

[root@Donbot andy]# greyhole --debug ste-java-interface.pdf
Debugging file operations for file named "/ste-java-interface.pdf" 

From DB
=======

From logs
=========
Feb 16 09:54:39 7 simplify_tasks: Simplifying pending tasks.
Feb 16 09:54:39 7 write: Now working on task ID 15: write Docs/ste-java-interface.pdf
Feb 16 09:54:39 6 write: File created: Docs/ste-java-interface.pdf - 1.56MB
Feb 16 09:54:39 7 write: Loading tombstones for Docs/./ste-java-interface.pdf... Got 0 tombstones.
Feb 16 09:54:39 7 write:   File /var/hda/files/docs/ste-java-interface.pdf is locked by another process. Will wait until it's unlocked to work on it.
Feb 16 09:54:39 7 sleep: Only locked files operations pending... Sleeping.
Feb 16 09:54:49 7 write: Now working on task ID 16: write Docs/ste-java-interface.pdf
Feb 16 09:54:49 6 write: File created: Docs/ste-java-interface.pdf - 1.56MB
Feb 16 09:54:49 7 write: Loading tombstones for Docs/./ste-java-interface.pdf... Got 0 tombstones.
Feb 16 09:54:50 7 write: Drives with available space: /var/hda/files/drives/drive1/gh (168GB avail)
Feb 16 09:54:50 7 write:   Saving 1 tombstones for Docs/./ste-java-interface.pdf
Feb 16 09:54:50 4 write: PHP Warning [2]: mkdir(): File exists in /usr/bin/greyhole on line 1980
--
Feb 16 09:54:50 4 write:   Failed to create directory /var/hda/files/drives/drive1/gh/.gh_graveyard/Docs/.
Feb 16 09:54:50 7 write:     Saving tombstones in /var/hda/files/drives/drive1/gh/.gh_graveyard/Docs/./ste-java-interface.pdf
Feb 16 09:54:50 7 write:   Copying file to /var/hda/files/drives/drive1/gh/Docs/ste-java-interface.pdf
Feb 16 09:54:50 7 write:     Creating symlink in share pointing to the above file copy.
Feb 16 09:54:50 7 write:   Saving 1 tombstones for Docs/./ste-java-interface.pdf
Feb 16 09:54:50 7 write:     Saving tombstones in /var/hda/files/drives/drive1/gh/.gh_graveyard/Docs/./ste-java-interface.pdf
Feb 16 09:54:50 7 read_smb_spool: Processing Samba spool...

From filesystem
===============
Landing Zone:
  ls: cannot access /ste-java-interface.pdf: No such file or directory

Graveyard:

File copies:

#6 Updated by gboudreau over 8 years ago

ls -la /var/hda/files/docs/ | head
ls -la /var/hda/files/docs/ste-java-interface.pdf

The drive1/gh folder's content has weird permissions (some owned by andy, others by uid 500). And the drive1 folder is owned by uid 500, group 'Andy'... All this seems pretty confused.
I'd reset the permissions on the whole /var/hda/files folder, including subdirectories:
chown -R andy:users /var/hda/files/

Following that, try a "greyhole--fsck", and see if the problem persists.

#7 Updated by andynj over 8 years ago

Oh wow, I ran both commands and everything seems to work now. And the files that had gone missing before showed up. I'll have to continue testing more when I actually get home and use my Mac (currently accessing via VPN from work on a Win7 machine, but I saw the same behavior).

Any idea how this could have happened and if I should worry about it happening again?

#8 Updated by gboudreau over 8 years ago

  • Status changed from New to Resolved

Can't know for sure what caused this.
It all depends on your created folders, when, with what user, etc.
Since it happened once, it could happen again.

At least now you know how to fix pretty easily, and you know your data is safe, even if it can appear otherwise! :)

#9 Updated by bigfoot65 about 7 years ago

  • Priority changed from High to Normal

#10 Updated by cpg about 7 years ago

  • Status changed from Resolved to Assigned
  • Assignee set to cpg

Issue #766 has been updated by sc4s2cg.

This bug is back in my HDA, running Ubuntu. It started after I added a new 2tb hardrive to greyhole.

When new files are copied to the shares, after a couple minutes the files themselves disappear. The folders are still there. I checked the indvidual hardrives and all files are still on them, they just disappear from the network shares.

Also available in: Atom