Project

General

Profile

Bug #672

Landing zone and root drive

Added by blackbird over 10 years ago. Updated about 10 years ago.

Status:
Assigned
Priority:
Normal
Assignee:
Target version:
-
Start date:
11/05/2010
Due date:
% Done:

0%


Description

Had some bad experience with Amahi and /or Greyhole.
I have a 160GB root drive and a 1TB drive for my shares. Root is not in the greyhole pool activated and the shares are all active for pool.
Now i have moved over 160GB to my Server and after nearly 155GB my server crashed (some ruby failure or so)
The reason was that my root drive is full. So i was confused because its not in my pool and all data should be stored in the 1TB drive.

cpg told me this is because of the "landing zone"-thing?!? But i usually plan to move more than 160GB and i dont want to use a OS drive which is bigger.
Rather i would like to use a smaller root drive (like a USB stick or something similar)

So actually my conclusion is :
If the root drive is smaller than the biggest client drive i ll never be able to make a backup from this client!
Is that right???? I cant believe it...

The second thing i have noticed is that my transfer speed fallen down under 10MB/s. I mean the from disk to disk transfer when greyhole workes AND the network transfer. This happens after my crash. Before the crash i had a transferspeed nearly 35-40MB/s. Also not really fast. With Windows on the Server and same Hardware i had 50-60MB/s. (win7 on client and Server)

So i think there is something wrong with Greyhole (maybe samba because of transfer speed? ) or with me because i understood something wrong.

Can u please help me solve the Problem :D

History

#1 Updated by blackbird over 10 years ago

and a second question : How can i reintegrate my shares from old setup to new setup without moving all around the world...There must be a way, because they are laing on the 1TB shares drive like greyhole put them in there.

I ask because i see NO WAY to reintegrate my shares in (let me say) a "noob"-way after i.e. my OS drives crashed.
I also found nothing practical in the Amahi Wiki. The part "Copying you data into Greyhole shares the first time" seems not to be practical for me because of the reason i wrote in the question above ;) By the way, i think the part is not really understandable for noobs. ;)
I have only this 1TB drive and cant move the files to somewhere to integrate them again.

#2 Updated by dinomic about 10 years ago

I too experienced this, having a 60GB SSD as the system drive, also not in the pool. As soon as there's a backlog of files to process via Greyhole, the SSD gets full up and the system just dies (as there's no more disk space on the system drive).

My suggestion would be that, the landing zone should default to the drive with the largest space. Since for Greyhole pooling to work at least one drive must be selected for the pool, then there will always be a pool drive to use as the landing zone, and hence users can avoid this issue by simply not having the system drive in the pool. My own personal workaround has been to have stick files for all Shares - not ideal.

#3 Updated by cpg about 10 years ago

  • Status changed from New to Assigned
  • Assignee set to gboudreau

I believe you are using MySQL for Greyhole, correct?
(you are on Amahi 6/F14, iirc)

Could it be that the SSD is so fast in accepting new files that Greyhole has trouble catching up?

Basically, Little's Law kicked in and the bottleneck has shifted. Perhaps the bottleneck has shifted from being sqlite to being the destination drives?

In essence, this has to be handled, since, no matter what, someone will hit a bottleneck and assuming that greyhole will catch up to the landing zone is not guaranteed for every system configuration?

#4 Updated by gboudreau about 10 years ago

  • Assignee changed from gboudreau to cpg

Indeed, the bottleneck has always been the destinations drives, when using MySQL.
Depending on how large your / partition is, and how much data you copy at once, you might hit that problem.
Most users will hit it at least once during their initial data migration, and never will after that.

Amahi should either provide guidance in choosing and migrating landing zones (Share's Location/Path), or handle the migration by itself.

Migration involves: moving /var/hda/files/share_name to /var/hda/files/drives/drive#/share_name (or anywhere else really), and changing the share definition in smb.conf to point to the new path, then restarting Greyhole & Samba. This should probably be done while Greyhole and Samba are offline, to prevent both services from trying to write / work in the old path, while the copy is ongoing.

My suggestion: provide a command-line script that handles the landing zone migration, and takes in parameters: share name & new share path.
Script would be something like:
- service smbd stop
- service greyhole stop
- change /etc/samba/smb.conf, to put new_share_path in the [ShareName] section
- mv old_share_path new_share_path
- service greyhole start
- service smbd start

#5 Updated by VirtualMe about 10 years ago

Not sure how long restarting the services would take, but could this also be used/shoehorned into having the landing zone be in one of the Greyhole shares? Then the initial copy of a file from the network would land in its destination and would only require n - 1 additional copies instead of n. So for shares that are set to have 2 copies of files, it would only require one additional copy instead of two.

#6 Updated by gboudreau about 10 years ago

No, this can't be done at this time. This would require the Samba VFS module to be much more intelligent than it is now, which would mean copying some of Greyhole's PHP code into C, to be able to take a decision on where to place the file.
I guess that would be possible in the Samba VFS, but never tried, so it's possible that this is just not possible to do in a VFS module.

#7 Updated by dinomic about 10 years ago

cpg, I think you're spot on - at least that's what I believe had happened, ie the SSD was quick to be written to, but the moving to the destination drives is slow in comparison. So if you're moving a lot of data, it won't take long before it fills up the landing zone. The impact of this when the landing zone is on the system drive is that the whole server just dies! No more HDA site, nothing!! Obviously, for headless installs, this is a big problem.

I stand by my original idea that, seeing as Greyhole is only used for the Storage Pool, and you have to explicitly set which drives are in the Storage Pool, it makes sense that the Amahi control panel either explicitly sets the landing zone to be one of the storage pool drives (maybe the one with the largest space), or the user is forced to select which drive they want to be the landing zone.

For my own purposes, in order to not run into this issue again, I'm explicitly setting the directory for each share so that it resides on a specific drive, eg the share "Family_Video" has the "Location" set to "/var/hda/files/drives/drive1/Family_Video", the "Films_Recorded" share has the location set to "/var/hda/files/drives/drive5/Films_Recorded", etc. This suits me, as I actually like to control which drive stuff is stored on. I'm able to spread my storage as I like, and if there's an issue with any drive, I pretty much know what that drive would have had on it, and which other drive has a copy of the same data.

Also available in: Atom