Project

General

Profile

Bug #686

Greyhole should default to MySQL due to performance issues - was Re: Greyhole needs more aggressive priority

Added by Zeiva almost 10 years ago. Updated almost 10 years ago.

Status:
Closed
Priority:
Urgent
Assignee:
Target version:
-
Start date:
11/29/2010
Due date:
% Done:

0%


Description

Greyholes queue gets filled up quickly and it can’t move data off the LZ as quickly as its written. This causes network transfers to stall on the client side with warnings about lack of space.

As the LZ is on the System partition by default this can cause other more serious issues for the system as a whole.

The only reasonable solution I can think of is to tweak greyhole to have a more aggressive approach to priority until it moves data off the LZ marginally quicker than its being written to the LZ when under heavy load.

History

#1 Updated by gboudreau almost 10 years ago

Can you help find the correct priority values ?

ionice -n 7 -p `cat /var/run/greyhole.pid`

Execute that, but replace 7 with a lower number. (Higher priority is 0.)
Let us know what value worked to fix the issue you're having.
(Maybe try with 0 or 1, then work your way up until the issue re-appears.)

You can monitor how fast greyhole file transfers are going by using iotop:

iotop -u root -o

Look for greyhole or rsync (spawned by greyhole).
The PRIO column should show "be/#", where # is what you used in the ionice command above.

#2 Updated by gboudreau almost 10 years ago

  • Assignee set to cpg
  • Priority changed from High to Urgent

After doing some tests, I confirmed that changing the priority of GH I/O doesn't change much the speed of file transfers.

But switching from SQLite to MySQL do make a difference:

SQLite:

$ time sudo ./greyhole-test
real 1m29.367s

MySQL:

$ time sudo ./greyhole-test
real 0m2.955s

The above test takes an old /var/log/messages file, and inserts 1000 tasks in the DB.
It takes 3 seconds to insert those 1k rows using MySQL, and 1m30s using SQLite.

cpg: Unless you find a way to optimize SQLite somehow, I suggest switching to MySQL.
This problem creates problems for most new users trying to migrate a lot of data.

#3 Updated by Zeiva almost 10 years ago

I can backup gboudreau on the priority change. It made no noticeably difference and checking with iotop before and after confirms this. The problem is not the speed of moving data but the speed of handling tasks from the looks of it.

I will be moving to MySQL as soon as my queue is done after which I'll report back with my findings.

I can also note that I've now seen several issues arise from the LZ getting full (as it’s on the OS drive) during the day when monitoring it from work. Ranging from VNC stopping to work correctly to the Amahi webgui giving me database errors (both solved by a reboot from SSH).

#4 Updated by cpg almost 10 years ago

Were you able to (carefully - not on the fly!) switch to MySQL for the DB engine and try to hose it with data to see how well it responds to load?

We'll switch the default, write a small howto and release ... but it would be good to have positive confirmation ...

#5 Updated by Zeiva almost 10 years ago

I'm still letting Greyhole work through the massive queue I've built up during this issue. Once I'm done I have some pretty nice folders to throw at it to test. What would you like me to start with, a 350GB~ folder with 250k of files (tons of pictures etc) or a 2TB folder with tons of large video files? I also have other more mixed folers which will also make for some nice testing I reckon.

I will be using the instructions listed here to switch to MySQL: http://forums.amahi.org/viewtopic.php?f=39&t=1372

#6 Updated by gboudreau almost 10 years ago

You should try with lots of files. The current problem is related to the number of tasks queued, so copying many files will created as many tasks as possible.

#7 Updated by Zeiva almost 10 years ago

I'll go for the one with tons of small files then. There are also bigger to avrage sized files mixed in there if that has any meaning.

#8 Updated by Zeiva almost 10 years ago

Currently testing greyhole with MySQL. So far the freezing is solved and handling lots of small files is far more efficent. I'll see how well it manages to keep the LZ clean when writing lots of large files next. Perhaps that might need tweaking as well.

#9 Updated by cpg almost 10 years ago

Please do continue testing as hard as you can! We'd like to release this soon!
Great job!

#10 Updated by cpg almost 10 years ago

  • Status changed from New to Closed

closing as the transition has been done successfully to mysql.

#11 Updated by cpg almost 10 years ago

  • Subject changed from Greyhole needs more aggressive priority to Greyhole should default to MySQL due to performance issues - was Re: Greyhole needs more aggressive priority

Also available in: Atom