Bug #686
Greyhole should default to MySQL due to performance issues - was Re: Greyhole needs more aggressive priority
0%
Description
Greyholes queue gets filled up quickly and it can’t move data off the LZ as quickly as its written. This causes network transfers to stall on the client side with warnings about lack of space.
As the LZ is on the System partition by default this can cause other more serious issues for the system as a whole.
The only reasonable solution I can think of is to tweak greyhole to have a more aggressive approach to priority until it moves data off the LZ marginally quicker than its being written to the LZ when under heavy load.
History
#1 Updated by gboudreau over 11 years ago
Can you help find the correct priority values ?
ionice -n 7 -p `cat /var/run/greyhole.pid`
Execute that, but replace 7 with a lower number. (Higher priority is 0.)
Let us know what value worked to fix the issue you're having.
(Maybe try with 0 or 1, then work your way up until the issue re-appears.)
You can monitor how fast greyhole file transfers are going by using iotop:
iotop -u root -o
Look for greyhole or rsync (spawned by greyhole).
The PRIO column should show "be/#", where # is what you used in the ionice command above.
#2 Updated by gboudreau over 11 years ago
- Assignee set to cpg
- Priority changed from High to Urgent
After doing some tests, I confirmed that changing the priority of GH I/O doesn't change much the speed of file transfers.
But switching from SQLite to MySQL do make a difference:
SQLite:
$ time sudo ./greyhole-test real 1m29.367s
MySQL:
$ time sudo ./greyhole-test real 0m2.955s
The above test takes an old /var/log/messages file, and inserts 1000 tasks in the DB.
It takes 3 seconds to insert those 1k rows using MySQL, and 1m30s using SQLite.
cpg: Unless you find a way to optimize SQLite somehow, I suggest switching to MySQL.
This problem creates problems for most new users trying to migrate a lot of data.
#3 Updated by Zeiva over 11 years ago
I can backup gboudreau on the priority change. It made no noticeably difference and checking with iotop before and after confirms this. The problem is not the speed of moving data but the speed of handling tasks from the looks of it.
I will be moving to MySQL as soon as my queue is done after which I'll report back with my findings.
I can also note that I've now seen several issues arise from the LZ getting full (as it’s on the OS drive) during the day when monitoring it from work. Ranging from VNC stopping to work correctly to the Amahi webgui giving me database errors (both solved by a reboot from SSH).
#4 Updated by cpg over 11 years ago
Were you able to (carefully - not on the fly!) switch to MySQL for the DB engine and try to hose it with data to see how well it responds to load?
We'll switch the default, write a small howto and release ... but it would be good to have positive confirmation ...
#5 Updated by Zeiva over 11 years ago
I'm still letting Greyhole work through the massive queue I've built up during this issue. Once I'm done I have some pretty nice folders to throw at it to test. What would you like me to start with, a 350GB~ folder with 250k of files (tons of pictures etc) or a 2TB folder with tons of large video files? I also have other more mixed folers which will also make for some nice testing I reckon.
I will be using the instructions listed here to switch to MySQL: http://forums.amahi.org/viewtopic.php?f=39&t=1372
#6 Updated by gboudreau over 11 years ago
You should try with lots of files. The current problem is related to the number of tasks queued, so copying many files will created as many tasks as possible.
#7 Updated by Zeiva over 11 years ago
I'll go for the one with tons of small files then. There are also bigger to avrage sized files mixed in there if that has any meaning.
#8 Updated by Zeiva over 11 years ago
Currently testing greyhole with MySQL. So far the freezing is solved and handling lots of small files is far more efficent. I'll see how well it manages to keep the LZ clean when writing lots of large files next. Perhaps that might need tweaking as well.
#9 Updated by cpg over 11 years ago
Please do continue testing as hard as you can! We'd like to release this soon!
Great job!
#10 Updated by cpg over 11 years ago
- Status changed from New to Closed
closing as the transition has been done successfully to mysql.
#11 Updated by cpg over 11 years ago
- Subject changed from Greyhole needs more aggressive priority to Greyhole should default to MySQL due to performance issues - was Re: Greyhole needs more aggressive priority
Also available in: Atom