Performance concerns related to MySQL

Discussion corner for Developers of Serendipity.
Post Reply
jdrodrigues
Regular
Posts: 36
Joined: Sun Feb 12, 2006 9:27 pm
Location: London
Contact:

Performance concerns related to MySQL

Post by jdrodrigues »

I am on a shared hosting environment, with a move to a dedicated server / VPN is planned for the near future. On occasion our site ( http://musicix.net ) fails to respond for several seconds to over a minute, before loading. Usually we have no problems with the response time. However sometimes it feels as if it's waiting for something to happen, and my concern is that it has something to do with the database. We do have over 2000 long articles and over 200 categories, so that accounts for any usual lag in load times on a shared server. But as I said, sometimes it will just sit there with nothing happening for more than a minute. I also realize that a shared server is far from ideal for a site like ours, and a lot of our performance problems could be down to that.

I've included warnings from the Runtime Information generated by phpMySQL, any comments/suggestions would be greatly appreciated:

Code: Select all

Slow_queries
499
The number of queries that have taken more than long_query_time seconds. MySQL - Documentation

Handler_read_rnd
62 M
The number of requests to read a row based on a fixed position. This is high if you are doing a lot of queries that require sorting of the result. You probably have a lot of queries that require MySQL to scan whole tables or you have joins that don't use keys properly.

Handler_read_rnd_next
3,035.69 M
The number of requests to read the next row in the data file. This is high if you are doing a lot of table scans. Generally this suggests that your tables are not properly indexed or that your queries are not written to take advantage of the indexes you have.


Created_tmp_disk_tables
493 k
The number of temporary tables on disk created automatically by the server while executing statements. If Created_tmp_disk_tables is big, you may want to increase the tmp_table_size value to cause temporary tables to be memory-based instead of disk-based.

[ tmp table size is 33,554,432 ]


Key_reads
32 M
The number of physical reads of a key block from disk. If Key_reads is big, then your key_buffer_size value is probably too small. The cache miss rate can be calculated as Key_reads/Key_read_requests.

[ key buffer size is 536,870,912 ]


Select_full_join
112 k
The number of joins that do not use indexes. If this value is not 0, you should carefully check the indexes of your tables.

Select_range_check
46
The number of joins without keys that check for key usage after each row. (If this is not 0, you should carefully check the indexes of your tables.)

Opened_tables
1,782 k
The number of tables that have been opened. If opened tables is big, your table cache value is probably too small.

[The table cache value is 256 - is that bad?]

( for reference - no warning: Table_locks_immediate 39 M
The number of times that a table lock was acquired immediately. )

Table_locks_waited
57 k
The number of times that a table lock could not be acquired immediately and a wait was needed. If this is high, and you have performance problems, you should first optimize your queries, and then either split your table or tables or use replication.
Many thanks in advance,
Joel
Joel Rodrigues, The Music Information Exchange
News For Musicians & The Music Industry
webdev@musicix.net / http://musicix.net
garvinhicking
Core Developer
Posts: 30022
Joined: Tue Sep 16, 2003 9:45 pm
Location: Cologne, Germany
Contact:

Re: Performance concerns related to MySQL

Post by garvinhicking »

Hi!

Hm, the large key buffers etc. do indicate that your blog really takes a lot of MySQL performance. It might be that there's something left to optimize. You might want to try Serendipity 1.1.1, as this has some SQL optimizations that might take effect for you.

Plus, the list of your used event plugins could help to analyze DB traffic. Your sidebar plugins don't seem as if they create much SQL traffic.

There's one option to allow debug statistics of SQL queries in your blog, by editing include/db/mysql.inc.php and set the $benchmark variable to true, everywhere you see it declared.

If you don't use per-category templates, you might want to reduce S9Y additional traffic by making your CSS file static and call that from your index.tpl template instead of the dynamic CSS URL, and also make the CSS print stylesheet static...

HTH,
Garvin
# Garvin Hicking (s9y Developer)
# Did I help you? Consider making me happy: http://wishes.garv.in/
# or use my PayPal account "paypal {at} supergarv (dot) de"
# My "other" hobby: http://flickr.garv.in/
TKS
Regular
Posts: 199
Joined: Thu Nov 11, 2004 1:34 am
Location: VA, USA
Contact:

Post by TKS »

I also have this problem quite often at http://linux-blog.org

Especially alarming to me is this information displayed inside phpmyadmin:

Slow_queries 5,248 The number of queries that have taken more than long_query_time seconds. MySQL - Documentation


It seems it takes quite sometime for my blog to fully display. Once it starts going, things are fine...but the initial display is very slow. I've got another blog on the EXACT same server that displays in half the time...but he hasn't been updating since .8 like I have either.

Do I need to clear out some of my tables? I'm thinking I should track spam attacks via flatfile and clean out that table (or even drop it if possible) so that isn't searched during queries (perhaps this is slowing things down).

I'd like to get this thing performing like it used to if at all possible. Maybe disable exit tracking? I'm just guessing here...

I did notice that there are 57MB in my serendipity_visitors table...would it be wise to disable this? If so, how?
TKS
Regular
Posts: 199
Joined: Thu Nov 11, 2004 1:34 am
Location: VA, USA
Contact:

Post by TKS »

Wow...I didn't see the # of slow queries that jdrodrigues was posting about...my 5 thousand plus seem HUGE compared to that.

Does anyone have input on how I can resolve this issue? I'm thinking of scrapping everything and going with a new site to solve the problems...
TKS
Regular
Posts: 199
Joined: Thu Nov 11, 2004 1:34 am
Location: VA, USA
Contact:

Post by TKS »

Ok...In my case, I found out what was causing the problems...2 plugins made my page render time drop from 30 seconds to around 5 seconds...


1. Extended properties for entries
2. Statistics (visitors)

I installed google analytics anyway, so my visitors are tracked and I wasn't using anything special with Extended properties (2 entries which are fixed now).

Removing only extended properties sped things up around 8 seconds and statistics sped things up the rest of the way. Hope this helps you out jdrodrigues!
garvinhicking
Core Developer
Posts: 30022
Joined: Tue Sep 16, 2003 9:45 pm
Location: Cologne, Germany
Contact:

Post by garvinhicking »

Hi!

Yes, sadly the visitors plugin is known to cause huge data collections, causing a page to easily slow down :( :(

Thanks for mentioning this here!

Regards,
Garvin
# Garvin Hicking (s9y Developer)
# Did I help you? Consider making me happy: http://wishes.garv.in/
# or use my PayPal account "paypal {at} supergarv (dot) de"
# My "other" hobby: http://flickr.garv.in/
jdrodrigues
Regular
Posts: 36
Joined: Sun Feb 12, 2006 9:27 pm
Location: London
Contact:

Post by jdrodrigues »

Hello everyone...

I removed the visitor statistics plug-in a long time ago - it never worked properly anyway. But I can't get rid of the extended properties plug-in - I need that.

I'll try changing the stylesheet to a static link and enable SQL query debug, and let you knwo what happens.

I'm quite concerned about these though:
Select_full_join
112 k
The number of joins that do not use indexes. If this value is not 0, you should carefully check the indexes of your tables.

Select_range_check
46
The number of joins without keys that check for key usage after each row. (If this is not 0, you should carefully check the indexes of your tables.)

Unfortunately because of all the changes I've made to little bits of code here and there, moving to 1.1 would take more time than I have to spare just now.
Joel Rodrigues, The Music Information Exchange
News For Musicians & The Music Industry
webdev@musicix.net / http://musicix.net
garvinhicking
Core Developer
Posts: 30022
Joined: Tue Sep 16, 2003 9:45 pm
Location: Cologne, Germany
Contact:

Post by garvinhicking »

Hi!

Maybe you'd like to debug all incoming queries a little bit in detail.

Check out this version of the mysql layer:

http://svn.berlios.de/viewcvs/serendipi ... iew=markup

In the serendipity_dbquery you can set $benchmark to true. It will then log to a database table. You need to create two tables first, though:

Code: Select all

CREATE TABLE `BLOGLOG` (
  `request` varchar(255) NOT NULL default '',
  `timestamp` timestamp(14) NOT NULL,
  `sql` text NOT NULL,
  `exec_time` varchar(50) NOT NULL default '0',
  `ip` varchar(20) NOT NULL default ''
) TYPE=MyISAM;

CREATE TABLE `BLOGLOG_TOTAL` (
  `sql` text NOT NULL,
  `counter` int(11) NOT NULL default '0'
) TYPE=MyISAM;
Then you can check the tables to see which queries are created when. You can go trhough those queries with "EXPLAIN ... " and see which one of them do not use any indexes.

Be sure to know that if you use an GROUP BY or ORDER BY statement, MySQL will need to create temporary tables/use filesort. But it should use an index for every query.

If you find queries that don't use indices, please report here.

Best regards,
Garvin
# Garvin Hicking (s9y Developer)
# Did I help you? Consider making me happy: http://wishes.garv.in/
# or use my PayPal account "paypal {at} supergarv (dot) de"
# My "other" hobby: http://flickr.garv.in/
jdrodrigues
Regular
Posts: 36
Joined: Sun Feb 12, 2006 9:27 pm
Location: London
Contact:

Post by jdrodrigues »

Hi,

I've installed the benchmark version of the mysql.inc.php file and created the tables. I'll let you know what happens.

Do you think rebuilding indexes is necessary/will help performance?
Joel Rodrigues, The Music Information Exchange
News For Musicians & The Music Industry
webdev@musicix.net / http://musicix.net
garvinhicking
Core Developer
Posts: 30022
Joined: Tue Sep 16, 2003 9:45 pm
Location: Cologne, Germany
Contact:

Post by garvinhicking »

Hi!
I've installed the benchmark version of the mysql.inc.php file and created the tables. I'll let you know what happens.
Check those regularly, they easily grow very large.
Do you think rebuilding indexes is necessary/will help performance?
You could run "optimize table" on each table, yes.

Best regards,
Garvin
# Garvin Hicking (s9y Developer)
# Did I help you? Consider making me happy: http://wishes.garv.in/
# or use my PayPal account "paypal {at} supergarv (dot) de"
# My "other" hobby: http://flickr.garv.in/
jdrodrigues
Regular
Posts: 36
Joined: Sun Feb 12, 2006 9:27 pm
Location: London
Contact:

Post by jdrodrigues »

Hi Garvin,

There seems to be a problem with :

$start = microtime_float();


I don't remember what it was because I quickly changed $benchmark to false.

- Joel
Joel Rodrigues, The Music Information Exchange
News For Musicians & The Music Industry
webdev@musicix.net / http://musicix.net
garvinhicking
Core Developer
Posts: 30022
Joined: Tue Sep 16, 2003 9:45 pm
Location: Cologne, Germany
Contact:

Post by garvinhicking »

Hi!

Might be your PHP version; microtime_float() is not available in all of them, check the PHP documentation for when it was added?

Regards,
Garvin
jdrodrigues wrote:Hi Garvin,

There seems to be a problem with :

$start = microtime_float();


I don't remember what it was because I quickly changed $benchmark to false.

- Joel
# Garvin Hicking (s9y Developer)
# Did I help you? Consider making me happy: http://wishes.garv.in/
# or use my PayPal account "paypal {at} supergarv (dot) de"
# My "other" hobby: http://flickr.garv.in/
jdrodrigues
Regular
Posts: 36
Joined: Sun Feb 12, 2006 9:27 pm
Location: London
Contact:

Post by jdrodrigues »

Hello again...

I've decided to move web hosts again...

This time to Verio VPS (virtual private server), along with an upgrade (hopefully) to s9y 1.2 and more recent versions of PHP & MySQL

Will let you know how things work then. So much to do, so little time...
Joel Rodrigues, The Music Information Exchange
News For Musicians & The Music Industry
webdev@musicix.net / http://musicix.net
jdrodrigues
Regular
Posts: 36
Joined: Sun Feb 12, 2006 9:27 pm
Location: London
Contact:

Post by jdrodrigues »

Well, I did it! I've now got http://musicix.net running on Verio USA's VPS3 hosting. I also upgraded s9y to 1.1.2 and I've got PHP 5.1.5
and MySQL 5.0.24!

I have not experienced any time lags waiting for the site to react so far. I do see a couple of warnings however:

Code: Select all


Handler_read_rnd	

574 k	 

The number of requests to read a row based on a fixed position. This is high if you are doing a lot of queries that require sorting of the result. You probably have a lot of queries that require MySQL to scan whole tables or you have joins that don't use keys properly.

Handler_read_rnd_next	

6,907 k	 

The number of requests to read the next row in the data file. This is high if you are doing a lot of table scans. Generally this suggests that your tables are not properly indexed or that your queries are not written to take advantage of the indexes you have.


Created_tmp_disk_tables	

5,443	 

The number of temporary tables on disk created automatically by the server while executing statements. If Created_tmp_disk_tables is big, you may want to increase the tmp_table_size value to cause temporary tables to be memory-based instead of disk-based.

Select_full_join	

904	 

The number of joins that do not use indexes. If this value is not 0, you should carefully check the indexes of your tables.


Sort_merge_passes	

174	 

The number of merge passes the sort algorithm has had to do. If this value is large, you should consider increasing the value of the sort_buffer_size system variable


Opened_tables	

801	 

The number of tables that have been opened. If opened tables is big, your table cache value is probably too small.
I am reading http://dev.mysql.com/books/hpmysql-excerpts/ch06.html for ideas on resolving/improving some of these issues, though I don't know how much I'll be able to change before eventually moving to a dedicated server. I'm now up to 2095+ articles and 267 categories.

Any thoughts & comments are welcome! - Joel
Joel Rodrigues, The Music Information Exchange
News For Musicians & The Music Industry
webdev@musicix.net / http://musicix.net
Post Reply