Recent site performance issues

[Karlos]

As most of you will know (if not, see here: http://www.amiga.org/forums/showthread.php?t=54618), the site has been sluggish of late and with intermittent errors popping up.

We've been monitoring the situation for a while and we've ascertained that the site is occasionally tip toeing over the maximum allowed concurrent connections to the database at peak times.

Until a more permanent solution can be found (read a beefier server/hosting package), I've been given the go-ahead on an evil plan I hatched to try and mitigate the issue to some extent. This plan is to basically cache content (for a short time) that is served to all those visitors that aren't logged in.

Allow me to expand on that:

On any normal day, logged-in users are typically outnumbered by anonymous visitors by anything up to a factor of 10 or more. By caching the generated content of several key pages for a few short minutes, the total number of database queries per second can be dramatically reduced and with it, the danger that the server will go over the limit.

On the downside, if you are a regular lurker, you won't necessarily see the most up-to-the-minute view of the page, so perhaps it's time to log in at long last

Also on the downside, the thread view statistics will be skewed, since for every cached serving, no counters will be updated. Still, better than not being able to see the site at all, right?

When I upload the patch, there could be some momentary shenanigans. Please stand by...

[Karlos]

Sneaky kludges ftw!

Think of it as enforcing the Royston Vasey policy :lol:

vBulletin is a beast when it comes to resource utilisation. What I've done is to rename several key php files and add subsitutes for them that add the caching aspect. It's managed in a way that requires no database queries at all when you hit the cache. However, you do hit the disk, so it isn't for free, either.

[Karlos]

One other caveat to note is that if you aren't logged in, the indication as to which threads have been read or not will also not be accurate since you will likely be viewing a cached version of the content that someone else was responsible for generating.

[Karlos]

Memcached not any help?

Sure, but you need to be running it and that usually means you have your own dedicated server.

[Karlos]

Oh I see. Time for the owners to put the subscription fees and ad revenue towards a server then.

Whatever they do longer term, we need to at least mitigate the recent problems. Taking the strain off the database will hopefully accomplish that. Every time you read a page in a thread there are several database queries and typically more writes than there are than reads. For a "cached" page, you are getting several filesystem reads: one to check the file exists and another  to check it's changed time and finally retrieving it's contents. The first two reads actually are only one since checking for the file existing causes an fstat() call and php caches the results of those over the lifetime of a script.

[Karlos]

Ok, it looks like this plan requires more work - it seems likely that at the moment you log in, that first page gets cached.

I will revert the changes. Luckily, that's not difficult.

-edit-

*done*

[Karlos]

There are some other weird things happening with the site.  When I was reading this thread, I happened to look up at the top of the page and it said "Welcome, Transition, you last logged in today at 5:16 AM".

I am not Transition, I am Lando.  I logged out and logged back in as me, how did I ever get logged in as Transition?

You didn't. You were served a page that Pyromania managed to get cached after logging in. If you had gone anywhere, you'd not have had access to any of his account.

[Karlos]

Anyway, theory is good, implementation needs more work :lol:

[Karlos]

This would be my dream come true. Can you imagine what i could do having Admin powers? Can you imagine how annoying i could be if i had full control of the site? For a start i could annoy Karlos even more than i do now...

Except you couldn't, because all you were getting was a copy of a page with his name on it.

Anyway, time for take #2...

[Karlos]

@Karlos

As long as you are coding new bugs into the site, could you reimplement the "possession"  bug so that I can become Piru?

I never got to be him.  I feel so left out.

I have my pony trekking gear all packed up along with my portable Finnish-English dictionary.

Well, you just missed your chance earlier. Except you'd not necessarily assume any of his powers

[Karlos]

Right, we are at iteration #2 of this crazy scheme. Hopefully you'll all still be yourselves for the next 15 mins or I'm abandoning it for now :lol:

[Karlos]

Well, this concludes testing for now. I'm going to turn off the cache mechanism for now and pick it up again when I can monitor it for a while

[Karlos]

Normal service resumed...

[Karlos]

Notice

The cache is activated again for a third round of testing after last night. If you aren't logged in, don't be surprised when pages don't seem to update or if visiting a thread that on the main page claims has no new posts suddenly has a bunch - it's all down to when the cached version of a given page was saved. The cache is "dumb" for a reason - tying to invalidate cached pages on the basis that a linked page changed would require code that would defeat the entire point of trying to reduce server load.

If, when you log in, something isn't right, then please flag it up.

Only the main index, forum index and thread displays have any caching.

[Karlos]

I wonder if the gibberish was gz compressed content. The proxy should be telling the site not to accept this content encoding type - the cache has been written to serve both kinds of data (though it is always stored compressed).

Amusingly enough, I had serious delays just now, but my other "incognito" guest session was perfectly fast