Amiga.org
Amiga.org specific forums => Amiga.org Discussion and Site Feedback => Topic started by: Karlos on July 10, 2011, 07:54:50 PM
-
As most of you will know (if not, see here: http://www.amiga.org/forums/showthread.php?t=54618), the site has been sluggish of late and with intermittent errors popping up.
We've been monitoring the situation for a while and we've ascertained that the site is occasionally tip toeing over the maximum allowed concurrent connections to the database at peak times.
Until a more permanent solution can be found (read a beefier server/hosting package), I've been given the go-ahead on an evil plan I hatched to try and mitigate the issue to some extent. This plan is to basically cache content (for a short time) that is served to all those visitors that aren't logged in.
Allow me to expand on that:
On any normal day, logged-in users are typically outnumbered by anonymous visitors by anything up to a factor of 10 or more. By caching the generated content of several key pages for a few short minutes, the total number of database queries per second can be dramatically reduced and with it, the danger that the server will go over the limit.
On the downside, if you are a regular lurker, you won't necessarily see the most up-to-the-minute view of the page, so perhaps it's time to log in at long last :)
Also on the downside, the thread view statistics will be skewed, since for every cached serving, no counters will be updated. Still, better than not being able to see the site at all, right?
When I upload the patch, there could be some momentary shenanigans. Please stand by...
-
Sneaky kludges ftw! :)
-
Sneaky kludges ftw! :)
Think of it as enforcing the Royston Vasey policy :lol:
vBulletin is a beast when it comes to resource utilisation. What I've done is to rename several key php files and add subsitutes for them that add the caching aspect. It's managed in a way that requires no database queries at all when you hit the cache. However, you do hit the disk, so it isn't for free, either.
-
One other caveat to note is that if you aren't logged in, the indication as to which threads have been read or not will also not be accurate since you will likely be viewing a cached version of the content that someone else was responsible for generating.
-
Think of it as enforcing the Royston Vasey policy :lol:
vBulletin is a beast when it comes to resource utilisation. What I've done is to rename several key php files and add subsitutes for them that add the caching aspect. It's managed in a way that requires no database queries at all when you hit the cache. However, you do hit the disk, so it isn't for free, either.
Memcached not any help?
-
Memcached not any help?
Sure, but you need to be running it and that usually means you have your own dedicated server.
-
Sure, but you need to be running it and that usually means you have your own dedicated server.
Oh I see. Time perhaps for the owners to put the subscription fees and ad revenue towards a server then?
-
Oh I see. Time for the owners to put the subscription fees and ad revenue towards a server then.
Whatever they do longer term, we need to at least mitigate the recent problems. Taking the strain off the database will hopefully accomplish that. Every time you read a page in a thread there are several database queries and typically more writes than there are than reads. For a "cached" page, you are getting several filesystem reads: one to check the file exists and another to check it's changed time and finally retrieving it's contents. The first two reads actually are only one since checking for the file existing causes an fstat() call and php caches the results of those over the lifetime of a script.
-
There are some other weird things happening with the site. When I was reading this thread, I happened to look up at the top of the page and it said "Welcome, Transition, you last logged in today at 5:16 AM".
I am not Transition, I am Lando. I logged out and logged back in as me, how did I ever get logged in as Transition?
-
Good job Karlos, all those explanations and solutions are right-on in my mind.
Amiga.org is umm, too health of a community at times. A good problem to have my boss would say.
-
There are some other weird things happening with the site. When I was reading this thread, I happened to look up at the top of the page and it said "Welcome, Transition, you last logged in today at 5:16 AM".
I am not Transition, I am Lando. I logged out and logged back in as me, how did I ever get logged in as Transition?
It was a momentary flux in the Matrix.
:)
-
There are some other weird things happening with the site. When I was reading this thread, I happened to look up at the top of the page and it said "Welcome, Transition, you last logged in today at 5:16 AM".
I am not Transition, I am Lando. I logged out and logged back in as me, how did I ever get logged in as Transition?
Yeah, I'm logged in. On the front page for this thread is says " Pyromania" is the last person to comment in this thread. I've checked it a few times before posting this and Lando is the last person to post in the thread.
So did "Pyromania" actually post in the thread and I can't see it even though I am logged in (for a while and several refreshes) or does the system think Lando was Pyromania?
EDIT:
Ok, now after submitting my post I see the Goose and Pyromania had both posted. I couldn't see either post even after opening up another refreshing a couple times to see if it was an error and then trying another browser window and refreshing. Weird.
-
Ok, it looks like this plan requires more work - it seems likely that at the moment you log in, that first page gets cached.
I will revert the changes. Luckily, that's not difficult.
-edit-
*done*
-
There are some other weird things happening with the site. When I was reading this thread, I happened to look up at the top of the page and it said "Welcome, Transition, you last logged in today at 5:16 AM".
I am not Transition, I am Lando. I logged out and logged back in as me, how did I ever get logged in as Transition?
You didn't. You were served a page that Pyromania managed to get cached after logging in. If you had gone anywhere, you'd not have had access to any of his account.
-
Anyway, theory is good, implementation needs more work :lol:
-
There are some other weird things happening with the site. When I was reading this thread, I happened to look up at the top of the page and it said "Welcome, Transition, you last logged in today at 5:16 AM".
I am not Transition, I am Lando. I logged out and logged back in as me, how did I ever get logged in as Transition?
This would be my dream come true. Can you imagine what i could do having Admin powers? Can you imagine how annoying i could be if i had full control of the site? For a start i could annoy Karlos even more than i do now...
-
@Karlos. Exactly who are those people lurking about here without logging in? I've always wondered about this. Are they the same 100 or so people or are they different all the time? Some stats would be good.
-
This would be my dream come true. Can you imagine what i could do having Admin powers? Can you imagine how annoying i could be if i had full control of the site? For a start i could annoy Karlos even more than i do now...
Except you couldn't, because all you were getting was a copy of a page with his name on it.
Anyway, time for take #2...
-
@Karlos
As long as you are coding new bugs into the site, could you reimplement the "possession" bug so that I can become Piru? :D
I never got to be him. I feel so left out. :D
I have my pony trekking gear all packed up along with my portable Finnish-English dictionary. :)
-
@Karlos
As long as you are coding new bugs into the site, could you reimplement the "possession" bug so that I can become Piru? :D
I never got to be him. I feel so left out. :D
I have my pony trekking gear all packed up along with my portable Finnish-English dictionary. :)
Well, you just missed your chance earlier. Except you'd not necessarily assume any of his powers :(
-
Right, we are at iteration #2 of this crazy scheme. Hopefully you'll all still be yourselves for the next 15 mins or I'm abandoning it for now :lol:
-
Damnit, I was rather looking forward to logging in and being someone else.
I had some proper havok planned. :(
-
If I can't become Piru and assume his powers of PFS3 Megacoding Excellence then maybe you could just let me become Franko and assume his powers of Frankohol-induced hilariousness :D
-
Well, this concludes testing for now. I'm going to turn off the cache mechanism for now and pick it up again when I can monitor it for a while :)
-
Normal service resumed...
-
Thanks for all the hard work Karlos. We know you are getting things done and we appreciate it!
But I feel a bit left out.
no one wanted to be me...:(
-
Notice
The cache is activated again for a third round of testing after last night. If you aren't logged in, don't be surprised when pages don't seem to update or if visiting a thread that on the main page claims has no new posts suddenly has a bunch - it's all down to when the cached version of a given page was saved. The cache is "dumb" for a reason - tying to invalidate cached pages on the basis that a linked page changed would require code that would defeat the entire point of trying to reduce server load.
If, when you log in, something isn't right, then please flag it up.
Only the main index, forum index and thread displays have any caching.
-
@Karlos
Thanks for the update. I tried logging in about 10 minutes ago and sat there forever. Tried again and logged in OK. About 15 minutes ago I tried viewing the "classic" version of the site with IBrowse and received gibbrish in this thread.
Anyways, thanks for your efforts to improve access. Using OWB right now at the main website.
---
redfox
-
Well ... that was interesting. I submitted my last post and sat at 228KB / 228KB for about three minutes before the rest of the webpage finished downloading to my machine.
---
redfox
-
I wonder if the gibberish was gz compressed content. The proxy should be telling the site not to accept this content encoding type - the cache has been written to serve both kinds of data (though it is always stored compressed).
Amusingly enough, I had serious delays just now, but my other "incognito" guest session was perfectly fast :)
-
-- double post ---
-
When you log in, you will be subject to any db-induced delays. When you aren't, you'll probably find it a lot faster provided the cache is still warm. Unless you are the unlucky one that requests a page that has timed out and needs to be regenerated :D
-
If the problem can be fixed "by throwing hardware at it", drop me a PM... I've got a big stack'o'servers looking at me daily begging for a decent job...
-
Well ... that was interesting. I submitted my last post and sat at 228KB / 228KB for about three minutes before the rest of the webpage finished downloading to my machine.
---
redfox
The kind of bugs I am looking for just now are things like "Becoming Piru", which is indicative of a serious bug in the cache system.
The site crawling like a wounded snail after you logged in is basically down to the DB server being too busy to respond to vBulletin's queries. Serving cached pages doesn't actually hit vBulletin at all, let alone the DB.
-
If the problem can be fixed "by throwing hardware at it", drop me a PM... I've got a big stack'o'servers looking at me daily begging for a decent job...
You should talk to the owner, I'm just the guy with duct tape :lol:
-
@Karlos
A few moments ago I tried to logout and eventually timed out.
OWB Requester contents as follows ...
Loading http://www.amiga.org/forums/login.php?do=logout&logouthash=1310416568-ec38b8e5174139829ab680a7dd065b55c7469090 failed!
Failure when receiving data from the peer
Please check the correct address for typing errors such as ww.example.com instead of http://www.example.com
If you are unable to load any pages, check your computer's network connection.
If your computer or network is protected by a firewall or proxy, make sure that OWB is permitted to access the Web.
Is this also due to DB server being too busy?
---
redfox
-
@Karlos
A few moments ago I tried to logout and eventually timed out.
OWB Requester contents as follows ...
Loading http://www.amiga.org/forums/login.php?do=logout&logouthash=1310416568-ec38b8e5174139829ab680a7dd065b55c7469090 failed!
Failure when receiving data from the peer
Please check the correct address for typing errors such as ww.example.com instead of http://www.example.com
If you are unable to load any pages, check your computer's network connection.
If your computer or network is protected by a firewall or proxy, make sure that OWB is permitted to access the Web.
Is this also due to DB server being too busy?
---
redfox
Very likely, unfortunately. Absolutely every page you try to access (including the logout one) results in several DB queries that if it's very busy can end up in a queue and you wait and wait. However, if the db server is feeling particularly bitchy, it just drops the connection to the application (or prevents the application from opening it in the first place) and then you get a proper 500 error, which may or may not be intercepted by vBulletin's error handler and if so, you get vB's "Database Error" page.
-
So, anybody found themselves logged in as anybody else yet?
If not, I may leave the cache enabled overnight...
-
So, anybody found themselves logged in as anybody else yet?
If not, I may leave the cache enabled overnight...
No. I'm still me. This sucks. :confused:
:roflmao:
-
Unfortunately, it seems the cache mechanism just isn't going to function well enough without a lot of additional work. Some folks couldn't log in and the cache got polluted with IP ban notifications caused when something unwelcome tried to crawl the site meaning all and sundry were suddenly shocked to see they'd been IP banned.
So, we're back to square one.
-
Karlos,
Before I forget it (and sorry to be off-topic), if the "top" stats on the front page are slamming the server, I honestly don't think anyone here would even miss them if you went into the Admin and killed those blocks altogether.
I know when I'm visiting Amiga.org, I don't even notice them (outside of the latest blog posts block that is). Matter of fact, I think the "latest blog posts" block is entirely why I added those options. Not sure.
Wayne
-
Karlos,
Before I forget it (and sorry to be off-topic), if the "top" stats on the front page are slamming the server, I honestly don't think anyone here would even miss them if you went into the Admin and killed those blocks altogether.
I've left those cached (updates every 15mins) as they have no dependency on the viewer being logged in or out.
-
I know you guys are working hard on getting things working, so this is not to complain but just an update.
Last 48 hours seems like the site is super slow. I get two different message when it times out now. One that is still looks like the Amiga.org site and says the server times out, the other one that somes times comes up is all white and has the vBulletin logo in the top left and says the database timed out.
-
I'm getting frequent timeouts as well, but when I'm not everything seems to be proceeding at normal speed.
-
404 errors on some posts too that I've noticed. :-/
-
4:06 AM, MST - when I went to post this, the server hung on sending the post page for a good 45 secs:
--- snip ---
vBulletin Message
The server is too busy at the moment. Please try again later.
--- snip ---
After the timeout/too busy msg, I hit a.org in my bookmarks and it came up immediately. Odd stuff.
-
Just to add to the information, I'm seeing slowdowns, the odd 500 error and today I saw an IIS6 style error page saying there was a Database error and to refresh. The site performance for me is achingly slow and I'm on a 50Mb/s connection and every other site I visit is almost instant...
Paul
-
4:06 AM, MST - when I went to post this, the server hung on sending the post page for a good 45 secs:
--- snip ---
vBulletin Message
The server is too busy at the moment. Please try again later.
--- snip ---
After the timeout/too busy msg, I hit a.org in my bookmarks and it came up immediately. Odd stuff.
LOL. I got this twice today so thought i would quote you on it and when i tried it came up the third time. Too funny!
But i did find it strange that the server was too busy with only 65 people online. What does this mean Karlos!? What does this mean??
-
Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator, webmaster@amiga.org and inform them of the time the error occurred, and anything you might have done that may have caused the error.
More information about this error may be available in the server error log.
Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.
Apache mod_fcgid/2.3.6 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635 Server at amiga.org Port 80
Plus
The server is too busy at the moment.
Basically, the new owners are cheap bastards, have pocketed the donation money & kitty money, have re-instated ads while letting the site languish on the cheapest shittiest deal they could find.
Putting it any other way is just beating around the bush.
-
Wow I've had more errors today than a CUSA press release... :D
Thought at first it was a sneak attack on me account again, so I sharpened up the squirrels teeth and put on their little black balaclavas ready to launch and all out tickling stick attack... :madashell:
Luckily I read these posts first and it was a false alarm (stand down , Humprey, Albert, Jimmy and the rest of the SNP (that's Squirrel Ninja Platoon))... :)
Seems everything is fine and it's just the server/site falling apart... phew... ;)
-
Wow... looks like Karlos has abandoned ship... :eek:
Posted a few moments ago on Wayne's site by Karlos...
I'm kinda gutted at the state of the place. I'm no so bothered about who is and who isn't flavour of the month over there; these things always go round in cycles from my experience.
What upsets me is the seeming lack of basic upkeep of the place. The cracks are really showing now, with frequent delays, db dropouts and associated errors.
I've done what I can for the place. First there was the migration and all the 3rd party xoops stuff to import. Since moving to vb, all 20 or so of classic users got short shrift, so I created a proxy just for them. The place was breached twice (injected php, nothing was added to or taken from the db) by automated attacks. Both times I was able to fix it, do the post mortem and plug whatever hole had allowed it in the first place. We've had various bugs crop up that I've fixed or worked around (that bloody shareads() thing was the worst, living as it did inside eval()'d code).
More recently, the site started to grind, so I reviewed all the basic performance stuff. And when fixing htaccess rules, slow queries, insert/update prority policies, on-the-fly image re-sampling and a dozen other aspects wasn't enough, I started a "donut" cache project to try and reduce the query overhead. In the end, the latter needs much more work to be reliable and robust. I was going to look into that this weekend.
And then it hit me. Why am I even bothering? It's not as if *I* own the place and as far as I can tell, there's nothing actually wrong with it that couldn't be fixed by getting a heavier duty hosting package.
So, yesterday I logged out for the first time since it migrated.
Full Link...
Karlos Abandons Ship... (http://whyzzat.com/threads/why-do-i-even-care-any-more.1941/page-2)
-
We will do what is required to solve the problem including moving Amiga.org to the more robust DiscreetFX servers. Problem will get resolved ASAP.
-
Well, the fact that the errors are really hitting us now (The weekend) seams clear that it IS usage centered. That or the host sets the limits by the week or something...
-
Wow... looks like Karlos has abandoned ship...
Well, it doesn't say that exactly...
Just that he's been working his @$$ off trying to keep this boat afloat and he's stepping back for a bit...
I can tell from what I've read of the situation and what I know of Karlos as a web guy; he's persistent, a creative problem solver, and he cares..
That last one is a great feature to have in an admin, but it's a dangerous one too.. ;-) You don't want your best techs burning out..
He's earned a break with all the great work he's done...
Take your time Karlos, and thanx for everything you've done so far.. (and for more when you get back!!!)
:)
desiv
"Hey, isn't desiv the guy who disagrees with Karlos alot?? What's up?" (I hear you say)
Yes, but I respect his abilities as a Web guy.. I am not, and know how hard that is...
-
Wow... looks like Karlos has abandoned ship... :eek:
Posted a few moments ago on Wayne's site by Karlos...
You're such a -=Moderated=- Franko. He said no such thing. All I see is a post about all the work he's done. Looks like he just took a breather for a minute. Sometimes we get really deep into something. Looks like he sees the best decision is not software patches and repairs, but hardware.
We all need a break sometimes.
Stop being a -=Moderated=- troll sometimes.
-
@ Desiv
I sure as hell hope Karlos aint giving it up altogether, that guy works his arse off round here all hours of the day and night keeping things running as smooth as he can, hopefully he's just taking a well earned break like you say otherwise I dread to think how this place would keep afloat (technically wise I mean) without him... :)
Apart from Karlos & Argo the other so called mods don't seem to bother their arse with the site "SlvrDrgn" or whatever he's called pops in here less than Brigadoon makes an appearance, perhaps it time for some new more dedicated mods to help out who can devote more time to the site along with the new server, I vote for for Xdelusion & J_Golden as both these guys are well liked, fair & decent guys that all here would respect along with Karlos & Argo... :)
-
You're such a -=Moderated=- Franko. He said no such thing. All I see is a post about all the work he's done. Looks like he just took a breather for a minute. Sometimes we get really deep into something. Looks like he sees the best decision is not software patches and repairs, but hardware.
We all need a break sometimes.
Stop being a -=Moderated=- troll sometimes.
-=Moderated=- I never said anything bad about him -=Moderated=-... :rolleyes:
Gawd... it's -=Moderated=- like you that really need to take a bloody long hard look in a mirror and see what -=Moderated=- you are, -=Moderated=-...
-
Thought at first it was a sneak attack on me rectum again, so I sharpened up me nipples and put on me little black bra ready to launch and all out tickling dick attack... :madashell:
Luckily I read these posts first and it was a false alarm (stand down , Humprey, Albert, Jimmy and the rest of the SNP (that's Squirrel Ninja Platoon))... :)
Seems everything is fine... phew... ;)
:roflmao:
-
God I need a life. I'm such a dunderheid, I'm sorry I'm a bad idiot... :rolleyes:
Gawd...I'm a dickhead that really needs to take a bloody long hard look in a mirror and see what an utter bawbags I am, bloody a-hole go bile yer heid...
:roflmao:
-
:roflmao:
Imitation is the sincerest form of flattery... so they say but in your case it's more of Imitation is just a sad wee nebbit faced jobbie who can't come up with anything original even after rubbing his two brain cells together... :roflmao:
PS: at least your last post proves you do have a slight sense of humour even though you are right wee fandan at times... ;)
-
Imitation is the sincerest form of flattery... so they say but in my case it's more of Imitation is just a sad wee nebbit faced jobbie who can't come up with anything original even after I tried rubbing my two brain cells together... :roflmao:
PS: at least this last proves I do not have the slight sense of humour and I'm a total fandan... ;)
:roflmao:
-
@ AmigaHairyDick -=Moderated=-
Try harder wee man, cause yer mince at this game ya wee -=Moderated=-... :)
-
Even when it's harder, I have a small wee wee, oh man, also, I am a turd burglar... :)
:roflmao:
-
-
Ok ok, Franko and Amigaheretic, as much as BOTH of you are making me cry with laughter (literally) lets calm down a bit. Not sayin' you have to kiss and makeup, but let's at least keep somewhat on topic, eh?
I'd like to +1 what desiv said. There are far too many mods on right now and fewer who know how things work behind the curtain. This site would crash and burn without them and we should show a bit more support.
Have a cold one on me, Karlos, you deserve it.
Just send me the bill via Paypal and I'll cover it no problem! <--Not a joke. Pm me, bud and $4-5 is yours...