The Bad Gateway thing is really annoying

[tig]

I'd respond to one of the other threads on "Bad Gateway" but I can't open any of the messages because I get "Bad Gateway".

This is happening more and more regularly and is making the site useless.

I find that clearing my browser cookies will fix it for a while, but then I lose all my cookies!

Can we get this fixed please!

[tsmall07]

You can just delete the cookie for this site individually and it will fix it.

[tig]

You can just delete the cookie for this site individually and it will fix it.

Ok, I just went and found how to delete individual site cookies in Chrome.

http://www.guidingtech.com/6295/delete- ... es-chrome/

However, it's still annoying that this site does this. I'd be happy to help try to fix it. This site is important to me and I'd like to make it better.

[tsmall07]

It's only happened to me once since the initial problem. I think most everyone has been problem free. I don't think it's a problem with the site at this point. For some reason, your computer is trying to access the old server.

[Justin_FL]

Didn't realize anyone was having this problem.

I'll log into the server via ssh and pull up the system logs this weekend (can't do it at work). The Plesk interface is for dummies so I can't review anything.

I probably need to tweak the Apache settings. Rackforce forced us onto a new server earlier this month and just ran a generic migration script. So everything is back to basic Plesk config. Pisses me off, but I usually kept notes of the changes. Would like to dump Plesk since with each update things get broken...

[tig]

It's only happened to me once since the initial problem. I think most everyone has been problem free. I don't think it's a problem with the site at this point. For some reason, your computer is trying to access the old server.

it happens to me at least every few days. It happens on two computers at home and one at work.

[tig]

Didn't realize anyone was having this problem.

I'll log into the server via ssh and pull up the system logs this weekend (can't do it at work). The Plesk interface is for dummies so I can't review anything.

I probably need to tweak the Apache settings. Rackforce forced us onto a new server earlier this month and just ran a generic migration script. So everything is back to basic Plesk config. Pisses me off, but I usually kept notes of the changes. Would like to dump Plesk since with each update things get broken...

I appreciate the work it takes to keep something like this running smoothly. Thanks for doing it!

[Justin_FL]

I haven't checked the logs but found out nginx could be the problem. I turned it off through the web interface, so let me know if there is any improvement. This was something that was not installed on the old server.

[tig]

For the past few weeks I have not had the bad gateway problem.

But today, on two PCs, I encountered it again. I had to delete the mye28.com cookies in Chrome to fix.

Just letting you know...

[demetk]

Same here. It happened on a couple of occasions quite recently. Just letting you know.

[wkohler]

I got it a few times this morning and am perfectly happy with it.

Better than Bad Dell.

[tig]

Or Good Apple?

[Justin_FL]

Will need feedback.

Believe I tweaked the settings in the extra special place Plesk needs them to be changed @ 8:40pm EDT. Can't just change the config files in the normal config file locations, because with every update Plesk completely rewrites them. How charming. Nginx was turned back on and lost my increased buffer size tweaks after the update, so once your cookie got too big it gives you the 502s.

[unt0uchable]

I haven't checked the logs but found out nginx could be the problem. I turned it off through the web interface, so let me know if there is any improvement. This was something that was not installed on the old server.

Got this error today for the first time in a while. The bad gateway also showed this here"nginx" which you mentioned here so I thought I'd let you know. I cleared cookies to see if it would help and it has so far, but still unsure why it cropped up.

Thanks for keeping this site running nearly flawless!!! You guys rock!

[ldsbeaker]

I finally got it this morning.

First World problems...

[Justin_FL]

The site was offline on purpose for about 20-30 minutes, causing the 502 gateway error.

A certain thread had triggered a huge amount of search queries that were overwhelming the SQL server and had pegged the CPU utilization. I suspended the site for a bit to let the action settle down with the search crawler bots and such.

So yeah, please don't bother with searches, i.e. "find all posts" on a certain 40K post user since it will timeout before finishing. Get enough of these going concurrently and the website slows to a crawl, which is what was happening this AM.

[ldsbeaker]

Dang. That is messed up.

Wait, you're saying that I bumped that thread and it killed the website?

[Jeremy]

Wait, you're saying that I bumped that thread and it killed the website?

Pretty much.

I told Skip it was his fault, but if you want to share the blame feel free to take it.

[ldsbeaker]

Wait, you're saying that I bumped that thread and it killed the website?

Pretty much.

I told Skip it was his fault, but if you want to share the blame feel free to take it.

No, thank you.

The thread still pulls up in the Google Search, but no thread exists. Dang. That was not my intent.

The thread is gone, and Chris is gone.. hummm.

[skip535i]

But Skip is still here :-)

[unt0uchable]

Well, this all happened to me again this morning. I got the 502 3 times, 2 of which were while trying to post a reply the a thread. I tried the IP site too and nothing. I'll assume you were working on something again and I just don't have anything better to do at work at 9 am. lol

Thanks again for keeping this place running!!!

[LarryM]

I'm having problems accessing the site too; it's been very slow to respond and today I got the "Bad Gateway" message.

I appreciate all the work involved in maintaining the site but please get this issue solved soon!

[charter21p5]

This is an issue with the site. It is probably an issue with the front end webserver accessing the database.

http://en.wikipedia.org/wiki/List_of_HT ... rver_Error

If you look at the above site it details the error codes. This will need to be resolved by the system administrator.

HTH

[charter21p5]

So the issue is Google crawling the site causing high cpu on the SQL server?

I have noticed this error many times within the last week and it appears to be occuring at peak user times in the evenings.

[Jeremy]

So the issue is Google crawling the site causing high cpu on the SQL server?

No, it isn't Google's fault. The problem manifests when server load is especially high. Site crawlers (from all the various engines) hitting the site certainly increases the load, but it should be able to handle it. We're working on it. It might be a software issue, it might be a hardware issue. It's proving to be a little tough to nail down precisely what the root of the issue is.

[davintosh]

More problems/downtime last night? Most of the evening it wasn't available at all, then just the notice that it was down for maintenance. That was awfully inconvenient.

Not sure if it's a related problem or not, but I haven't been able to get to mye28.com from work at all this week (productivity at work should have improved this week, but...) DNS is able to find the proper IP address (69.10.146.61) and I can ping it, but when I try to pull up the site there's no response from the server. Same issue on multiple computers on the network, so it's not a local issue. Flushed the cache on the DNS, reset the firewall, etc... no joy. I've opened a ticket with our ISP to see if it's just us or other customers of theirs...

[charter21p5]

So the issue is Google crawling the site causing high cpu on the SQL server?

No, it isn't Google's fault. The problem manifests when server load is especially high. Site crawlers (from all the various engines) hitting the site certainly increases the load, but it should be able to handle it. We're working on it. It might be a software issue, it might be a hardware issue. It's proving to be a little tough to nail down precisely what the root of the issue is.

The error message is stating the issue due to the connection or communication issue between the front end web server and backend database. When the cpu is high on your sql device are you able to see who are the top talkers via http? If so put an access-list or firewall rule blocking them and see if the cpu drops to normal or did software updates occur on these servers just prior to this issue happening and now the standard baseline of traffic is causing high cpu? If this is the case then maybe rolling back the updates will fix the issue.


If the cpu is high on your http server then it might not even be related to http and I would check all top talkers via any protocol or for sure check the process list to see which is causing high cpu.

To the other gentlemen's comments about dns it is clear dns resolution is fine since the site resolves to an IP address no issue.

For what it's worth I have never seen bad hardware cause high cpu, however if the hardware is at max capacity then it's time for an upgrade if the traffic is valid and increasing.

[Justin_FL]

I've been using ZBBlock to help the server cope with the load and block spam from common sources as recorded by StopForumSpam. Tons and tons of bad bots have been blocked. Dave, PM me your external IP address and I'll see if it is in the black list. Unfortunately even 1 incident that could be a false flagging will keep the IP blocked for up to a year if SFS is the source of the data.

The site problem is a storage issue with the host, the past 2-3 weeks wherever the sites virtual HD is stored at becomes very slow at certain times. They will switch us to a different storage unit. More RAM would help the server but the disk i/o should be way better than 1MB/s, plus we don't need to swap that much at our current size but the little we do makes the system unstable when the disk doesn't respond with high IO wait averages... CPU usage is almost nothing except for certain SQL queries which is to be expected.

[charter21p5]

The issue I have only seen is the gateway error which resolves itself during non-peak usage times. My experiences have been I cannot get to the site in the evenings EST time zone.

[Justin_FL]

The issue I have only seen is the gateway error which resolves itself during non-peak usage times. My experiences have been I cannot get to the site in the evenings EST time zone.

Yup, that is correct. The reverse proxy is waiting on the web server, which is then waiting on the SQL server for a response. And when the SQL server can't get what it needs from memory, it needs to read the disk, and the disk doesn't respond in a timely fashion = massive slow downs. At a certain point every task on the server will bottleneck on i/o and we go completely offline...

[charter21p5]

Good to know you guys are on top of it!

It's a great site and I always enjoy checking out the new posts, especially with eye candy photos!

Steve

[Jeremy]

Our provider is moving our VM to different hardware tonight. Hopefully it resolves the issue.

So, plan on the site being down tonight for a "few hours" starting some time around 2000 PDT. That's usually when it die anyway, so it'll be business as usual. Hopefully this will be the last time it does that though.

Big thanks to Justin for doing the legwork (fingerwork? electron-work?) to pinpoint what we hope proves to be the root of the current issues.

[charter21p5]

No news is good news right? This thread has died since you guys fixed the site issues. I figured I would let you know the changes made have really improved site access. I have not had 1 error since.