Author Topic: Outage August 4 & 5  (Read 15600 times)

0 Members and 1 Guest are viewing this topic.

Offline John Raabe

  • Administrator
  • Journeyman
  • *****
  • Posts: 5,549
  • Whidbey Island, WA
    • CountryPlans - About Us
Outage August 4 & 5
« on: August 05, 2013, 03:32:08 PM »
Sorry, there was a server crash that was more complex than the hosting service expected. We were out for several hours over two days.

The tables are now all repaired and the forum has been cleaned, compressed and updated.
None of us are as smart as all of us.

Offline OlJarhead

  • Journeyman
  • *****
  • Posts: 3,709
    • Team SEWA
Re: Outage August 4 & 5
« Reply #1 on: August 05, 2013, 06:34:41 PM »
Thanks John, glad it has been fixed!

Offline hpinson

  • Administrator
  • Senior Member
  • *****
  • Posts: 803
Re: Outage August 4 & 5
« Reply #2 on: August 05, 2013, 09:26:04 PM »
Flush them session tables!  :)

Thanks for all the effort you put into it John and Admins.

Offline John Raabe

  • Administrator
  • Journeyman
  • *****
  • Posts: 5,549
  • Whidbey Island, WA
    • CountryPlans - About Us
Re: Outage August 4 & 5
« Reply #3 on: August 07, 2013, 10:33:30 PM »
Hopefully, things will stay solid. Keep those fingers crossed.

I notice we have some banned IP addresses attempting to log-on and being denied access.
None of us are as smart as all of us.

Offline John Raabe

  • Administrator
  • Journeyman
  • *****
  • Posts: 5,549
  • Whidbey Island, WA
    • CountryPlans - About Us
Re: Outage August 4 & 5
« Reply #4 on: August 09, 2013, 05:50:13 PM »
We had a recent reboot of the VPN server. This is number 3 or 4 and something is going on that is crashing the forum and sometimes the whole website. I am well over my head on this and don't even know who to call. Any suggestions or referrals?
None of us are as smart as all of us.

Offline OlJarhead

  • Journeyman
  • *****
  • Posts: 3,709
    • Team SEWA
Re: Outage August 4 & 5
« Reply #5 on: August 09, 2013, 06:24:41 PM »
Wish I could help John but you've got me!  If I have issues like this with my site I contact the hosting company that provides the server. 

I'm guessing it must be some kind of hack or it's a problem on the server itself (corrupt drive or memory perhaps).

Sorry for the trouble!!!

Offline John Raabe

  • Administrator
  • Journeyman
  • *****
  • Posts: 5,549
  • Whidbey Island, WA
    • CountryPlans - About Us
Re: Outage August 4 & 5
« Reply #6 on: August 09, 2013, 06:38:16 PM »
Thanks for your thoughts. Site5 (host) is not really suggesting any causes or solutions.
None of us are as smart as all of us.

Offline OlJarhead

  • Journeyman
  • *****
  • Posts: 3,709
    • Team SEWA
Re: Outage August 4 & 5
« Reply #7 on: August 09, 2013, 06:42:42 PM »
That's not what I'd like to hear from my host!

You could always move the site ;)  I've been looking at moving mine though it's been with GOGVO for 10 years now.  Cost is part of it because I could move to a virtual server with same capabilities I have now and save about 75% of the cost (I pay for a dedicated server and dedicated bandwidth monthly -- $100/mo) but as you probably know, it's a scary idea! lol  My database is SO huge!

Not sure what you see but with 4500+ members and over 10,000 uniques a month generating over 1,000,000 page views I cringe when I start to think of moving the site! lol

Offline John Raabe

  • Administrator
  • Journeyman
  • *****
  • Posts: 5,549
  • Whidbey Island, WA
    • CountryPlans - About Us
Re: Outage August 4 & 5
« Reply #8 on: August 10, 2013, 09:51:52 AM »
I have been getting some help from Dusty at Site 5 - he has a lot of experience with our SMF forum software and suggested a few tweaks. One of which was a list of the highest number of hits from certain IP addresses. Most were Google, Microsoft, etc. but two where suspicious - one in China and the second in Turkey. I don't think we have members in those countries so I have blocked those IP ranges.
None of us are as smart as all of us.

Offline OlJarhead

  • Journeyman
  • *****
  • Posts: 3,709
    • Team SEWA
Re: Outage August 4 & 5
« Reply #9 on: August 10, 2013, 10:28:39 AM »
I have been getting some help from Dusty at Site 5 - he has a lot of experience with our SMF forum software and suggested a few tweaks. One of which was a list of the highest number of hits from certain IP addresses. Most were Google, Microsoft, etc. but two where suspicious - one in China and the second in Turkey. I don't think we have members in those countries so I have blocked those IP ranges.

I do a lot of htaccess ip blocks for that reason.. Do you have a good list?  If not I can forward one to you.

Offline John Raabe

  • Administrator
  • Journeyman
  • *****
  • Posts: 5,549
  • Whidbey Island, WA
    • CountryPlans - About Us
Re: Outage August 4 & 5
« Reply #10 on: August 10, 2013, 10:48:13 AM »
The two blocks I made were done through the host SiteAdmin panel. I see that I do have an old htaccess file in my main directory but it hasn't been used.
None of us are as smart as all of us.

Offline MountainDon

  • Administrator
  • Journeyman
  • *****
  • Posts: 16,530
  • Jemez Mountains, NM; that's pronounced HEY-mess
    • My 15.75 x 30 Jemez Cabin
Re: Outage August 4 & 5
« Reply #11 on: August 10, 2013, 12:33:05 PM »
Using the dashboard or cPanel is a great method for permanently blocking an IP or a range of IP's. Simple to do too. Better than the ban in the SMF software as it reduces server load. There are a few IP ranges we have banned in SMF that might be best blocked at the server level. I can produce a list if wanted.
Just because something has been done and has not failed, doesn’t mean it is good design.

If you didn't have the time or money to do it right in the first place, when will you get the time and money to fix it?

Offline OlJarhead

  • Journeyman
  • *****
  • Posts: 3,709
    • Team SEWA
Re: Outage August 4 & 5
« Reply #12 on: August 10, 2013, 12:50:09 PM »
Using the dashboard or cPanel is a great method for permanently blocking an IP or a range of IP's. Simple to do too. Better than the ban in the SMF software as it reduces server load. There are a few IP ranges we have banned in SMF that might be best blocked at the server level. I can produce a list if wanted.

add that to the htaccess file on root and it's very effective ;)

Offline John Raabe

  • Administrator
  • Journeyman
  • *****
  • Posts: 5,549
  • Whidbey Island, WA
    • CountryPlans - About Us
Re: Outage August 4 & 5
« Reply #13 on: August 10, 2013, 01:19:58 PM »
Thanks to MD and OJ for the messages and the deny list of sites to block. I'm going slowly on this.

I now know how to quickly repair the database if the site crashes again.

Since we won't be able to read a notification from the forum when it is down, please email me at countryplans@gmail.com. That will work even when the whole site is down.
None of us are as smart as all of us.

Offline Don_P

  • Journeyman
  • *****
  • Posts: 4,145
Re: Outage August 4 & 5
« Reply #14 on: August 10, 2013, 05:26:38 PM »
If you see activity from outside the ARIN network and it isn't from someone you know, block the entire range rather than individual IP's, especially from APNIC or RIPE. You will not lose anything worthwhile.

Offline John Raabe

  • Administrator
  • Journeyman
  • *****
  • Posts: 5,549
  • Whidbey Island, WA
    • CountryPlans - About Us
Re: Outage August 4 & 5
« Reply #15 on: August 10, 2013, 11:07:04 PM »
Another forced reboot of the VPN about 2 hours ago.

"We are still seeing the forum at .../public_html/smf/index.php' as the main source of the high usage."

None of us are as smart as all of us.

Offline MountainDon

  • Administrator
  • Journeyman
  • *****
  • Posts: 16,530
  • Jemez Mountains, NM; that's pronounced HEY-mess
    • My 15.75 x 30 Jemez Cabin
Re: Outage August 4 & 5
« Reply #16 on: August 11, 2013, 07:43:08 AM »
I think the logs should indicate what IP's are causing the high use.
Those would likely be spammers, potential spammers, I would think. Or an automated hack attempt. ???

Just because something has been done and has not failed, doesn’t mean it is good design.

If you didn't have the time or money to do it right in the first place, when will you get the time and money to fix it?

Offline hpinson

  • Administrator
  • Senior Member
  • *****
  • Posts: 803
Re: Outage August 4 & 5
« Reply #17 on: August 11, 2013, 03:07:53 PM »
Hi John. Could you clarify a few things (by PM if you wish).

You are running Apache, MySQL, and PHP on a Linux virtual server, or are you running these on a Windows virtual server?

Or is it a shared server, and the crash is taking down other sites as well?

What is failing -- the web server (Apache) or the database (MySQL)?

How much memory is available for dedicated use on the machine? (example: 8GB)

Are you confident that Apache, PHP, and MySQL are each configured to handle what I assume is a very high load? I'm suspecting there may be a good deal of tweaking that could improve performance in these areas.

Somewhere before I saw someone mention corrupt session tables. Those can be related to not enough memory being allocated to the database. Are you having to repair session tables after each crash?

Does your hosting company indicate that you are under some sort of attack which is bringing down the web server or database?

Offline John Raabe

  • Administrator
  • Journeyman
  • *****
  • Posts: 5,549
  • Whidbey Island, WA
    • CountryPlans - About Us
Re: Outage August 4 & 5
« Reply #18 on: August 11, 2013, 03:39:22 PM »
Good questions:

Apache, MySQL, and PHP on a Linux VPN virtual server. I have recently (3 mo) moved over from shared service.
We have been having to repair the MySQL database after a crash. I am getting notices that there was a forced reboot of the server.
I do not know the memory allocation, but I can see there is 43GB of bandwidth and the usage is well under that.

None of us are as smart as all of us.

Offline John Raabe

  • Administrator
  • Journeyman
  • *****
  • Posts: 5,549
  • Whidbey Island, WA
    • CountryPlans - About Us
Re: Outage August 4 & 5
« Reply #19 on: August 11, 2013, 04:21:52 PM »
Here is the latest report from Site 5 (after the forced reboot Aug 10th)

Hello John,

Those all sound good. You could use the .htaccess of that person but should always review it first in case there are other rewrites or code outside of just ip's being blocked.

The following are the top IP's hitting the sites on the 10th.
3172 ./countryplans.com:66.249.72.71
2615 ./countryplans.com:157.55.32.116
1865 ./countryplans.com:157.56.229.246
1517 ./countryplans.com:208.167.230.27
1398 ./countryplans.com:24.160.20.91
1398 ./countryplans.com:157.55.32.147
1045 ./countryplans.com:157.55.32.107
807 ./countryplans.com:157.55.36.54
689 ./countryplans.com:157.55.32.141
610 ./countryplans.com:2.33.163.95

As these are mostly bots as you mentioned, it may help to employ a robots.txt file and control the access of them, as well as adjust the crawl rates at google and bing's web master tools. There are directions on setting this up and configuring at: http://kb.site5.com/bots/how-to-use-the-robots-txt-file/ and http://www.mcanerin.com/en/search-engine/robots-txt.asp

For instance this is what Google and Bing are doing.
=====
66.249.72.71 - - [10/Aug/2013:07:12:24 -0500] "GET /smf/index.php?topic=1528.0 HTTP/1.1" 200 9033 "-" "Mozilla/5.0 (iPhone; U; CPU
iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible;
Googlebot-Mobile/2.1; +http://www.google.com/bot.html)"
66.249.72.71 - - [10/Aug/2013:07:12:32 -0500] "GET /smf/index.php?topic=11571.0 HTTP/1.1" 200 14312 "-" "Mediapartners-Google"
66.249.72.71 - - [10/Aug/2013:07:12:37 -0500] "GET /smf/index.php?topic=8206.5;wap2 HTTP/1.1" 200 739 "-" "Mozilla/5.0 (compatible;
Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.72.71 - - [10/Aug/2013:07:13:20 -0500] "GET /smf/index.php?topic=5025.0 HTTP/1.1" 200 17017 "-" "Mozilla/5.0 (compatible; Go
oglebot/2.1; +http://www.google.com/bot.html)"
66.249.72.71 - - [10/Aug/2013:07:14:03 -0500] "GET /smf/index.php?topic=6811.0;wap2 HTTP/1.1" 200 771 "-" "Mozilla/5.0 (compatible;
Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.72.71 - - [10/Aug/2013:07:14:40 -0500] "GET /smf/index.php?PHPSESSID=43883beec6f32555b5c116a2a5f1e341&topic=12965.25 HTTP/1.
1" 200 17962 "-" "Mediapartners-Google"
66.249.72.71 - - [10/Aug/2013:07:14:46 -0500] "GET /smf/index.php?topic=1418.70;wap2 HTTP/1.1" 200 2592 "-" "Mozilla/5.0 (compatibl
e; Googlebot/2.1; +http://www.google.com/bot.html)"
-----
157.55.32.116 - - [10/Aug/2013:07:19:16 -0500] "GET /smf/index.php?topic=10885.msg144162;topicseen HTTP/1.1" 200 11882 "-" "Mozilla
/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.32.116 - - [10/Aug/2013:07:19:17 -0500] "GET /smf/index.php?topic=10503.msg164176 HTTP/1.1" 200 8941 "-" "Mozilla/5.0 (compa
tible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.32.116 - - [10/Aug/2013:07:19:18 -0500] "GET /smf/index.php?topic=3419.msg131714 HTTP/1.1" 200 14425 "-" "Mozilla/5.0 (compa
tible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.32.116 - - [10/Aug/2013:07:19:19 -0500] "GET /smf/index.php?topic=12280.msg159288 HTTP/1.1" 200 7210 "-" "Mozilla/5.0 (compa
tible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.32.116 - - [10/Aug/2013:07:19:19 -0500] "GET /smf/index.php?topic=6019.msg78341 HTTP/1.1" 200 13040 "-" "Mozilla/5.0 (compat
ible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.32.116 - - [10/Aug/2013:07:19:20 -0500] "GET /smf/index.php?topic=9261.msg123070 HTTP/1.1" 200 17741 "-" "Mozilla/5.0 (compa
tible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.32.116 - - [10/Aug/2013:07:19:21 -0500] "GET /smf/index.php?topic=7766.msg99741 HTTP/1.1" 200 12410 "-" "Mozilla/5.0 (compat
ible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
None of us are as smart as all of us.

Offline hpinson

  • Administrator
  • Senior Member
  • *****
  • Posts: 803
Re: Outage August 4 & 5
« Reply #20 on: August 11, 2013, 07:46:51 PM »
Hi John. I think they are barking up the wrong tree. The visitation numbers above are tiny and should not cause even a modest VPS to blink. Also, you want to encourage Google and Bing indexing (some other bots not) not block like they seem to be suggesting.    One more question -- do you have Google Analytics setup for this site? That can be very helpful for this sort of analysis.  My gut feeling is that something is targeting a database process via PHP (the scripting language that runs SMF) and causing a hang in the session tables -- maybe or maybe not intentionally.

Offline Don_P

  • Journeyman
  • *****
  • Posts: 4,145
Re: Outage August 4 & 5
« Reply #21 on: August 12, 2013, 04:33:50 AM »
Quote
I think they are barking up the wrong tree.
I tend to agree but did notice the last of the 10 listed was 610 hits from one IP in Milan, would it be worthwhile to look a bit further down the list?

Offline muldoon

  • Journeyman
  • *****
  • Posts: 1,618
  • Muldoon, TX
Re: Outage August 4 & 5
« Reply #22 on: August 12, 2013, 08:32:41 AM »
I do not frequent as often as I used to and just saw this today. 

I do not think those numbers are indicative of a denial of service attack that would consume the resources necessary to have forced reboots. 

From below, it sounds as the forum script is using the resources, here is something on 20 some odd things you can do to reduce the footprint of smf.  I wouldn't blindly just start doing it, but it may be useful to the tech to try to find where the problem is. 

http://www.simplemachines.org/community/index.php?topic=293441.0


Offline John Raabe

  • Administrator
  • Journeyman
  • *****
  • Posts: 5,549
  • Whidbey Island, WA
    • CountryPlans - About Us
Re: Outage August 4 & 5
« Reply #23 on: August 12, 2013, 08:53:58 AM »
Thanks to all for those suggestions. I think we are going to look into the idea of some process or script using up the resources. Tuning up SMF and the access for bots and other IPs can come later. It doesn't appear to be an attack or denial of service attempt (according to the host).
None of us are as smart as all of us.

Offline MountainDon

  • Administrator
  • Journeyman
  • *****
  • Posts: 16,530
  • Jemez Mountains, NM; that's pronounced HEY-mess
    • My 15.75 x 30 Jemez Cabin
Re: Outage August 4 & 5
« Reply #24 on: August 12, 2013, 12:17:44 PM »
Google and Bing have always been present on the forum. I have not kept track of the visits over the years but the current number of visits does not seem any greater than in the past. I believe trying to blame those bots for troubles is short sighted.

OTOH, John the month of July shows an large number of page views (in the SMF software, "more stats" list down near the bottom of the basic 'forum boards' page. 1.3M vs 680K a year ago. So something is making the page view counter increment more rapidly than before.  ???

Just because something has been done and has not failed, doesn’t mean it is good design.

If you didn't have the time or money to do it right in the first place, when will you get the time and money to fix it?