CountryPlans Design/Build Forum

General => Forum News => Topic started by: John Raabe on August 05, 2013, 01:32:08 PM

Title: Outage August 4 & 5
Post by: John Raabe on August 05, 2013, 01:32:08 PM
Sorry, there was a server crash that was more complex than the hosting service expected. We were out for several hours over two days.

The tables are now all repaired and the forum has been cleaned, compressed and updated.
Title: Re: Outage August 4 & 5
Post by: OlJarhead on August 05, 2013, 04:34:41 PM
Thanks John, glad it has been fixed!
Title: Re: Outage August 4 & 5
Post by: hpinson on August 05, 2013, 07:26:04 PM
Flush them session tables!  :)

Thanks for all the effort you put into it John and Admins.
Title: Re: Outage August 4 & 5
Post by: John Raabe on August 07, 2013, 08:33:30 PM
Hopefully, things will stay solid. Keep those fingers crossed.

I notice we have some banned IP addresses attempting to log-on and being denied access.
Title: Re: Outage August 4 & 5
Post by: John Raabe on August 09, 2013, 03:50:13 PM
We had a recent reboot of the VPN server. This is number 3 or 4 and something is going on that is crashing the forum and sometimes the whole website. I am well over my head on this and don't even know who to call. Any suggestions or referrals?
Title: Re: Outage August 4 & 5
Post by: OlJarhead on August 09, 2013, 04:24:41 PM
Wish I could help John but you've got me!  If I have issues like this with my site I contact the hosting company that provides the server. 

I'm guessing it must be some kind of hack or it's a problem on the server itself (corrupt drive or memory perhaps).

Sorry for the trouble!!!
Title: Re: Outage August 4 & 5
Post by: John Raabe on August 09, 2013, 04:38:16 PM
Thanks for your thoughts. Site5 (host) is not really suggesting any causes or solutions.
Title: Re: Outage August 4 & 5
Post by: OlJarhead on August 09, 2013, 04:42:42 PM
That's not what I'd like to hear from my host!

You could always move the site ;)  I've been looking at moving mine though it's been with GOGVO for 10 years now.  Cost is part of it because I could move to a virtual server with same capabilities I have now and save about 75% of the cost (I pay for a dedicated server and dedicated bandwidth monthly -- $100/mo) but as you probably know, it's a scary idea! lol  My database is SO huge!

Not sure what you see but with 4500+ members and over 10,000 uniques a month generating over 1,000,000 page views I cringe when I start to think of moving the site! lol
Title: Re: Outage August 4 & 5
Post by: John Raabe on August 10, 2013, 07:51:52 AM
I have been getting some help from Dusty at Site 5 - he has a lot of experience with our SMF forum software and suggested a few tweaks. One of which was a list of the highest number of hits from certain IP addresses. Most were Google, Microsoft, etc. but two where suspicious - one in China and the second in Turkey. I don't think we have members in those countries so I have blocked those IP ranges.
Title: Re: Outage August 4 & 5
Post by: OlJarhead on August 10, 2013, 08:28:39 AM
I have been getting some help from Dusty at Site 5 - he has a lot of experience with our SMF forum software and suggested a few tweaks. One of which was a list of the highest number of hits from certain IP addresses. Most were Google, Microsoft, etc. but two where suspicious - one in China and the second in Turkey. I don't think we have members in those countries so I have blocked those IP ranges.

I do a lot of htaccess ip blocks for that reason.. Do you have a good list?  If not I can forward one to you.
Title: Re: Outage August 4 & 5
Post by: John Raabe on August 10, 2013, 08:48:13 AM
The two blocks I made were done through the host SiteAdmin panel. I see that I do have an old htaccess file in my main directory but it hasn't been used.
Title: Re: Outage August 4 & 5
Post by: MountainDon on August 10, 2013, 10:33:05 AM
Using the dashboard or cPanel is a great method for permanently blocking an IP or a range of IP's. Simple to do too. Better than the ban in the SMF software as it reduces server load. There are a few IP ranges we have banned in SMF that might be best blocked at the server level. I can produce a list if wanted.
Title: Re: Outage August 4 & 5
Post by: OlJarhead on August 10, 2013, 10:50:09 AM
Using the dashboard or cPanel is a great method for permanently blocking an IP or a range of IP's. Simple to do too. Better than the ban in the SMF software as it reduces server load. There are a few IP ranges we have banned in SMF that might be best blocked at the server level. I can produce a list if wanted.

add that to the htaccess file on root and it's very effective ;)
Title: Re: Outage August 4 & 5
Post by: John Raabe on August 10, 2013, 11:19:58 AM
Thanks to MD and OJ for the messages and the deny list of sites to block. I'm going slowly on this.

I now know how to quickly repair the database if the site crashes again.

Since we won't be able to read a notification from the forum when it is down, please email me at countryplans@gmail.com. That will work even when the whole site is down.
Title: Re: Outage August 4 & 5
Post by: Don_P on August 10, 2013, 03:26:38 PM
If you see activity from outside the ARIN network and it isn't from someone you know, block the entire range rather than individual IP's, especially from APNIC or RIPE. You will not lose anything worthwhile.
Title: Re: Outage August 4 & 5
Post by: John Raabe on August 10, 2013, 09:07:04 PM
Another forced reboot of the VPN about 2 hours ago.

"We are still seeing the forum at .../public_html/smf/index.php' as the main source of the high usage."

Title: Re: Outage August 4 & 5
Post by: MountainDon on August 11, 2013, 05:43:08 AM
I think the logs should indicate what IP's are causing the high use.
Those would likely be spammers, potential spammers, I would think. Or an automated hack attempt. ???

Title: Re: Outage August 4 & 5
Post by: hpinson on August 11, 2013, 01:07:53 PM
Hi John. Could you clarify a few things (by PM if you wish).

You are running Apache, MySQL, and PHP on a Linux virtual server, or are you running these on a Windows virtual server?

Or is it a shared server, and the crash is taking down other sites as well?

What is failing -- the web server (Apache) or the database (MySQL)?

How much memory is available for dedicated use on the machine? (example: 8GB)

Are you confident that Apache, PHP, and MySQL are each configured to handle what I assume is a very high load? I'm suspecting there may be a good deal of tweaking that could improve performance in these areas.

Somewhere before I saw someone mention corrupt session tables. Those can be related to not enough memory being allocated to the database. Are you having to repair session tables after each crash?

Does your hosting company indicate that you are under some sort of attack which is bringing down the web server or database?
Title: Re: Outage August 4 & 5
Post by: John Raabe on August 11, 2013, 01:39:22 PM
Good questions:

Apache, MySQL, and PHP on a Linux VPN virtual server. I have recently (3 mo) moved over from shared service.
We have been having to repair the MySQL database after a crash. I am getting notices that there was a forced reboot of the server.
I do not know the memory allocation, but I can see there is 43GB of bandwidth and the usage is well under that.

Title: Re: Outage August 4 & 5
Post by: John Raabe on August 11, 2013, 02:21:52 PM
Here is the latest report from Site 5 (after the forced reboot Aug 10th)

Hello John,

Those all sound good. You could use the .htaccess of that person but should always review it first in case there are other rewrites or code outside of just ip's being blocked.

The following are the top IP's hitting the sites on the 10th.
3172 ./countryplans.com:66.249.72.71
2615 ./countryplans.com:157.55.32.116
1865 ./countryplans.com:157.56.229.246
1517 ./countryplans.com:208.167.230.27
1398 ./countryplans.com:24.160.20.91
1398 ./countryplans.com:157.55.32.147
1045 ./countryplans.com:157.55.32.107
807 ./countryplans.com:157.55.36.54
689 ./countryplans.com:157.55.32.141
610 ./countryplans.com:2.33.163.95

As these are mostly bots as you mentioned, it may help to employ a robots.txt file and control the access of them, as well as adjust the crawl rates at google and bing's web master tools. There are directions on setting this up and configuring at: http://kb.site5.com/bots/how-to-use-the-robots-txt-file/ and http://www.mcanerin.com/en/search-engine/robots-txt.asp

For instance this is what Google and Bing are doing.
=====
66.249.72.71 - - [10/Aug/2013:07:12:24 -0500] "GET /smf/index.php?topic=1528.0 HTTP/1.1" 200 9033 "-" "Mozilla/5.0 (iPhone; U; CPU
iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible;
Googlebot-Mobile/2.1; +http://www.google.com/bot.html)"
66.249.72.71 - - [10/Aug/2013:07:12:32 -0500] "GET /smf/index.php?topic=11571.0 HTTP/1.1" 200 14312 "-" "Mediapartners-Google"
66.249.72.71 - - [10/Aug/2013:07:12:37 -0500] "GET /smf/index.php?topic=8206.5;wap2 HTTP/1.1" 200 739 "-" "Mozilla/5.0 (compatible;
Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.72.71 - - [10/Aug/2013:07:13:20 -0500] "GET /smf/index.php?topic=5025.0 HTTP/1.1" 200 17017 "-" "Mozilla/5.0 (compatible; Go
oglebot/2.1; +http://www.google.com/bot.html)"
66.249.72.71 - - [10/Aug/2013:07:14:03 -0500] "GET /smf/index.php?topic=6811.0;wap2 HTTP/1.1" 200 771 "-" "Mozilla/5.0 (compatible;
Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.72.71 - - [10/Aug/2013:07:14:40 -0500] "GET /smf/index.php?PHPSESSID=43883beec6f32555b5c116a2a5f1e341&topic=12965.25 HTTP/1.
1" 200 17962 "-" "Mediapartners-Google"
66.249.72.71 - - [10/Aug/2013:07:14:46 -0500] "GET /smf/index.php?topic=1418.70;wap2 HTTP/1.1" 200 2592 "-" "Mozilla/5.0 (compatibl
e; Googlebot/2.1; +http://www.google.com/bot.html)"
-----
157.55.32.116 - - [10/Aug/2013:07:19:16 -0500] "GET /smf/index.php?topic=10885.msg144162;topicseen HTTP/1.1" 200 11882 "-" "Mozilla
/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.32.116 - - [10/Aug/2013:07:19:17 -0500] "GET /smf/index.php?topic=10503.msg164176 HTTP/1.1" 200 8941 "-" "Mozilla/5.0 (compa
tible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.32.116 - - [10/Aug/2013:07:19:18 -0500] "GET /smf/index.php?topic=3419.msg131714 HTTP/1.1" 200 14425 "-" "Mozilla/5.0 (compa
tible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.32.116 - - [10/Aug/2013:07:19:19 -0500] "GET /smf/index.php?topic=12280.msg159288 HTTP/1.1" 200 7210 "-" "Mozilla/5.0 (compa
tible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.32.116 - - [10/Aug/2013:07:19:19 -0500] "GET /smf/index.php?topic=6019.msg78341 HTTP/1.1" 200 13040 "-" "Mozilla/5.0 (compat
ible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.32.116 - - [10/Aug/2013:07:19:20 -0500] "GET /smf/index.php?topic=9261.msg123070 HTTP/1.1" 200 17741 "-" "Mozilla/5.0 (compa
tible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.32.116 - - [10/Aug/2013:07:19:21 -0500] "GET /smf/index.php?topic=7766.msg99741 HTTP/1.1" 200 12410 "-" "Mozilla/5.0 (compat
ible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
Title: Re: Outage August 4 & 5
Post by: hpinson on August 11, 2013, 05:46:51 PM
Hi John. I think they are barking up the wrong tree. The visitation numbers above are tiny and should not cause even a modest VPS to blink. Also, you want to encourage Google and Bing indexing (some other bots not) not block like they seem to be suggesting.    One more question -- do you have Google Analytics setup for this site? That can be very helpful for this sort of analysis.  My gut feeling is that something is targeting a database process via PHP (the scripting language that runs SMF) and causing a hang in the session tables -- maybe or maybe not intentionally.
Title: Re: Outage August 4 & 5
Post by: Don_P on August 12, 2013, 02:33:50 AM
Quote
I think they are barking up the wrong tree.
I tend to agree but did notice the last of the 10 listed was 610 hits from one IP in Milan, would it be worthwhile to look a bit further down the list?
Title: Re: Outage August 4 & 5
Post by: muldoon on August 12, 2013, 06:32:41 AM
I do not frequent as often as I used to and just saw this today. 

I do not think those numbers are indicative of a denial of service attack that would consume the resources necessary to have forced reboots. 

From below, it sounds as the forum script is using the resources, here is something on 20 some odd things you can do to reduce the footprint of smf.  I wouldn't blindly just start doing it, but it may be useful to the tech to try to find where the problem is. 

http://www.simplemachines.org/community/index.php?topic=293441.0

Title: Re: Outage August 4 & 5
Post by: John Raabe on August 12, 2013, 06:53:58 AM
Thanks to all for those suggestions. I think we are going to look into the idea of some process or script using up the resources. Tuning up SMF and the access for bots and other IPs can come later. It doesn't appear to be an attack or denial of service attempt (according to the host).
Title: Re: Outage August 4 & 5
Post by: MountainDon on August 12, 2013, 10:17:44 AM
Google and Bing have always been present on the forum. I have not kept track of the visits over the years but the current number of visits does not seem any greater than in the past. I believe trying to blame those bots for troubles is short sighted.

OTOH, John the month of July shows an large number of page views (in the SMF software, "more stats" list down near the bottom of the basic 'forum boards' page. 1.3M vs 680K a year ago. So something is making the page view counter increment more rapidly than before.  ???

Title: Re: Outage August 4 & 5
Post by: hpinson on August 12, 2013, 10:27:14 AM
Don, is there a way in the SMF admin panel to store session cookies on the file system rather than in the database directly?  Also, how many minutes are sessions timeout set for in the SMF admin panel?  Try 30 minutes (will log users off after 30 and free up some resources).

Title: Re: Outage August 4 & 5
Post by: MountainDon on August 12, 2013, 11:49:50 AM
Quote
how many minutes are sessions timeout set for in the SMF admin panel?  Try 30 minutes (will log users off after 30 and free up some resources).

was 90, now 30

Quote
is there a way in the SMF admin panel to store session cookies on the file system rather than in the database
???    there is this in the 'server settings" admin area....
"Use database driven sessions"   The info pop up has this to say...
"This option makes use of the database for session storage - it is best for load balanced servers, but helps with all timeout issues and can make the forum faster."

This choice was selected to be active.

I have changed it to be non-selected, non active.



There is also the setting...
"Seconds before an unused session timeout"
with the info popup...
"This is the number of seconds for sessions to last after they haven't been accessed. If a session is not accessed for too long, it is said to have "timed out". Anything higher than 2400 is recommended."

This was set at 4000. It is now set at 2500.



John, those are all in   
Administration Center »
Server Settings »
Cookies and Sessions
Title: Re: Outage August 4 & 5
Post by: hpinson on August 12, 2013, 01:05:41 PM
Exactly -- now lets see if the crashing problem goes away, and if forum performance suffers (as far as I can tell it is still quite good).

Only downside is that people who leave their browsers logged in will get logged out after 30 minutes.  If there are complaints consider pushing to 40 - 50 - or even 60 minutes, but the shorter the more available resources.



Title: Re: Outage August 4 & 5
Post by: John Raabe on August 12, 2013, 01:57:02 PM
Those look like good changes. 30 mins of unattended time seems plenty. We are all used to restarting open windows that time out. Thanks Don and HP!

There was a surge in page views in July. Up from an Ave 21,938/day for 2012 to 42,500/day in 2013. For Aug '13 we are averaging 36,660/day so far.
Title: Re: Outage August 4 & 5
Post by: John Raabe on August 12, 2013, 02:57:25 PM
Don:

Here is something I found in an SMF support forum article on the best performance tweaks

Move your uploaded avatars directory

In the administration panel, go to Attachments and Avatars, and then go to Avatar Settings.

For Upload avatars to... select: Specific directory... For Upload directory: put in the raw, publicly accessible server directory you want them added to ie /var/www/yoursite/docs/forums/avs

So assuming /docs was your webroot, you would put /forums/avs for your upload URL. Don't do what I did and make it a subdirectory of your main avatars directory, SMF still handles it like an attachment directory for some reason so if someone uploads a .png avatar, someone else decides to use it and the first person replaces it with a .jpg avatar, the first will be deleted.

The benefit of this borders on the ridiculous. Every single avatar load is another php call, which loads a session, the user's settings and permissions, the board and its theme. In addition, two additional calls to the database are made to load the avatar itself, and to increment the view counter. So not only are you reducing the processor load on your server, but you are also reducing your database load. If five different avatars are displayed on a page, that's five additional php calls and dozens of database queries getting made.

After you do this access index.php?action=manageattachments;sa=moveAvatars to use the hidden feature to move your avatars over.


This setting is in

    CountryPlans Design/Build Forum »
    Administration Center »
    Attachments and Avatars »
    Avatar Settings

It looks like we have an Avatar directory under the SMF dir. Should we build another folder outside the SMF forum - perhaps avatars in the public section?

PS - Looking at the error logs on Site 5 a lot of our errors are attempting to load avatar images (ie: smf/avatars/dilbert.gif). Those errors are coming in at 5 to 10 per minute. They are mostly image requests. Some are for an "apple-touch-icon.png" which lots of sites apparently get as an error message from page requests from iphones.

PPS - Looking at the file manager the above avatar directory only has old preloaded images from 2004, not the images most members are using as avatars.

Title: Re: Outage August 4 & 5
Post by: MountainDon on August 12, 2013, 03:21:17 PM
I take that to mean place an  avatar  folder in the   smf    folder. Then direct the uploaded user avaters to that directory.

???  Does doing that have any effect on already uploaded user avatars?

It is also possible to do a maintenance run on removing avatars for members not active in over "nnn"  number of days.   Here, admins only have access (http://countryplans.com/smf/index.php?action=admin;area=manageattachments;sa=maintenance;dd752ba0d157=7ad951a94cdd11a08cb80c659d424e03)
Title: Re: Outage August 4 & 5
Post by: John Raabe on August 12, 2013, 03:59:45 PM
That looks like we only have 30 avatars, that may be the old default ones from 2004. Where are the rest?  ???
Title: Re: Outage August 4 & 5
Post by: MountainDon on August 12, 2013, 04:36:31 PM
For several years now users have only been able to choose from the SMF built in avatars or to link to a remotely stored (ie. photobucket) avatar. No uploads of an avatar has been allowed. Same as no uploads of images or other files.

I wonder what causes most server load? Avatars on our server someplace or the call to bring an avatar from the remote host to the viewers own computer?  ???
Title: Re: Outage August 4 & 5
Post by: hpinson on August 12, 2013, 08:40:54 PM
It is a URI text string call from the database. The browser renders.
Title: Re: Outage August 4 & 5
Post by: John Raabe on August 12, 2013, 09:53:58 PM
Yes, I realize now that the avatar load on our forum is probably quite low since few people use the default images. Most people have avatars that are just a short piece of code.

Still, Most of the errors in the Site 5 log are image requests that don't complete.
Title: Re: Outage August 4 & 5
Post by: hpinson on August 13, 2013, 07:14:04 AM
It's the right approach though for sure. As you work through the error log, squash each error (for example bad paths to Avatars ).  Each error does take its toll.

From your message above it looks like maybe an incomplete or bad path to the local avatars entered in the SMF admin panel.

smf/avatars/dilbert.gif

A complete path would be from the root:  For example.

/path/to/smf/avatars/
Title: Re: Outage August 4 & 5
Post by: John Raabe on August 13, 2013, 07:54:59 AM
Here is a summary of 1 minute of the Site 5 error log (15 mins ago)
9 errors
4 (403.shtml errors) to a forum post that was started in 2012 (this resolves fine when you copy and paste the URL address.)
4 errors calling an image, apple-touch-icon.png or icon-precomposed.png. These are expected to be the main public_html directory and don't exist.
1 error to smf/index.php (which exists)

Further down (2 mins earlier) are a batch of errors for gif files in the smf/avatars directory that don't exist.

Does that tell us anything?
Title: Re: Outage August 4 & 5
Post by: hpinson on August 13, 2013, 08:15:01 AM
What is the URL address?

A 403 errors means that some web reference to another server is not resolving -- probably content (an image?) that existed once and is now gone - and somebody looked at a forum page with this problem.  The target web server is saying I can't find what you are looking for is "Forbidden" or in other words can't be found. I know there is quite a bit of this missing image content in older posts, so would accept this error as a normal occurance and probably ignore.

One hack to fix the not-resolving avatar images is to place 1 pixel .png, .jpg, or .gif images with the corresponding names in the referenced directory.  My gut reaction os that you can safely ignore these, especially as the error count is so low.

Not sure what the error to the index.php refers to. Would have to look at the actual error log and not the control panels version of it -- or perhaps you are seeing the actual log in the CP, but error reporting is not set to an informative enough level.
Templates: 3: Ads (default), Portal (default), Printpage (default).
Sub templates: 4: init, print_above, main, print_below.
Language files: 3: SPortal.english (default), index+Modifications.english (default), Ads.english (default).
Style sheets: 1: portal (default).
Files included: 31 - 896KB. (show)
Cache hits: 8: 0.00156s for 35,102 bytes (show)
Queries used: 16.

[Show Queries]