CountryPlans Forum

General => Forum News => Topic started by: John Raabe on November 12, 2007, 12:30:03 PM

Title: Compressing or archiving messages
Post by: John Raabe on November 12, 2007, 12:30:03 PM
I have gotten the word from my internet host provider - we need to reduce the number of messages in the forum message folder. Each thread is (at least) one message and we are limited to 2500 messages. Today there are over 3600 threads.

I am researching ways to archive some of these shorter and older posts. I would like to keep as much as possible available for searching as many of the older threads are still active and contain useful information to somebody.  ;)

Any thoughts or suggestions will be appreciated.
Title: Re: Compressing or archiving messages
Post by: glenn kangiser on November 12, 2007, 06:19:49 PM
Is it the forum itself or private messages?

I don't know what to do about this yet -- will try to study up a bit.

Sounds like the problem is at the internet host then - not the Yabb software.  

Title: Re: Compressing or archiving messages
Post by: glenn kangiser on November 12, 2007, 06:27:43 PM
Possibly we could splice like information together but sounds like a lot of work.

I could eliminate non-moving topics in the off topics - archive them into text documents or something if there is something of value there.  

Can you expand on what the problem is --  too many small files? Are large files a problem -- amount of storage? Amount of bandwidth?  Just trying to get the bigger picture so I can work on the problem. :)

Would switching the rest of the forum attached photos and files to Photobucket help?  I can work on that as I find them.  I think you can turn off the ability to directly attach photos and force Photobucket only attachments also.

Just kicking out a few ideas.
Title: Re: Compressing or archiving messages
Post by: MountainDon on November 13, 2007, 12:34:05 AM
Thinking on this. Adding up the numbers under the Topics heading I see that it must be the numbers of topics that is the concern. There's 2182 in the General  Forum.   Plus all the others, looks close to 3600 total.  ??

I have no idea how the ISP has things setup, how he operates, but it would seem that IF the General / General Forum was separated from the rest of the topics there would be less that 2500 Topics in each grouping.

My ISP offers up to 10 sub domains for no extra cost as long as the total disk space and bandwidth does not go over the package limits. The main page could remain the same, but clicking on the General would switch to the other domain, the other limit of 2500 topics.  ??    :-/  

Just thinking out loud.  :)

Archiving? If archived, would the archived items would be under a new/different URL, like the old forum???  That could still be searched but it's inconvenient. But better than nothing, better than losing messages.  

But then If the General / General Forum was spun off from the rest would a search be able to search both locations?   :-?
Title: Re: Compressing or archiving messages
Post by: John Raabe on November 13, 2007, 01:14:01 AM
Hi Guys:

Here's what I understand... each topic is a message file and the file gets bigger as the replies pile up.

I get the impression that my host isn't concerned about the size of the files but the total number.  In that case we could perhaps delete the least visited and discussed messages to get the file list down to the 2500 max the Site5 host wants.

I am confirming this now... If true, our best path may be to delete the smaller and less active threads and keep the more active ones.

Another problem is that FTP programs will only show 2000 files and you can't see the rest of the message lisitings until you have deleted files below this limit. Thus some of the files that need patching cannot be found and corrected until the level in the tub goes down. :P

Here is the biggest file so far in the message directory:
http://www.countryplans.com/cgi-bin/yabb2/YaBB.pl?num=1107141843 (Started in Jan. 2005)
Title: Re: Compressing or archiving messages
Post by: MarkAndDebbie on November 13, 2007, 08:00:39 AM
I'm curious about the 2000 file limit for your ftp program. Do you know if it is a server imposed restriction? There is not anything in the FTP spec that would limit the number of files show.

1. What ftp programs have you tried?
2. Is it 2000 total files or 2000 files per directory? If per directory would making more divisions (forums etc) help?
Title: Re: Compressing or archiving messages
Post by: glenn kangiser on November 13, 2007, 09:55:43 AM
Something seems wrong here as the Dodge Truck forum is tons bigger with no problem.  They also have a ton more forums and topics and are more active than us.  

At this moment, DTR

QuoteCurrently Active Users: 1191 (682 members and 509 guests)  Threads: 152,258, Posts: 1,599,532, Members: 43,395

http://www.dieseltruckresource.com/dev/index.php
Title: Re: Compressing or archiving messages
Post by: John Raabe on November 13, 2007, 10:53:49 AM
The Dodge truck forum is running on vBulletin - http://www.vbulletin.com/
(https://i56.photobucket.com/albums/g166/jraabe/snap.jpg)

I will explore how the migration might work and if it could help us out of this bind.

The 2000 file limit of FTP appears to be built-in to WS_FTP and Cute FTP - those are two I checked.

(https://i56.photobucket.com/albums/g166/jraabe/snap2.jpg)

If anyone knows of a program that does not have this limit I would appreciate it.
Title: Re: Compressing or archiving messages
Post by: glenn kangiser on November 13, 2007, 11:01:36 AM
Possibly we could get in contact with the DTR guys for a clue.  Their forum is a friendly forum like ours so they would probably help with info.  Maybe I can find someone to message.
Title: Re: Compressing or archiving messages
Post by: John Raabe on November 13, 2007, 11:27:05 AM
I'm waiting to make a few calls here... Another forum solution we should look at is Invision - http://www.invisionpower.com/community/board/convert.html

It looks like a conversion from Yabb is possible...

One of the newer features of both of these upgrades is built-in blogs. People building a house could setup an internal blog for storing photos and updates and then link to it in forum posts. Such treads would be less likely to get lost.
Title: Re: Compressing or archiving messages
Post by: glenn kangiser on November 13, 2007, 11:29:26 AM
Sounds good.  I messaged Lary Ellis at DTR - don't know when I'll get a reply.
Title: Re: Compressing or archiving messages
Post by: glenn kangiser on November 13, 2007, 11:33:01 AM
Got it already -- great guys there.  I sent him a thank you.

QuoteJack Thorpe  Jack Thorpe is online now
Founder / Webmaster
                         
Join Date: May 2002
Posts: 3,140
Yabb
Glenn,

I can tell you that YaBB has some problems. I used to used it before I moved over to vBulletin a few years ago. Once I got to about 3000 members, it started taxing my hardware really bad, and it would kill my servers. It's just too taxing on any large forum. I'm not sure if they've improved it now or not since I haven't used it in about 4 years.

I would recommend making the move to vb. It's kind of a pain at first but it's worth it. If you're using the cgi version of Yabb, then it'll be even worse. You might have to make the move to simple machines, then make the move to vb later.
__________________
2007 6.7l Dually
Quad Cab
Title: Re: Compressing or archiving messages
Post by: glenn kangiser on November 13, 2007, 11:39:23 AM
Apparently this is what Jack was talking about. Maybe an easier way to do the conversion.

http://www.simplemachines.org/
Title: Re: Compressing or archiving messages
Post by: MarkAndDebbie on November 13, 2007, 12:59:24 PM
Have you tried windows explorer? You can put the ftp://... in the address bar. If the address bar is not on - you can turn it on under view/toolbars/address bar. I do not think there is a number of files limit.
Title: Re: Compressing or archiving messages
Post by: John Raabe on November 13, 2007, 01:26:05 PM
MarkandDebbie:

Neat trick with Explorer... I hadn't seen that before. (tips: http://www.developingwebs.net/publishing/ie.php)

Unfortunately, when I get into the yabb2 message directory the same 1134603564.txt file is the last one on the sorted list. That is the same limit of the FTP programs. Yet, this thread is numbered 1194888603.txt.
Title: Re: Compressing or archiving messages
Post by: MarkAndDebbie on November 13, 2007, 02:07:50 PM
So I checked on explorer - it has no limit. It seems it can be set as a limit on the server to avoid denial of service attacks. You could ask your host to increase the limit.
Title: Re: Compressing or archiving messages
Post by: John Raabe on November 13, 2007, 03:35:58 PM
Ya, this seems to be cast in place on the host server side. They are no going to change that file level... tried that.

Now I am looking at the process of migrating the Yabb forum to either vbulletin or invision power board. Sounds nasty!!
Title: Re: Compressing or archiving messages
Post by: glenn kangiser on November 13, 2007, 03:43:09 PM
Nasty has it's places but moving the board isn't one of them. ::)
Title: Re: Compressing or archiving messages
Post by: John Raabe on November 13, 2007, 07:14:41 PM
I have spent the day looking into transitioning to vbulletin or Invisions Power Board... but they both seem complex, especially with the conversion.

Then, I looked at the [highlight]Simple Machine Forum[/highlight] where there is a thread on conversions from Yabb:
http://www.simplemachines.org/community/index.php?board=135.0

While not exactly THAT simple it may be a better step than either of the above forum options.

SMF  (http://www.simplemachines.org/)is a nice looking forum (much like YABB) and... it's also free.

   "SMF is written in the popular language PHP and uses a MySQL database. It is designed to provide you with all the features you need from a bulletin board while having an absolute minimal impact on the resources of the server. SMF is the next generation of forum software - and best of all it is and will always remain completely free!"

A bit later.... I've registered at SMF and done a sample post. It seems to do everything the Yabb forum does (without the file bloat).

http://www.simplemachines.org/community/index.php?topic=205700.msg1308192#msg1308192.
Title: Re: Compressing or archiving messages
Post by: MarkAndDebbie on November 13, 2007, 08:07:44 PM
If money was no object - you would go with http://communityserver.org/. Probably the hosted solution. They have a free version - but I think you'd have to restructure the "company" to use it.

Barring that SMF looked promising.
Title: Re: Compressing or archiving messages
Post by: glenn kangiser on November 13, 2007, 08:59:53 PM
I assume Jack from DTR knows something of the problems of switching to VB from YaBB so suggested Simplemachines because of the problems we may encounter with the cgi version of YaBB, and looking in the about section of the SMF I found that the people there were formerly from YaBB

http://www.simplemachines.org/about/
Title: Re: Compressing or archiving messages
Post by: glenn kangiser on November 14, 2007, 12:20:16 AM
The SMF specs are impressive.

Quote1,270,967 Posts in 166,776 Topics by 116,952 Members
Title: Re: Compressing or archiving messages
Post by: John Raabe on November 14, 2007, 07:47:32 PM
Hey, thanks to Don and Glenn for helping to check this forum stuff out.

The SMF foundation and history seems like a good match for this board.

So, what do you think?

Shall we take a deep breath and try and migrate this sucker???

We can keep this thread open and see if we can brainstorm any problems that come up... ([highlight]PROBLEMS?[/highlight], we don't need no stinking PROBLEMS!)

John
Title: Re: Compressing or archiving messages
Post by: MountainDon on November 14, 2007, 08:07:43 PM
You are welcome. Just having fun messing with it.  :)

You are the boss/owner John and the one who will bear the brunt of the changeover labor. But SMF would look like a good choice.



On a similar but other vein, I'm webmaster for our local dance club and have been considering a member forum for things for sale, trade, questions, and what not. I D/L'd the SMF files and was going to start playing with it, but ran into a problem. Add to that, my ISP is migrating me from one platform to another... I may just wait till the migration gets completed.
Title: Re: Compressing or archiving messages
Post by: glenn kangiser on November 14, 2007, 10:06:15 PM
Like you need another project, John, but it seems to be the only way to handle it without losing more of our forum at random times.

I'm here for whatever I can do to help. :)
Title: Re: Compressing or archiving messages
Post by: MountainDon on November 14, 2007, 10:13:29 PM
QuoteI'm here for whatever I can do to help. :)
Dittos
Title: Re: Compressing or archiving messages
Post by: John Raabe on November 16, 2007, 05:46:20 PM
I have been able to get the basic SMF installed.

(https://i56.photobucket.com/albums/g166/jraabe/smf.jpg)

Now to see about the conversion of all the members and messages... :-*

PS - It looks like SMF is a good middle stage even if we end up wanting the feature set of vbulletin. This gets us into the MySQL database world which will be easier to convert to another similar forum.
Title: Re: Compressing or archiving messages
Post by: glenn kangiser on November 16, 2007, 05:49:33 PM
I figured you were busy with that.  We'll be here waiting anxiously to see what happens. :)
Title: Re: Compressing or archiving messages
Post by: MountainDon on November 16, 2007, 06:06:54 PM
Way cool John.   :)
Title: Re: Compressing or archiving messages
Post by: John Raabe on November 18, 2007, 01:01:29 PM
Here is a link to the SMF countryplans forum.

Feel free to play with it. It's on our CP site. However, posts will be lost when we convert the existing yabb forum and overwrite the current files and messages.

http://countryplans.com/smf/index.php?board=1.0
Title: Re: Compressing or archiving messages
Post by: glenn kangiser on November 19, 2007, 11:55:08 AM
This is exciting ---Looks pretty good so far.