Author Topic: Compressing or archiving messages  (Read 11424 times)

0 Members and 1 Guest are viewing this topic.

Offline John Raabe

  • Administrator
  • Journeyman
  • *****
  • Posts: 5,550
  • Whidbey Island, WA
    • CountryPlans - About Us
Compressing or archiving messages
« on: November 12, 2007, 08:30:03 AM »
I have gotten the word from my internet host provider - we need to reduce the number of messages in the forum message folder. Each thread is (at least) one message and we are limited to 2500 messages. Today there are over 3600 threads.

I am researching ways to archive some of these shorter and older posts. I would like to keep as much as possible available for searching as many of the older threads are still active and contain useful information to somebody.  ;)

Any thoughts or suggestions will be appreciated.
None of us are as smart as all of us.

Offline glenn kangiser

  • The Troglodyte
  • Administrator
  • Journeyman
  • *****
  • Posts: 24,906
  • Central California Sierras- Home of Yosemite NP
    • Underground Cabin
Re: Compressing or archiving messages
« Reply #1 on: November 12, 2007, 02:19:49 PM »
Is it the forum itself or private messages?

I don't know what to do about this yet -- will try to study up a bit.

Sounds like the problem is at the internet host then - not the Yabb software.  

"Always work from the general to the specific." J. Raabe

Glenn's Underground Cabin  http://countryplans.com/smf/index.php?topic=151.0

Please put your area in your sig line so we can assist with location specific answers.

Offline glenn kangiser

  • The Troglodyte
  • Administrator
  • Journeyman
  • *****
  • Posts: 24,906
  • Central California Sierras- Home of Yosemite NP
    • Underground Cabin
Re: Compressing or archiving messages
« Reply #2 on: November 12, 2007, 02:27:43 PM »
Possibly we could splice like information together but sounds like a lot of work.

I could eliminate non-moving topics in the off topics - archive them into text documents or something if there is something of value there.  

Can you expand on what the problem is --  too many small files? Are large files a problem -- amount of storage? Amount of bandwidth?  Just trying to get the bigger picture so I can work on the problem. :)

Would switching the rest of the forum attached photos and files to Photobucket help?  I can work on that as I find them.  I think you can turn off the ability to directly attach photos and force Photobucket only attachments also.

Just kicking out a few ideas.
« Last Edit: November 12, 2007, 02:30:33 PM by glenn-k »
"Always work from the general to the specific." J. Raabe

Glenn's Underground Cabin  http://countryplans.com/smf/index.php?topic=151.0

Please put your area in your sig line so we can assist with location specific answers.

Offline MountainDon

  • Administrator
  • Journeyman
  • *****
  • Posts: 16,680
  • Jemez Mountains, NM; that's pronounced HEY-mess
    • My 15.75 x 30 Jemez Cabin
Re: Compressing or archiving messages
« Reply #3 on: November 12, 2007, 08:34:05 PM »
Thinking on this. Adding up the numbers under the Topics heading I see that it must be the numbers of topics that is the concern. There's 2182 in the General  Forum.   Plus all the others, looks close to 3600 total.  ??

I have no idea how the ISP has things setup, how he operates, but it would seem that IF the General / General Forum was separated from the rest of the topics there would be less that 2500 Topics in each grouping.

My ISP offers up to 10 sub domains for no extra cost as long as the total disk space and bandwidth does not go over the package limits. The main page could remain the same, but clicking on the General would switch to the other domain, the other limit of 2500 topics.  ??    :-/  
 
Just thinking out loud.  :)

Archiving? If archived, would the archived items would be under a new/different URL, like the old forum???  That could still be searched but it's inconvenient. But better than nothing, better than losing messages.  

But then If the General / General Forum was spun off from the rest would a search be able to search both locations?   :-?
« Last Edit: November 12, 2007, 09:08:33 PM by MountainDon »
Just because something has been done and has not failed, doesn’t mean it is good design.

Offline John Raabe

  • Administrator
  • Journeyman
  • *****
  • Posts: 5,550
  • Whidbey Island, WA
    • CountryPlans - About Us
Re: Compressing or archiving messages
« Reply #4 on: November 12, 2007, 09:14:01 PM »
Hi Guys:

Here's what I understand... each topic is a message file and the file gets bigger as the replies pile up.

I get the impression that my host isn't concerned about the size of the files but the total number.  In that case we could perhaps delete the least visited and discussed messages to get the file list down to the 2500 max the Site5 host wants.

I am confirming this now... If true, our best path may be to delete the smaller and less active threads and keep the more active ones.

Another problem is that FTP programs will only show 2000 files and you can't see the rest of the message lisitings until you have deleted files below this limit. Thus some of the files that need patching cannot be found and corrected until the level in the tub goes down. :P

Here is the biggest file so far in the message directory:
http://www.countryplans.com/cgi-bin/yabb2/YaBB.pl?num=1107141843 (Started in Jan. 2005)
« Last Edit: November 12, 2007, 09:33:35 PM by jraabe »
None of us are as smart as all of us.

Offline MarkAndDebbie

  • Senior Member
  • ****
  • Posts: 205
  • NW GA-1.5 Story- Husband & Wife Team
Re: Compressing or archiving messages
« Reply #5 on: November 13, 2007, 04:00:39 AM »
I'm curious about the 2000 file limit for your ftp program. Do you know if it is a server imposed restriction? There is not anything in the FTP spec that would limit the number of files show.

1. What ftp programs have you tried?
2. Is it 2000 total files or 2000 files per directory? If per directory would making more divisions (forums etc) help?

Offline glenn kangiser

  • The Troglodyte
  • Administrator
  • Journeyman
  • *****
  • Posts: 24,906
  • Central California Sierras- Home of Yosemite NP
    • Underground Cabin
Re: Compressing or archiving messages
« Reply #6 on: November 13, 2007, 05:55:43 AM »
Something seems wrong here as the Dodge Truck forum is tons bigger with no problem.  They also have a ton more forums and topics and are more active than us.  

At this moment, DTR

Quote
Currently Active Users: 1191 (682 members and 509 guests)  Threads: 152,258, Posts: 1,599,532, Members: 43,395

http://www.dieseltruckresource.com/dev/index.php
"Always work from the general to the specific." J. Raabe

Glenn's Underground Cabin  http://countryplans.com/smf/index.php?topic=151.0

Please put your area in your sig line so we can assist with location specific answers.

Offline John Raabe

  • Administrator
  • Journeyman
  • *****
  • Posts: 5,550
  • Whidbey Island, WA
    • CountryPlans - About Us
Re: Compressing or archiving messages
« Reply #7 on: November 13, 2007, 06:53:49 AM »
The Dodge truck forum is running on vBulletin - http://www.vbulletin.com/


I will explore how the migration might work and if it could help us out of this bind.

The 2000 file limit of FTP appears to be built-in to WS_FTP and Cute FTP - those are two I checked.



If anyone knows of a program that does not have this limit I would appreciate it.
None of us are as smart as all of us.

Offline glenn kangiser

  • The Troglodyte
  • Administrator
  • Journeyman
  • *****
  • Posts: 24,906
  • Central California Sierras- Home of Yosemite NP
    • Underground Cabin
Re: Compressing or archiving messages
« Reply #8 on: November 13, 2007, 07:01:36 AM »
Possibly we could get in contact with the DTR guys for a clue.  Their forum is a friendly forum like ours so they would probably help with info.  Maybe I can find someone to message.
"Always work from the general to the specific." J. Raabe

Glenn's Underground Cabin  http://countryplans.com/smf/index.php?topic=151.0

Please put your area in your sig line so we can assist with location specific answers.

Offline John Raabe

  • Administrator
  • Journeyman
  • *****
  • Posts: 5,550
  • Whidbey Island, WA
    • CountryPlans - About Us
Re: Compressing or archiving messages
« Reply #9 on: November 13, 2007, 07:27:05 AM »
I'm waiting to make a few calls here... Another forum solution we should look at is Invision - http://www.invisionpower.com/community/board/convert.html

It looks like a conversion from Yabb is possible...

One of the newer features of both of these upgrades is built-in blogs. People building a house could setup an internal blog for storing photos and updates and then link to it in forum posts. Such treads would be less likely to get lost.
None of us are as smart as all of us.

Offline glenn kangiser

  • The Troglodyte
  • Administrator
  • Journeyman
  • *****
  • Posts: 24,906
  • Central California Sierras- Home of Yosemite NP
    • Underground Cabin
Re: Compressing or archiving messages
« Reply #10 on: November 13, 2007, 07:29:26 AM »
Sounds good.  I messaged Lary Ellis at DTR - don't know when I'll get a reply.
"Always work from the general to the specific." J. Raabe

Glenn's Underground Cabin  http://countryplans.com/smf/index.php?topic=151.0

Please put your area in your sig line so we can assist with location specific answers.

Offline glenn kangiser

  • The Troglodyte
  • Administrator
  • Journeyman
  • *****
  • Posts: 24,906
  • Central California Sierras- Home of Yosemite NP
    • Underground Cabin
Re: Compressing or archiving messages
« Reply #11 on: November 13, 2007, 07:33:01 AM »
Got it already -- great guys there.  I sent him a thank you.

Quote
Jack Thorpe  Jack Thorpe is online now
Founder / Webmaster
                          
Join Date: May 2002
Posts: 3,140
Yabb
Glenn,

I can tell you that YaBB has some problems. I used to used it before I moved over to vBulletin a few years ago. Once I got to about 3000 members, it started taxing my hardware really bad, and it would kill my servers. It's just too taxing on any large forum. I'm not sure if they've improved it now or not since I haven't used it in about 4 years.

I would recommend making the move to vb. It's kind of a pain at first but it's worth it. If you're using the cgi version of Yabb, then it'll be even worse. You might have to make the move to simple machines, then make the move to vb later.
__________________
2007 6.7l Dually
Quad Cab
"Always work from the general to the specific." J. Raabe

Glenn's Underground Cabin  http://countryplans.com/smf/index.php?topic=151.0

Please put your area in your sig line so we can assist with location specific answers.

Offline glenn kangiser

  • The Troglodyte
  • Administrator
  • Journeyman
  • *****
  • Posts: 24,906
  • Central California Sierras- Home of Yosemite NP
    • Underground Cabin
Re: Compressing or archiving messages
« Reply #12 on: November 13, 2007, 07:39:23 AM »
Apparently this is what Jack was talking about. Maybe an easier way to do the conversion.

http://www.simplemachines.org/
"Always work from the general to the specific." J. Raabe

Glenn's Underground Cabin  http://countryplans.com/smf/index.php?topic=151.0

Please put your area in your sig line so we can assist with location specific answers.

Offline MarkAndDebbie

  • Senior Member
  • ****
  • Posts: 205
  • NW GA-1.5 Story- Husband & Wife Team
Re: Compressing or archiving messages
« Reply #13 on: November 13, 2007, 08:59:24 AM »
Have you tried windows explorer? You can put the ftp://... in the address bar. If the address bar is not on - you can turn it on under view/toolbars/address bar. I do not think there is a number of files limit.

Offline John Raabe

  • Administrator
  • Journeyman
  • *****
  • Posts: 5,550
  • Whidbey Island, WA
    • CountryPlans - About Us
Re: Compressing or archiving messages
« Reply #14 on: November 13, 2007, 09:26:05 AM »
MarkandDebbie:

Neat trick with Explorer... I hadn't seen that before. (tips: http://www.developingwebs.net/publishing/ie.php)

Unfortunately, when I get into the yabb2 message directory the same 1134603564.txt file is the last one on the sorted list. That is the same limit of the FTP programs. Yet, this thread is numbered 1194888603.txt.
None of us are as smart as all of us.

Offline MarkAndDebbie

  • Senior Member
  • ****
  • Posts: 205
  • NW GA-1.5 Story- Husband & Wife Team
Re: Compressing or archiving messages
« Reply #15 on: November 13, 2007, 10:07:50 AM »
So I checked on explorer - it has no limit. It seems it can be set as a limit on the server to avoid denial of service attacks. You could ask your host to increase the limit.

Offline John Raabe

  • Administrator
  • Journeyman
  • *****
  • Posts: 5,550
  • Whidbey Island, WA
    • CountryPlans - About Us
Re: Compressing or archiving messages
« Reply #16 on: November 13, 2007, 11:35:58 AM »
Ya, this seems to be cast in place on the host server side. They are no going to change that file level... tried that.

Now I am looking at the process of migrating the Yabb forum to either vbulletin or invision power board. Sounds nasty!!
None of us are as smart as all of us.

Offline glenn kangiser

  • The Troglodyte
  • Administrator
  • Journeyman
  • *****
  • Posts: 24,906
  • Central California Sierras- Home of Yosemite NP
    • Underground Cabin
Re: Compressing or archiving messages
« Reply #17 on: November 13, 2007, 11:43:09 AM »
Nasty has it's places but moving the board isn't one of them. ::)
« Last Edit: November 13, 2007, 11:44:10 AM by glenn-k »
"Always work from the general to the specific." J. Raabe

Glenn's Underground Cabin  http://countryplans.com/smf/index.php?topic=151.0

Please put your area in your sig line so we can assist with location specific answers.

Offline John Raabe

  • Administrator
  • Journeyman
  • *****
  • Posts: 5,550
  • Whidbey Island, WA
    • CountryPlans - About Us
Re: Compressing or archiving messages
« Reply #18 on: November 13, 2007, 03:14:41 PM »
I have spent the day looking into transitioning to vbulletin or Invisions Power Board... but they both seem complex, especially with the conversion.

Then, I looked at the [highlight]Simple Machine Forum[/highlight] where there is a thread on conversions from Yabb:
http://www.simplemachines.org/community/index.php?board=135.0

While not exactly THAT simple it may be a better step than either of the above forum options.

SMF is a nice looking forum (much like YABB) and... it's also free.

    "SMF is written in the popular language PHP and uses a MySQL database. It is designed to provide you with all the features you need from a bulletin board while having an absolute minimal impact on the resources of the server. SMF is the next generation of forum software - and best of all it is and will always remain completely free!"

A bit later.... I've registered at SMF and done a sample post. It seems to do everything the Yabb forum does (without the file bloat).

http://www.simplemachines.org/community/index.php?topic=205700.msg1308192#msg1308192.
« Last Edit: November 13, 2007, 03:45:41 PM by jraabe »
None of us are as smart as all of us.

Offline MarkAndDebbie

  • Senior Member
  • ****
  • Posts: 205
  • NW GA-1.5 Story- Husband & Wife Team
Re: Compressing or archiving messages
« Reply #19 on: November 13, 2007, 04:07:44 PM »
If money was no object - you would go with http://communityserver.org/. Probably the hosted solution. They have a free version - but I think you'd have to restructure the "company" to use it.

Barring that SMF looked promising.

Offline glenn kangiser

  • The Troglodyte
  • Administrator
  • Journeyman
  • *****
  • Posts: 24,906
  • Central California Sierras- Home of Yosemite NP
    • Underground Cabin
Re: Compressing or archiving messages
« Reply #20 on: November 13, 2007, 04:59:53 PM »
I assume Jack from DTR knows something of the problems of switching to VB from YaBB so suggested Simplemachines because of the problems we may encounter with the cgi version of YaBB, and looking in the about section of the SMF I found that the people there were formerly from YaBB

http://www.simplemachines.org/about/
"Always work from the general to the specific." J. Raabe

Glenn's Underground Cabin  http://countryplans.com/smf/index.php?topic=151.0

Please put your area in your sig line so we can assist with location specific answers.

Offline glenn kangiser

  • The Troglodyte
  • Administrator
  • Journeyman
  • *****
  • Posts: 24,906
  • Central California Sierras- Home of Yosemite NP
    • Underground Cabin
Re: Compressing or archiving messages
« Reply #21 on: November 13, 2007, 08:20:16 PM »
The SMF specs are impressive.

Quote
1,270,967 Posts in 166,776 Topics by 116,952 Members
"Always work from the general to the specific." J. Raabe

Glenn's Underground Cabin  http://countryplans.com/smf/index.php?topic=151.0

Please put your area in your sig line so we can assist with location specific answers.

Offline John Raabe

  • Administrator
  • Journeyman
  • *****
  • Posts: 5,550
  • Whidbey Island, WA
    • CountryPlans - About Us
Re: Compressing or archiving messages
« Reply #22 on: November 14, 2007, 03:47:32 PM »
Hey, thanks to Don and Glenn for helping to check this forum stuff out.

The SMF foundation and history seems like a good match for this board.

So, what do you think?

Shall we take a deep breath and try and migrate this sucker???

We can keep this thread open and see if we can brainstorm any problems that come up... ([highlight]PROBLEMS?[/highlight], we don't need no stinking PROBLEMS!)

John
None of us are as smart as all of us.

Offline MountainDon

  • Administrator
  • Journeyman
  • *****
  • Posts: 16,680
  • Jemez Mountains, NM; that's pronounced HEY-mess
    • My 15.75 x 30 Jemez Cabin
Re: Compressing or archiving messages
« Reply #23 on: November 14, 2007, 04:07:43 PM »
You are welcome. Just having fun messing with it.  :)

You are the boss/owner John and the one who will bear the brunt of the changeover labor. But SMF would look like a good choice.



On a similar but other vein, I'm webmaster for our local dance club and have been considering a member forum for things for sale, trade, questions, and what not. I D/L'd the SMF files and was going to start playing with it, but ran into a problem. Add to that, my ISP is migrating me from one platform to another... I may just wait till the migration gets completed.
Just because something has been done and has not failed, doesn’t mean it is good design.

Offline glenn kangiser

  • The Troglodyte
  • Administrator
  • Journeyman
  • *****
  • Posts: 24,906
  • Central California Sierras- Home of Yosemite NP
    • Underground Cabin
Re: Compressing or archiving messages
« Reply #24 on: November 14, 2007, 06:06:15 PM »
Like you need another project, John, but it seems to be the only way to handle it without losing more of our forum at random times.

I'm here for whatever I can do to help. :)
"Always work from the general to the specific." J. Raabe

Glenn's Underground Cabin  http://countryplans.com/smf/index.php?topic=151.0

Please put your area in your sig line so we can assist with location specific answers.

 

Templates: 5: index (default), Ads (default), Portal (default), Display (default), GenericControls (default).
Sub templates: 12: init, html_above, adsheaders_above, body_above, adsindex_above, portal_above, main, portal_below, adsindex_below, body_below, adsheaders_below, html_below.
Language files: 3: SPortal.english (default), index+Modifications.english (default), Ads.english (default).
Style sheets: 1: portal (default).
Files included: 38 - 1121KB. (show)
Cache hits: 13: 0.00315s for 40,632 bytes (show)
Queries used: 24.

[Show Queries]