Welcome, Guest. Please login or register.

Author Topic: Amiga UseNet archives - only one copy left now  (Read 3531 times)

Description:

0 Members and 1 Guest are viewing this topic.

Offline OSS542Topic starter

  • Full Member
  • ***
  • Join Date: Mar 2002
  • Posts: 119
    • Show only replies by OSS542
Amiga UseNet archives - only one copy left now
« on: January 17, 2006, 10:23:49 AM »
When the Amiga was first introduced in the mid 1980's, the Web as we know it would not exist for another decade.  For the next 15 years, many of the leading people involved shared their knowledge and passion for creativity via UseNet newsgroups.  There is a great deal information of historical significance contained within those postings.  If I recall correctly, there used to be archives of the various Amiga newsgroups scattered about the net back in the day.  Over time, the various archives have been lost.

In my poking about for them, I have come to the conclusion that Google now has the only existing copies of significant archives for many of those groups.  Some of the older pre-1990 groups are locked and archive-only now, which is perhaps just as well.  The others are now part of a larger collection called Google groups, in which users create their own special interest groups, and to which Google has been making their own alterations at their whim.  Access to the old postings is now much more limited than it once was.  You can, for example, now view only what you can access manually.  Automated access is, at present, strictly prohibited.  The number of queries you can make in a day has had to be limited, thanks to the strain on Google caused by heavy access having to do with web page ranking.

I was reading the other day about the time back in 2001 when Google took over DejaNews.  I remember how DejaNews simply cut off access suddenly one day to all the archives, apparently at the time, for good.  I read about how technicians were still making backup tapes while others were unplugging equipment to be hauled away.  I was concerned about how very close they came to losing the only copies we have of many of these postings forever.

It seems now that Google has sole control of the only reasonably complete Amiga UseNet archives from the beginning to the present.  I have to wonder what would happen if, someday, the "powers that be" at Google decided to charge on a "per message" basis for access to the UseNet archives.  I wonder what we might do if they were to decide that old Amiga archives were no longer profitable or relevant, and should be discarded to save money or space.

I do not wish to accuse Google or anyone else of evil intent.  I am, however, quite concerned that one single entity has sole control over the disposition of and access to a body of work created by so many.  It could simply disappear for good, for any reason or none, and we would now have nothing to say in the matter.

How could we have allowed this to happen ?  What might we do to assure the preservation of these archives for the future, as a reminder of what so many of us stood for, and for that matter, lived for ?

Thoughts anyone ?
 

Offline MrZammler

  • Hero Member
  • *****
  • Join Date: Feb 2002
  • Posts: 720
    • Show only replies by MrZammler
Re: Amiga UseNet archives - only one copy left now
« Reply #1 on: January 17, 2006, 10:39:14 AM »
Couldnt agree more. Usenet was the best source of information for the amiga, and might still be. (As in searching to solve a problem).

Those archives should be kept somewhere, esp. IMHO comp.sys.amiga.hardware.
Anyway is the only way
 

Offline OSS542Topic starter

  • Full Member
  • ***
  • Join Date: Mar 2002
  • Posts: 119
    • Show only replies by OSS542
Re: Amiga UseNet archives - only one copy left now
« Reply #2 on: January 17, 2006, 11:21:32 AM »
I am hoping that somewhere there are still archives that might  be pieced together.   Downloading from Google groups is not terribly practical at the moment due to the restrictions in place to limit access.  For example, the comp.sys.amiga.hardware group has some 78,000+ topics listed, which may average about 5 or 10 messages apiece.  That would be around 400,000 messages for that group alone.  It's not a lot of storage space these days, but automated downloads are prohibited, and one person doing that by hand is not practical.  Google does have the a beta test web access api facility available, but that is restricted to the main google web index and is locked out of google groups.  It is also very limited in capability, and is further limited to 1000 queries a day.  Downloading the good Amiga groups (not all are) would take a couple of weeks at that rate, but would be practical if the google web api's were not locked out of google groups.  Google also requires that you register with them to get a special identifiable access key used to control your usage.  The supplied libraries only work with .NET and Java, but it may be possible to roll your own queries in C if you know that.

 

Offline MrZammler

  • Hero Member
  • *****
  • Join Date: Feb 2002
  • Posts: 720
    • Show only replies by MrZammler
Re: Amiga UseNet archives - only one copy left now
« Reply #3 on: January 17, 2006, 11:44:36 AM »
Just checked api.google.com, it uses soap, so it would be easy to do stuff in C. But, anyway, if there is no access through soap for groups then it's no good.

It's not easy either to create a script to fetch the pages through plain http... damn. Either we think of a way for it, or hope google groups does not shut down.
Anyway is the only way
 

Offline OSS542Topic starter

  • Full Member
  • ***
  • Join Date: Mar 2002
  • Posts: 119
    • Show only replies by OSS542
Re: Amiga UseNet archives - only one copy left now
« Reply #4 on: January 17, 2006, 11:47:27 AM »
One additional thought which might be something to consider...

I wonder if we could ask someone from one of the university Aminet sites to request copies of the older Amiga messages from Google groups to host on one of the university sites ?  It is possible they might be more receptive to a request from a non-profit organization interested in historical preservation.  The storage requirements for the text-only groups should be managable for a fixed period in the past.

One might, for example ask for the postings from 1984-2000 from a few of the more important groups.  This is the time period for much of the historically significant work on the classic Amigas I think.  I might propose the following groups as a thought :

pre 1990, now locked and archive-only, (much important early history here)

   comp.sys.amiga
   comp.sys.amiga.tech
   net.micro.amiga
   amiga.hardware

a partial list of currently active ones :

   comp.sys.amiga.hardware
   comp.sys.amiga.misc
   comp.sys.amiga.programmer
   comp.sys.amiga.applications
   comp.sys.amiga.graphics
   comp.sys.amiga.audio
   comp.sys.amiga.introduction
   comp.sys.amiga.announce
   comp.sys.amiga.reviews
   comp.sys.amiga.software

I've left out quite a few which I or others might care to add, but those stand out from a technological standpoint.  The locked groups from before 1990 particularly need to be preserved before they are lost.   The activity on just about all the others is in serious decline or gone all together, so additional archiving on some of them will probably stop soon as well.

Indexing for search might have to be considered later.  I am thinking that the important thing right now is to at least save the text in an accessable place to prevent its eventual loss to the future.

 

Offline OSS542Topic starter

  • Full Member
  • ***
  • Join Date: Mar 2002
  • Posts: 119
    • Show only replies by OSS542
Re: Amiga UseNet archives - only one copy left now
« Reply #5 on: January 17, 2006, 01:28:19 PM »
Quote

MrZammler wrote:

It's not easy either to create a script to fetch the pages through plain http... damn. Either we think of a way for it, or hope google groups does not shut down.


I have already written a set of C programs and scripts which can do exactly that.  The only problem is that google currently prohibits automated access.  Anything slow enough not to be noticed would still be technically against the rules, and too slow to be practical in any case.

We have two possibilities at least.  One is to try and persuade Google to release a set of "antiquated" text files to some sort of non-profit historical archive, preferably university based like Aminet.   Another is to try and persuade them to allow the web api to access Google groups.  I can take it from there.
 

Offline cgutjahr

  • Hero Member
  • *****
  • Join Date: Feb 2003
  • Posts: 697
  • Country: 00
    • Show only replies by cgutjahr
Re: Amiga UseNet archives - only one copy left now
« Reply #6 on: January 17, 2006, 03:13:19 PM »
I'm one of the Aminet admins. I did not discusss this with the rest of the team yet, but personally I like the idea of making a 'safe' backup of the Amiga related Usenet groups.

It should be noted though that if Aminet is going to carry these backups, it will be most likely in the form of ZIP archives (like one ZIP per year and group for example) or something like that. We certainly lack the manpower to implement anything resembling dejanews'/google's comfortable web interface - not to mention the fact that such a web interface would require hosting half a million postings as individual text files.

Of course, once the postings are available from Aminet, a third party could use the data to implement a web interface or similar things.

I agree that the best approach would be to approach google stressing the 'non-profit organisation' concept of Aminet. Any further suggestions?

Oh, and could somebody supply me with a contact adress for google? Somehow neither IB nor AWeb will let me visit the original google.com site, I always end up on the German version (despite following the link "google.com in English").
 

Offline drHirudo

  • Hero Member
  • *****
  • Join Date: Nov 2003
  • Posts: 539
    • Show only replies by drHirudo
    • http://hirudov.com
Re: Amiga UseNet archives - only one copy left now
« Reply #7 on: January 17, 2006, 03:58:34 PM »
Many newsgroups posts were archived on CDs, for example Net News Offline, which contain thousands messages (more than 217,000 in Net News Offline volume 2). I remember back in the days, where Google restored all the available archives, they used such CDs, because some of the archives at DejaNews where already deleted.

Offline OSS542Topic starter

  • Full Member
  • ***
  • Join Date: Mar 2002
  • Posts: 119
    • Show only replies by OSS542
Re: Amiga UseNet archives - only one copy left now
« Reply #8 on: January 18, 2006, 12:13:03 AM »
Quote

cgutjahr wrote:
I'm one of the Aminet admins. I did not discusss this with the rest of the team yet, but personally I like the idea of making a 'safe' backup of the Amiga related Usenet groups.


I am very glad to hear from you. Thanks for taking the time to answer!!!

Quote

It should be noted though that if Aminet is going to carry these backups, it will be most likely in the form of ZIP archives (like one ZIP per year and group for example) or something like that. We certainly lack the manpower to implement anything resembling dejanews'/google's comfortable web interface - not to mention the fact that such a web interface would require hosting half a million postings as individual text files.


Understood.  I am thinking that the important thing at present is just to safeguard the Usenet postings at at least one other site besides google to assure their preservation.  Once that is done, I'd be happy to offer my assistance in maintaining the collection.  We may in fact not wish to put too much effort into a search facility, at least not right away if ever.  We don't want to give Google the impression that we intend to offer some sort of "competition".  We simply want to assure the preservation of our collective creation and history.  It is also probably very important not to ask for anything too current.  I would suggest the period from 1994-2000 or thereabouts, and a small number of the best newsgroups.  I spent some time sifting through a large list of UseNet Amiga newsgroups to find the ones which met the following criteria:

1. a real amiga newsgroup, not spam infested

2. significant technological or creative information important to the history of the Amiga could be obtained

3. the primary posting language was English.  There are a number of very good groups in other languages e.g. Italian, Russian, German, but these are less likely to be widely useful to non-native speakers

A select list of newsgroups like this will make the preservation much more manageable.  I'll PM the list to you if you have an interest.


Quote

Of course, once the postings are available from Aminet, a third party could use the data to implement a web interface or similar things.

I agree that the best approach would be to approach google stressing the 'non-profit organisation' concept of Aminet. Any further suggestions?


I will work on coming up with ideas on this.  I think it would be important to emphasize any university connection Aminet may have if possible.  We want to avoid giving Google the impression we are a small club or local user's group.  Large corporations can be very reluctant to deal with such.  We should also emphasize the historical preservation aspect of what we wish to do here.  We do not want to give them the impression that we want to make a profit, we simply wish to preserve the history we as the Amiga community have created and made available for Google to use.

[/quote]
Oh, and could somebody supply me with a contact adress for google? Somehow neither IB nor AWeb will let me visit the original google.com site, I always end up on the German version (despite following the link "google.com in English").
[/quote]

Google apparently uses the characteristics of your IP address to redirect you to the google site it thinks appropriate for your location.  If you have cookies on, you can override this by using the following url:

http://www.google.com/ncr

Google prefers to be contacted by web form or email.  Contact information is available at the following url:

http://www.google.com/intl/en_extra/contact/search.html

The google support group is also available at the following url:

http://groups.google.com/group/google.public.support.general

I am also registered with the Google web api development program.  I am thinking of raising this and related issues there and see what comes of it.

No one there has recently raised the issue of making the Web API available to Google groups.  Recent emphasis there has been primarily on "Page Ranking" algorithms. This might be a very good time to remind people of Google groups.
 

Offline OSS542Topic starter

  • Full Member
  • ***
  • Join Date: Mar 2002
  • Posts: 119
    • Show only replies by OSS542
Re: Amiga UseNet archives - only one copy left now
« Reply #9 on: January 18, 2006, 12:16:21 AM »
Quote

drHirudo wrote:
Many newsgroups posts were archived on CDs, for example Net News Offline, which contain thousands messages (more than 217,000 in Net News Offline volume 2). I remember back in the days, where Google restored all the available archives, they used such CDs, because some of the archives at DejaNews where already deleted.


I remember the postings by Google looking for these as well.  Would you happen to know which Amiga UseNet postings would have been included on the NetNews offline CD's ?  Do you know of any other such collections ?
 

Offline drHirudo

  • Hero Member
  • *****
  • Join Date: Nov 2003
  • Posts: 539
    • Show only replies by drHirudo
    • http://hirudov.com
Re: Amiga UseNet archives - only one copy left now
« Reply #10 on: January 18, 2006, 06:22:15 AM »
Quote
Would you happen to know which Amiga UseNet postings would have been included on the NetNews offline CD's ? Do you know of any other such collections ?

I have NetNews Offline Vol.2 and it have 3 months of published articles since the release of NetNews Offline Vol.1

Here are the contents from the backcover:

comp.sys.amiga.*:          83.087 articles.
com.unix.amiga:               887 articles.
rec.games.video.cd32:         507 articles.
comp.sys.cbm                5.955 articles.
de.comp.sys.amiga.*:       14.535 articles.
de.comm.software.ums:         838 articles.
fide.ger.{amiga|amiprog}:   2.721 articles.
z-netz.rechner.amiga.*:    23.427 articles.

and some others not directly amiga related like comp.perphs.scsi, comp.lang.rexx.

The newsgroups post archives were archived by the

CD Publishing Corporation.

   NetNews on CD's -- USENET Newsfeed and Archive US$20-US$34

    NetNews on CD's is the premier USENET News on CD-ROM's product from CD Publishing. This subscription oriented product provides an archiving and distribution mechanism for USENET news, along with effective software search tools for quick and easy access to this huge base of valuable information. NetNews on CD's makes the wealth of information on USENET available when you need it, where you need it. NetNews on CD's alleviates the requirements for excessive disk storage and tape archival/retrieval facilities. It also removes the limitations of communication link speeds and prohibitively high newsfeed and/or telephone charges. Issues in this series are produced as soon as they are full, and can be purchased on a single issue or multi-issue subscription basis. NetNews on CD's is currently produced in ISO 9660 format.

also InfoMagic:
   The following products are available to ISOC members from InfoMagic. Address your orders to: info@infomagic.com or phone 609-683-5501 or FAX 609-683-5502. USENET CD-ROM List $20.00 ISOC Members $15.00 Assorted archives of the USENET news groups, including the following comp.sources groups: misc, games, reviewed, sun, unix, x containing source code for UNIX(tm), Sun, and X-Windows. Also includes the FAQ (Frequently Asked Questions Lists) for many other groups. ISO-9660 format with Rock Ridge extensions.

Other newsgroups post were on other CDs for example CPM posts in the Walnut Creek CD ROM archive and many other.


Offline OSS542Topic starter

  • Full Member
  • ***
  • Join Date: Mar 2002
  • Posts: 119
    • Show only replies by OSS542
Re: Amiga UseNet archives - only one copy left now
« Reply #11 on: January 23, 2006, 11:28:02 PM »
Just wondering....

Have you been able to discuss this idea with the rest of the Aminet team ?  
Did you have any luck with contacting google ?

Regards,
   OSS542