Amiga.org

Amiga computer related discussion => Amiga/MorphOS/AROS Programmers Forum => Topic started by: nyteschayde on January 23, 2017, 04:59:27 PM

Title: AmiGet - Aptitude like app for aminet
Post by: nyteschayde on January 23, 2017, 04:59:27 PM
It's in the early stages but I'm working on doing an aptitude, a Debian package manager for the CLI, for animet. It should be able to read from and search the INDEX file at the root of aminet.net, or a locally cached copy, for files and download them. It will be CLI based so you won't need the massive overhead of IBrowse, AWeb or Netsurf.

I haven't decided where, but it should also be able to eventually read a dependencies file from someplace that will reach out and grab dependencies for a given archive.

First steps


Early extras


Stretch


Thoughts? Anybody wanna help? This is a project to help refresh my C skills and Amiga skills simultaneously. I've already started.
Title: Re: AmiGet - Aptitude like app for aminet
Post by: eliyahu on January 23, 2017, 05:03:18 PM
@nyteschayde

sweet! how will you build the initial DB of dependencies, though? this is something i'd love to see, and i'd be happy to add support for AOS4 (if any changes needed) if the source is planned to be open.

-- eliyahu
Title: Re: AmiGet - Aptitude like app for aminet
Post by: nyteschayde on January 23, 2017, 05:16:24 PM
The Amiga community is not going to be benefited much by closed source. It'll be open. I'm considering the least obtrusive would be to have at least a two fold service.

1. It there is a .aminet.deps file in the archive, use it.
2. Otherwise look someplace else; if it's allowed and accepted, we can have a deps path in aminet that mirrors the same root structure but only hosts .deps text files that serve the same purpose

Benefits from this approach are twofold.

1. New archives can be specific.
2. We can add support for old files without touching them

Format is TBD but should likely support dependencies, dev dependencies, and optional dependencies
Title: Re: AmiGet - Aptitude like app for aminet
Post by: cgutjahr on January 23, 2017, 06:04:49 PM
Just so nobody gets confused: This is an official Aminet statement.

Quote

Read and search INDEX [...] Download and search full .readme files for potential matches.

INDEX is close to 7 MB right now, not sure what kind of target audience you have in mind? Downloading a 7 MB file every time you run the application won't be fun on an Amiga. Not to mention the amount of memory searching this file would require.

People downloading 81000 readmes because they want to search something? That's a no-go.

Plus, you'd have to implement a mechanism to make sure the local copy stays up to date. A quick look into our changelog will show you that we're constantly cleaning up the archive and/or the directory tree, or adding new files which won't show up in RECENT.

A much more sensible approach would be to actually ask us for ideas. Here's one: We expand the existing search engine so it displays results in some machine readable format (TSV?, JSON?) and external clients use that instead: they simply download aminet.net/json/some_search_string via HTTP and parse it.

Providing a downloadable version of the entire database would be another option, of course - but not for an Amiga based solution, obviously.

I was planning a new Aminet client anyway (and searching a coder for that job), but a graphical one. For CLI access, there's ADT (Aminet Download Tool) - a console FTP client with some extras (display recent, display directory listings in Aminet format, search for filenames and descriptions).

Most if not all GUI FTP clients on the Amiga cloned these features aswell, so IMHO instead of reinventing the wheel, a better idea might be to update ADT (it doesn't compile on Unix systems anymore, IIRC) and get the search daemon on Aminet (written in Python and C) running again - that way, all Amiga FTP clients would benefit.

But tbh, I don't see the need for a console based client anymore.

Quote

Unpack archive (requires lots of extras to be able to do this on the Amiga. LhA, LZX, UnRar, UnZip, XPK, etc)

No, that actually only requires the XAD library system. Aminet doesn't support RAR or LZX anyway.

Quote

I haven't decided where, but it should also be able to eventually read a dependencies file from someplace that will reach out and grab dependencies for a given archive.

This already exists, there's a 68k port of Grunch - and it's completely dead, because maintaining the database is both a nightmare and a lot of work. The MorphOS database used to be (still is?) quite up to date, but there's simply not enough interest on 68k anymore.

Here's what I had in mind, the target audience being Amigas with moderate to low specs and some form of network access:



Basically: get rid of the Browser (+MUI/Classact) requirement, do not rely on other computers in the household - but still offload all the resource heavy tasks to an external machine (i.e. the Aminet server).
Title: Re: AmiGet - Aptitude like app for aminet
Post by: cgutjahr on January 23, 2017, 06:16:03 PM
Quote from: nyteschayde;820523

Format is TBD but should likely support dependencies, dev dependencies, and optional dependencies

We tried to implement something like that a decade ago, based on Debian's deb format.

I'd have to search for the old drafts, but you need more than just software dependencies. First you need a way to reference packages without relying on their filename (think of mui-master-16.1.lha vs. mui-master-16.2.lha) - we used a NAME: field in the readme ("Name: MUI-Master"), IIRC - min. and/or max. version numbers of the dependency,  CPU/RAM/Chipset requirements... Plus, you'd also have to document compatibility with various OS versions, Chipsets, RTG...
Title: Re: AmiGet - Aptitude like app for aminet
Post by: nyteschayde on January 23, 2017, 06:17:04 PM
The more features you support in regards to searching, the better. I had some notion of allowing the users to download the INDEX and update it every once in a while on the local machine. But if you're offering an API endpoint on the aminet servers that handle searching, all the better. I'll happily consume it. Less work for me.

The ADT download tool, if it is the one I am thinking of, does not support search. It only allows you to download if you know the actual path. That's part of the problem. And it doesn't handle dependencies either. All the FTP clients are the same to my knowledge.

I'd disagree that there isn't any interest on 68k machines but I could be wrong. From my perspective, and I own a A1200/PPC that runs 4.1, there are far more classic owners running 68k setups that need 68k archives than there are 4.1 owners and users.

I like your ideas for a GUI approach (mostly) but I still think that a decent CLI tool should accompany it. I'll look into XAD but I am pretty sure there are LZX and at least ZIP files on aminet.
Title: Re: AmiGet - Aptitude like app for aminet
Post by: nyteschayde on January 23, 2017, 06:26:39 PM
I don't doubt it would be an undertaking and you're right, even with the version field, exact file names would need to be referenced on Aminet to even have a chance. It'll likely have to support getting files from other places eventually too. So that'll have to be built in to expectations early on.

Path/Category: text/edit (forexample)
Filename: blahblah.lha
CPU: 68000/68010/68020/68030/68040/68060/PPC/x86/x64
OS: {plat:AmigaOS/AROS/MorphOS, ver:3.1}
MMU: true/false
RAM: {type:CHIP, min:1MB}

PPC could possibly be broken out to include more specifics. There might be a need to have an optional script for each dependency.

Script: (DOS)
Cd ""
Copy Libs/somelibrary020.library :somelibrary.library

Script: (INSTALLER)
...INSTALLER script commands...

Script: (INSTALLER_PATH_RELATIVE_TO_ARCHIVE)
"somelib/Some Library Install"

There's a lot that can be added. One step at time, however, would be ideal.
Title: Re: AmiGet - Aptitude like app for aminet
Post by: nyteschayde on January 23, 2017, 06:29:32 PM
Quote from: cgutjahr;820525

Basically: get rid of the Browser (+MUI/Classact) requirement, do not rely on other computers in the household - but still offload all the resource heavy tasks to an external machine (i.e. the Aminet server).


I think we'd have to at least use gadtools.library if we wanted any chance of a decent looking OS2.x+ based GUI. I am not even sure when gadtools.library made its debut offhand. I think it was 2.x but it might have been 3.x

cgutjahr, are there any existing search APIs? PM me and we can exchange emails for putting on together if there isn't one. I'd really like to make some subtle refinements on the advanced search that exists today in Aminet. Sounds like you already have some ideas. I could help.
Title: Re: AmiGet - Aptitude like app for aminet
Post by: cgutjahr on January 23, 2017, 06:33:41 PM
Quote from: nyteschayde;820531

The ADT download tool, if it is the one I am thinking of, does not support search.

Yes, it does - in theory. Just like most Amiga specific FTP clients do, who simply reimplemented the protocol they saw in the ADT sources. We stopped running the search daemon on our server years ago, simply because nobody was using it and there was a compatibility issue (IIRC) which would have required some effort that nobody wanted to invest.

Quote

And it doesn't handle dependencies either. All the FTP clients are the same to my knowledge.

That's true for FTP clients, but like I said: Grunch handles dependencies and automatic installation - it just needs database maintainers.

Quote

I'd disagree that there isn't any interest on 68k machines but I could be wrong.

I'm not saying there's no interest in 68k in general - but there are not enough people willing to donate sufficient amount of their spare time to a project like that. Retro gaming, fine-tuning your decade old Workbench setup every few months, buying/cleaning/fixing hardware? Sure, a lot of people love doing that.

But people who reliably take care of a database, every other day? Most of these are on OS4/MOS/AROS these days.

Quote

I am pretty sure there are LZX and at least ZIP files on aminet.

http://aminet.net/search?query=*.lzx

There are ZIP files, of course, but we don't support proprietary formats (with the exception of MP3/MPEG, for historical reasons), our wiki lists supported filetypes (http://wiki.aminet.net/Uploading_instructions#Filename_conventions.2C_accepted_filetypes).
Title: Re: AmiGet - Aptitude like app for aminet
Post by: cgutjahr on January 23, 2017, 06:53:24 PM
Quote from: nyteschayde;820534
I think we'd have to at least use gadtools.library if we wanted any chance of a decent looking OS2.x+ based GUI. I am not even sure when gadtools.library made its debut offhand. I think it was 2.x but it might have been 3.x

Gadtools was part of 2.x. Not sure you need Gadtools. A menu bar, plus simple text and file requesters should cover most of our needs, IMHO.

Quote

are there any existing search APIs?

Not yet. The only other format option implemented so far is "PRE" (preformatted), and that's just half finished - it was a test for amiga-news.de's 'Aminet uploads' news items. Append &format=pre to the URL of any search results page to see how this works. Oh, and &amount=XXX can be used to determine the amount of files you want to see on the first page.

Quote

PM me and we can exchange emails for putting on together if there isn't one.

I don't see a need for private discussions. Basically, it would come down to having search.php provide whatever format people agree on.
Title: Re: AmiGet - Aptitude like app for aminet
Post by: nyteschayde on January 23, 2017, 08:26:00 PM
Quote from: cgutjahr;820543
Gadtools was part of 2.x. Not sure you need Gadtools. A menu bar, plus simple text and file requesters should cover most of our needs, IMHO.


Heh, I am a senior UI engineer and have been for several major companies. I don't think I'd be able to stop there. :)

I mentioned that this is going to be a refresher for me in regards to C, but I've worked for Facebook, Apple, Netflix, Google, and now PayPal as well as plenty of other companies. I do native mobile (iOS/ObjC), native MacOS (ObjC), all manner of web (HTML5/CSS3/JavaScript, NodeJS, PHP, Python, Java and more) plus all the in between and data storage solutions (SQL and NoSQL).  

So, when I said if you need help with the API side of things, I  meant it. :)

LinkedIn Profile (https://www.linkedin.com/in/nyteshade)
Title: Re: AmiGet - Aptitude like app for aminet
Post by: jkadaver on January 27, 2017, 10:06:22 AM
Very cool
Title: Re: AmiGet - Aptitude like app for aminet
Post by: apa on April 24, 2017, 11:03:59 AM
Any updates regarding this?
Title: Re: AmiGet - Aptitude like app for aminet
Post by: nyteschayde on May 02, 2017, 02:13:42 AM
Quote from: apa;824929
Any updates regarding this?


Sorry for the slow response to this. I have been sidetracked by life but plan to get back on it. I have done some of the plumbing work I needed but I'd say this is at the, roughly, 20% mark. Once I have gotten back to this and made some more progress, I'll post a GitHub link with some updated and pushed content.

Thanks for checking back in and reminding me.
Title: Re: AmiGet - Aptitude like app for aminet
Post by: NinjaCyborg on October 28, 2018, 02:44:13 PM
I think an interesting place to start that might be more manageable on scope would be to make a library installer. For example, user runs an app that needs reqtools.library. A single command 'amiga-apt install reqtools.library' would check if it was installed already, check a master index of the latest known good library versions which itself could just be an index file submission on aminet in a known fixed location, and then fetch the .lha file with the latest platform appropriate version and extract the library file (without source or other unnecessary cruft, but showing any relevant readme or copyright notice in a text window) into the user's LIBS: or preferred secondary Libs folder e.g. SYS:Local/Libs/

If that worked and got traction you could expand it to datatypes, device drivers and eventually even whole apps.
Title: Re: AmiGet - Aptitude like app for aminet
Post by: nyteschayde on October 29, 2018, 06:00:52 AM
Would be happy to work with you on it. My available time for programming other things has basically completely stalled for the moment.  :(
Title: Re: AmiGet - Aptitude like app for aminet
Post by: NinjaCyborg on October 29, 2018, 11:55:46 AM
Me too as much as I'd love to implement all my dream Amiga projects. However I reckon one could get pretty far prototyping this in shell scripts, using http handler and xad for the tricky bits.

P.S. it's me Tim