Envoy source and availability

[kolla]

Perhaps you have reading comprehension issues.

Or maybe you have such issues? As olsen points out, there are "clashes" that needs to be resorted manually for Envoy and TCP/IP stacks to work well on same SANA-II device. This was still needed last time I tried latest version of Envoy (which I admit is like many years ago now). This is what I meant with "hackish".

Here i from the 3.1 update on aminet (http://aminet.net/package/comm/net/Envoy3_1Update)

If you are using a TCP/IP protocol stack like AS225, INet-225, Miami, or AmiTCP, you need to have SANA-II networking devices that support multiple protocol stacks. Most devices do. You also should not use the same IP or ARP numbers for Envoy that you are using for TCP/IP. Unless you are specifying your own numbers during the configuration, this Installation will automatically increase the common defaults to avoid conflicts.

(It is also worth nothing that this update has binary patch for a2065.device and ariadne_ii.device - hm, why is that)

[kolla]

@olsen

On a slightly related note, with OS 3.2...
- what do C:Owner and C:Group actually do?
- what do C:List LFORMAT %U and %G actually show?

What user/group database is supposed to be in place here? Where is this supposed to go?
Integration with Roadshow usergroup.library? Old MuFS? Does Amiga need equivalent of nsswitch?

[olsen]

@olsen

On a slightly related note, with OS 3.2...
- what do C:Owner and C:Group actually do?

They set up which specific user and which members of a specific group may access the files, directories, etc. on a volume you control and which is made available for use through a networked file system that enforces these access controls. One example of this kind of file system being the one implemented by the Envoy filesystem.service, which in turn draws upon the local Envoy user and group database via accounts.library.

Basically, "Owner" and "Group" work much like "Protect" does, except that they change the FileInfoBlock.fib_User and FileInfoBlock.fib_Group information of the directories, files, etc. in question.

- what do C:List LFORMAT %U and %G actually show?

They show the name of the user and group, respectively, who/which may have access rights for the respective file, directory, etc.

It's the file system's business to translate the numeric IDs stored in the FileInfoBlock.fib_User and FileInfoBlock.fib_Group into human-readable information. If this is a networked file system, such as exported by Envoy's filesystem.service, then the mapping between the numeric IDs and their associated names is again performed through accounts.library.

If the file system cannot translate the numeric IDs for user and/or group, then the "List" command will just show the numbers as a fall-back.

What user/group database is supposed to be in place here? Where is this supposed to go?

If this is Envoy's filesystem.service providing access, you would first populate its database through the "Accounts Manager" tool, then assign the access rights for the local volume which you want to export.

Integration with Roadshow usergroup.library? Old MuFS? Does Amiga need equivalent of nsswitch?

The way this was designed, it's the network file system which needs to enforce the access rights. Because the client has to be granted permission to access the exported files, directories, etc. the file system server can effectively deny access.

So much for theory. The hashing algorithm behind the Envoy user/group database is very weak and could be cracked easily. Which is probably no surprise, given how little was known about one-way hash functions back in the day outside the circle of security experts. Deploying a Unix-like password hash function scheme based on DES in 1992 would have been a real firecracker. At about that time, there were two BSD 4.4Lite-2 source code variants, the domestic and the export version. Only the weakened export version was deemed "safe". So, no fancy password hash function for Envoy

I have no idea how effective MuFS actually was/is. The Amiga operating system is single user by default because the kind of controls a multi-user operating system provides would have been hard to implement without memory protection. Costly, too, since it would have had to be built around dedicated hardware (such as the Sun 2/3 made use of) making the entire system run more slowly (how does that fit into the feature set of a game machine?). Retrofitting security onto an operating system not designed to support it is always painful

[olsen]

Here i from the 3.1 update on aminet (http://aminet.net/package/comm/net/Envoy3_1Update)

(It is also worth nothing that this update has binary patch for a2065.device and ariadne_ii.device - hm, why is that)

Because there were bugs in each which both could and should be fixed?   The a2065.device is not exactly notorious for being full of surprises, but it should be. In the Roadshow bsdsocket.library I had to work around a2065.device bugs which were easily triggered. Even the original AmiTCP 3.0b2 code featured a workaround which passed the respective I/O request unit pointer in register A3 when calling AbortIO(). And unless S2_DEVICEQUERY features Sana2DeviceQuery.SizeAvailable = offsetof(struct Sana2DeviceQuery, RawMTU), some a2065.device versions (there are at least three which I know of) will crash instantly.

[kolla]

@olsen

Thank you for taking time to answer this, it’s a bit puzzling with these commands in OS 3.2 that don’t appear to do anything,  it’s not obvious they are only for network filesystems, in practice only Envoy. And back in the days, also MuFS (which integrated better with IP stacks, as uid/gid 0 is “root/admin” and not “nobody/guest”). Do you know of any other filesystem supporting this? Your own smbfs perhaps? I don’t see any attempt to open any library or resource when using owner/group/list…

As for bugs in sana2 devices, yes… that’s always a problem, not all sana2 devices were “prepared” to be shared.

MuFS was fairly effective when using standard commands/software, but without memory protection it’s all kinda moot. Geert jumped off Amiga when CBM folded, and has since been active Linux developer, where he to this day is the man signing off patches to Linux/68k (as well as powerpc iirc). He still has his A4000 I think

[olsen]

@olsen

Thank you for taking time to answer this, it’s a bit puzzling with these commands in OS 3.2 that don’t appear to do anything,  it’s not obvious they are only for network filesystems, in practice only Envoy.

Indeed, the expected file system behaviour is specific to Envoy's "EnvoyFileSystem" (mounts the shared network volumes) and its counterpart "filesystem.service" (enables a directory or volume to be shared and mounted).

And back in the days, also MuFS (which integrated better with IP stacks, as uid/gid 0 is “root/admin” and not “nobody/guest”).

With Envoy's "filesystem.service", both the user ID and the group ID value 0 refer to the Administrator. By default, this will deny access to all the directories, files, etc. exported by the volume. You have to deliberately add the users and groups who/which should be able to access the contents of the volume and then make use of the Owner and Group shell commands to grant them access to the specific directories and files which should be available. Which is a sound approach, I would say. This is not a "toy", it has a sensible technical foundation.

Do you know of any other filesystem supporting this? Your own smbfs perhaps? I don’t see any attempt to open any library or resource when using owner/group/list…

I am not that well-read when it comes to file systems and neworked file systems in particular   From what I gather, the approach taken by filesystem.service and EnvoyFileSystem emerges directly from the concepts which underpin Envoy: simplicity and usability vs. requiring the user to set up every little thing correctly and getting slapped in the face if just one of these things is not entirely correct. SMBv1/CIFS features administrative level management commands which are sent by the client to the server, which are comparable to the filesystem.service/EnvoyFileSystem message set (changing/querying authorization rights, ACLs, etc.), but on a far larger scale. They also work only within the SMB "ecosystem", e.g. Windows NT and beyond (the official CIFS documentation does not go into any kind of detail for these messages). My smbfs port is definitely a client which relies upon the server to grant it access to the resources it is entitled to. It has no business even trying to provide administrative access methods.

As for bugs in sana2 devices, yes… that’s always a problem, not all sana2 devices were “prepared” to be shared.

Not all SANA-II devices were doing the boring parts of the job to begin with. Namely, validating the OpenDevice() parameters, the I/O requests submitted to Begin() and AbortIO(), and dealing gracefully with wonky input, returning a helpful error code. The whole point of using exec devices is in being able to validate and reject invalid input in real time. If you cannot even accomplish that, why even bother? 

[NinjaCyborg]

here's a relevant follow up I'd love to hear an answer from @olsen about

Both Inet225 and Envoy offer service discovery and control mechanisms for 'daemons' effectively. Also, RexxMast could be considered to be a 'daemon' in the unix sense. While AddDataTypes could be considered to be a kind of registry for plugins.

Modern platforms obviously have systemctl, launchd etc for this kind of thing. What would a system wide service discovery and management solution for amiga OS look like?

[kolla]

I'd say "commodity interface", aka CX.

All MUI software by default offer a commodity interface, and it's darn handy.

[NinjaCyborg]

I didn't ask you, and once again, you totally misunderstood the question.

[F0LLETT]

I didn't ask you, and once again, you totally misunderstood the question.

Damn, Kola was actually trying to help and you shut him down like that.

[olsen]

here's a relevant follow up I'd love to hear an answer from @olsen about

Both Inet225 and Envoy offer service discovery and control mechanisms for 'daemons' effectively. Also, RexxMast could be considered to be a 'daemon' in the unix sense. While AddDataTypes could be considered to be a kind of registry for plugins.

Modern platforms obviously have systemctl, launchd etc for this kind of thing. What would a system wide service discovery and management solution for amiga OS look like?

Hm... I have worked with INet-225 when it became part of the Amiga-Surfer product for Amiga Technologies GmbH, but I did not explore the technical aspects which distinguished it, compared to AmiTCP (at the time the only contender). INet-225 did support inetd-style daemon services, but so did AmiTCP. It was baked into the BSD TCP/IP "stack". Envoy made use of local UDP broadcast, for peers to announce their presence. Not exactly what would be called service discovery these days, which would be Zero configuration networking (Zeroconf) which makes use of multicast DNS.

RexxMast certainly qualifies as a daemon in the same vein as the inetd-style daemon services offered by TCP/IP stacks. Except that this is a local service which does not support network operations or discovery. It could have been interesting to make an Envoy service that would have allowed for remote ARexx script execution. Envoy's networking model was designed as an extension of the exec FindPort/PutMsg/WaitPort/ReplyMsg message passing.

The DataTypes system is something else, in my opinion. Its focus is on representing on-disk data via BOOPSI objects, with datatypes.library finding the relevant disk-loaded classes to match the file contents. You may ask for DataTypes to produce an object for you, but you may not get it.

We don't really have a general service discovery mechanism in the Amiga operating system...

[NinjaCyborg]

> We don't really have a general service discovery mechanism in the Amiga operating system...

Yes exactly I was just wondering if it was something you'd ever thought of how it might work. Obviously we can Run ><NIL: headless things that open message ports and sockets and the like, and yes as kolla says commodities can give you something similar for GUI based applets that have ephemeral user interfaces but benefit from persistent sessions. But perhaps a hypothetical future Amiga OS would benefit from a centralised service management layer. AFAIK you didn't implement inetd in Roadshow?

what's the use case? here's one - every time you want to use an external rexxport, you need to check if the port exists, and if it doesn't, run the host for that port, if you even know where to find it. Or you want to open on a named PUBSCREEN. But you're first to arrive, and the screen isn't open yet.

[ncafferkey]

As I wrote, there is no wiggle room for changes to be made in how Envoy deals with SANA-II devices shared by another IP protocol stack (could be the Oxxi ACS client, could be AmiTCP) when broadcast ARP replies enter the picture. Whoever queued the first S2_READ command will deprive the other stack from learning about the ARP reply's message.

According to the SANA-II standard, stacks should be able to share received frames:

Another recommendation for the ``magic cookie`` is to use it to maintain a separate packet read queue for each device opener. This would allow multiple protocol stacks that all wish to receive the same packet type to work together without having to "know" about each other as Envoy and AS225 do right now. What does multiple protocol stack support mean? Basically this means that each opener gets all the packets necessary. If a packet comes in that fills a request for more than one opener of the device, all of them will get a copy of the packet. This feature should never be left out of a device design. If it is missing, the usefulness of the device is severely limited.

The SLIP and A2065 driver now do this, so it would be possible (for example) to run Envoy, AS225 and the AmiTCP package together on the same hardware without conflicts.

(https://wiki.amigaos.net/wiki/Revision_2_%2B_3#Buffer_Management)

I know this is only a recommendation, but it seems it should allow Envoy and a TCP stack to coexist, at least when using certain drivers, such as the A2065 one mentioned above and several of my own.

[olsen]

According to the SANA-II standard, stacks should be able to share received frames:
(https://wiki.amigaos.net/wiki/Revision_2_%2B_3#Buffer_Management)

Indeed, but in practice Envoy steals every single frame it receives before a TCP/IP stack or other consumer (Oxxi Novell Netware client, etc.) could claim it. I used to suspect that this was a limitation of the respective SANA-II drivers I saw this behaviour with, but it is a feature of the filter to say to the driver that the frame was intended for it, thank you very much, even if this turned out not to be the case. This likely was a bug in the Envoy implementation and is, as far as I know, still present in Envoy 3.x. The Envoy filter should never ever consume broadcast ARP frames and it should never consume IP frames with unicast UDP traffic unless intended for port 376. Yet it does

I know this is only a recommendation, but it seems it should allow Envoy and a TCP stack to coexist, at least when using certain drivers, such as the A2065 one mentioned above and several of my own.

My theory is that the code was not tested as thoroughly as it should have been and the handling of the SANA-II filter by Envoy slipped by undetected. It may have been tested only within an isolated network which was used by the Amiga Networking Group.

[ncafferkey]

But I don't see anything in the SANA-II docs to say that a driver shouldn't also offer a frame to other stacks if one stack's filter hook accepts the frame. So to me it seems more like a bug in the driver if Envoy interferes with another stack in this way, as it might be a valid case for multiple stacks to be interested in the same frame, and my interpretation of the docs is that a TRUE result from a stack's filter hook just means "yes, I'll take a copy of that", not "hands off, it's mine"

It might be interesting to do a test with 3c589.device, Envoy and Roadshow to see if they get on well together, as I think they should.

[kolla]

commodities can give you something similar for GUI based applets

Commodities don't necessarily have GUI, what makes you think so?
AutoPoint, ClickToFront and NoCapsLock for example, don 't.

what's the use case? here's one - every time you want to use an external rexxport, you need to check if the port exists, and if it doesn't, run the host for that port, if you even know where to find it.

Not sure what you mean with "external rexxport", but if you mean an arexx port of a program, and your complaint is that you need to run the program to make the port available... yes, is that hard? What I hear is that you want the system itself to detect that something is trying to open an arbitrarily named arexx port, and then launch the software that's somehow is registered with this arbitrary port name... correct? The question becomes, how to detect what port is attempted, and how to ensure the right software is started so that something can answer on the port correctly before the client gives up. In principle you can have a script that listens to all the ports you have set up in a "servers" like file, and when something connects, it will shut down its port and launch the software so it can open the real port... however, the client may have become unhappy at this point, because of the port "hand over", or simply because launching the software may take too long.

Or you want to open on a named PUBSCREEN. But you're first to arrive, and the screen isn't open yet.

Well, there's a (badom-tish) commodity for that - MUI:PSI, it can do just this, and I am sure there are other pub screen handlers that do the same. Should the OS come with a pubscreen commodity handler built in? In my view - Absolutely! But certain OS developers didn't think it's worth wasting time on, as there are already third party tools for this.