NComm and XModem / XModem-1K protocols

[CodePoet]

I'm sure the title of this post brought back some fond memories for other Amigans

To the gurus out there: I need a little help with the XModem / XModem-1K protocols (the latter being a simple extension with a larger packet size) - while not directly related to the Amiga itself, I still go overkill and run NComm under UAE for ALL my serial terminal work. I absolutely love it, and nothing windows or linux-based even comes close!

In a nutshell, I'm building some firmware upgrade jigs for Sony Bravia TVs. There's a microcontroller on my boards that simply reads firmware from one of two on-board flashroms, and sends it to the Sony DTAU boards using the XModem-1K protocol via a 16-18Pin FFC cable - the protocol is non-negotiable unfortunately, most of the branded TVs I've worked on use XModem-1K, my guess is for simplicity and ease of implementation.

Now, the XModem-1K protocol has a fixed packet length of 1028 bytes (4 bytes are headers/footers, and 1024 bytes /1KiB of file data) - if the file data should contain less than 1024 bytes, then it needs to be padded up with garbage. There's no implementation to send the file name or size over via XModem, so the receiving end relies on an EOT character (0x05) to determine when transmission has ended. Note that EOT/0x05 is sent alone AFTER all the 1028 bytes are transmitted, so the padded garbage is still present; you cant simply abandon transmission when thee file has finished streaming.

My question is, how is the receiving end supposed to determine where the file ends and garbage starts? One can't guarantee that the garbage characters aren't repeated elsewhere in the file, or even at the end of the file. Hyperterminal for example, sends 0x1A as a padding character according to Microschlong. While this appears to work, it's driving me nuts as to how the receiving end would filter it out without accidentally adding to or subtracting from the filesize.

When sending my own file data using XModem via NComm to another device, I can ensure the file is rounded off to the nearest 1K and use a known padding character that's easily identifiable. Unfortunately I can't do this with user provided files...

Any clues?

[Zac67]

Have you checked what's available on the web?
http://en.wikipedia.org/wiki/Xmodem
http://www.textfiles.com/programming/ymodem.txt

I remember XModem being a MAJOR pain back in the BBS days. I avoided it at all cost and was glad when ZModem became popular.

[CodePoet]

@Zac67

Haha of course I have, but thanks for the suggestion. The Wikipedia article you posted mentions the use of a mutual padding character that "can be identified and discarded", however if the character changes in one implementation of XModem, there's really no way to notify the receiving end - which is why I gave up and asked here, as Amigans seem closely tied to hardware.

The other issue is when sending binary files, you don't really know if any of the "reserved" or control characters are used elsewhere in the file - this is especially true of dms files we pirated via Xmodem and Ncomm back in the day

If I chose HyperTerminals implementation and used 0x1A as padding, what happens if the last byte of a file I transmit is also 0x1A? It ends up being one byte shorter...

CURSE YOU SONY!

[Castellen]

My question is, how is the receiving end supposed to determine where the file ends and garbage starts?

I always used Term over NComm, and still use it often for embedded software development from the Amiga.  But whatever floats your boat

I'm not at all familiar with the Xmodem protocol, but here's a few ideas.

Most protocols that have a fixed or dynamic length packet size but contain variable length data will have some kind of length field in the packet header.  It's possibly the case in your situation, as the data you're sending in the packet can contain any random data from 0x00 to 0xFF, so you can't rely on an EOF detection character.

As an example, I just wrote an application to communicate with Afga serial digital cameras.  Here's a simplified look at it's protocol:
= 1 byte
= 2 bytes
= 1 to 2048 bytes
= 2 bytes

So if you look at the first few bytes of each Xmodem packet, you'll possibly find something similar.  You should see 2 bytes which don't change in value until the very last packet which doesn't contain 1024 bytes.

e.g. the first two bytes might indicate the length of data to follow, so with 1024 (0x400) file bytes in the packet, you might see the first two bytes as 0x00, 0x04 or they may be in the opposite order depending on the word endianess.

The last two bytes might be a 2 byte checksum.  Perhaps the INT16 sum of each of the 1024 data bytes in the packet?

To look at the serial data in hex form (not ASCII), it's incredibly useful to use Term 4.8 and put it into hex mode.
Settings menu > Terminal > Emulation tab > Select Hex
No idea if NComm has this feature or not.

To 'sniff' the RS232 communications in a working transfer, just connect ground and RXD only of the Amiga serial port to TXD of whatever is sending the Xmodem data.
On the 25 pin port, pin 3 = RXD, 7 = GND
On 9-way ports, pin 2 = RXD, 5 = GND

Should be easy enough to work out.