CF/SD research and troubleshooting

[fgh]

I posted this over at EAB and amibay, thought I might as well post it here

There are many misunderstandings concerning removable media around. I thought I’d share what info I’ve come across..
(I learn as I go, so please let me know if you find any mistakes!)

General information

I have bought a brand new CF or SD to use with the amiga IDE, what's next?   To set up a SD or CF card as an amiga system drive you need to do a little research. I will not detail the procedure here.
Max card capacity for IDE without software patches is 4GB, and 8GB with direct-scsi mode (with PFS3 file system).
For PCMCIA and fat95 there is no limit on card capacity.


WinUAE considerations   Windows does not allow applications (WinUAE) to overwrite the MBR of active windows partitions.
So in order to write amiga partitions to the card, you need to remove all windows (MBR) partitions from the card first.

• In Vista or Win 7, the diskpart commandline utility supports removable media (use 'clean' command).
  
• For Win XP you need a tool like Partition Wizard or EASUS Partition Master.
• On Linux you can use the dd command
• On Mac OS use disk utility (Partition -> 1 partition -> Free space)

For accessing amiga flash media in WinUAE you can select either IDE0 or UAE hd-controller.
If you select IDE0, WinUAE will use scsi.device IDE driver from the kickstart file.
This way you won't need to change any HDToolbox tooltypes, and you might spot some errors already in WinUAE.
If you are done prepping the card and want faster file transfers, you can use 'UAE'.
(Using UAE hd-controller HDToolbox can still access the card but you need to change hdtoolbox's SCSI_DEVICE_NAME tooltype to uaehf.device.)


Max transfer and mask

• When using onboard IDE and scsi.device, a maxtransfer setting of 0x1fe00 (or lower) is required on all partitions with any file system (to avoid read- and write-errors).
     Be aware that wrong (too high) values are mentioned too many times on EAB.
  
• Specific max transfer settings are not required if you use IDEfix97 on internal IDE or when using PCMCIA
• I do not know which drivers for third party IDE controllers are affected. It is likely that drivers made before 1994 (before EIDE, and later ATA-2 in 1996) are affected.
• The maxtransfer setting limits the max size of one transfer. It has no effect on speed. (Even battlefield 3 uses 128kB and less :) )
  
• The reason for the max transfer issue has recently been found, see this thread.  (Also due to changing ATA specs)
  
• Mask settings are for DMA capable controllers and not required for Amiga IDE or PCMCIA use.

Removable vs fixed flash media (short version)   Original scsi.device IDE drivers works well with both fixed and removable type CF cards.
The Amiga PCMCIA driver compactflash.device also works with removable type cards.
Some CF cards do not work with compactflash.device, but this is for other reasons.

IDE drivers confirmed NOT working with removable type CF cards:

• IDEfix (Elaborate Bytes)
• Buddha (Elaborate Bytes/Individual Computers)
• XSurf (Elaborate Bytes/Individual Computers)
• Catweasel (Elaborate Bytes/Individual Computers)

Doobrey has made patches for IDEfix and the other Elaborate Bytes IDE drivers, that correctly identify CF cards, and allow removable type cards to work correctly.

SD cards are neither removable or fixed in themselves, as they are not ATA devices.
It's therefore up to the IDE-SD adapter which response is given.
All adapters I have tried report as removable, and consequently the drivers above will also fail with SD cards in those adapters.

See 'reasons for incompatibility' below for more information..


HDinstTool vs HDToolbox   Never use HDinstTool with a drive that was partitioned using HDToolbox if you want to keep any data on it.
HDinstTool has a nasty bug that will destroy partitions that are not defined and saved by HDinstTool originally.
Having said that, HDinstTool works better than original HDToolbox with drives >4gb.


Miscellaneous

• Any harddrive must be set up, and bootable partitions created, before they'll show up in the early startup screen.
• PCMCIA is hot swappable, IDE is not
• There is (physically) no way of low level formatting flash media.

You should not even do a normal full format with flash media, as it writes to all the blocks of the card, marking them as 'used' to the cards firmware.
This gives the flash media's internal wear-leveling less headroom, potentially reducing the speed and lifetime of the card.

• Always use quick format on any partition above the first 4GB of any drive.

The AmigaOS format function is not able to access >4GB and will corrupt data and partitions.
[/LIST]


Removable media troubleshooting

Here is a small list of common problems and solutions that might work.
It does seem like lots of cards are simply not compatible, but the hardware is not to blame in all cases.

CF or SD does not work in PCMCIA adapter   First - you need to install compactflash.device (pcmcia CF driver) and Fat95 (file system).
Use the latest versions as compatibility increased over the years.
For automount, move CF0 from Storage/Dosdrivers to Devs/Dosdrivers.

Your amiga can access cards in FAT (windows) format or in amiga formats through the PCMCIA.
It is most common to use FAT, so your PC can access the card as well.
The CF0 mountlist specifies the file system. If CF0 is set up with Fat95 (default), you can read a FAT16/32 card.
If you want to read a card with amiga filesystem, you need to change filesystem in the CF0 mountlist PLUS define the size etc of the card.
(FAT95 does this automagically.) You may use Giggledisk on aminet to create the mountlists.

FAT-formatted card (accessible on pc) shows up as CF0:NDOS in PCMCIA adapter

New cards come formatted either as superfloppy (without MBR partition table) or with a MBR partition table with one partition.
It seems that Fat95 accepts both types but not all variants.
Specifically FAT12 format seems to show up as NDOS, but there may be others as well. Let me know if you find out
Anyway, quick-formatting the card on the amiga should be solve it.

There were reports of cards not showing up at all before recreating their MBR, but I have been unable to confirm it.

Other possible problems

If your amiga has kickstart 3.1 and a memory expansion or accelerator that can take maximum 8MB RAM, any fast RAM above 4MB will conflict with PCMCIA address space and PCMCIA will not work reliably.
Most cards with this issue have a jumper to limit RAM to 4MB to avoid the conflict. Use it.

A1200 also has a PCMCIA bug that required reinserting the card after a reboot.
CFD v1.21 (2003) and newer fixes this bug. (As does the cardreset software). A600 does not have this bug.

People often suggest installing Cardpatch when troubleshooting CF over PCMCIA.
I don’t know if it is required for storage media, but it will not harm. (Applies to both a600 and a1200)


Only my first FAT partition partition shows up   FAT95 does not automatically mount multiple partitions.
You need to create seperate mountlists for these. (See giggledisk on aminet)


SDHC card does not work in PCMCIA SD- or multi-adapter   So far I have not found a single PCMCIA adapter that supports SDHC cards (4-32GB).
All the SDHC compatible adapters I have found are 32bit Cardbus, not 16bit PCMCIA.
Look specifically for a 16bit PCMCIA adapter supporting SDHC (SD High Capacity), and let me know when you find one!
For SD cards (2GB and below) there are still cheap PCMCIA adapters around, but expect them to become increasingly rare.


My SD card does not even work on my pc anymore   Sometimes SD cards gets screwed up, and the various operating systems might not be able to recover them with a normal format.
The SD Card Association has made a formatting tool that is reported to save many lost cards. Try it!


My CF or SD works in WinUAE but not in my real Amiga (Onboard IDE)   If it does not boot, launch hdtoolbox (boot with workbench install disk or any other wb installation).
Make sure that hdtoolbox is using the original scsi.device (SCSI_DEVICE_NAME tooltype) and that you are not running IDEfix without patches or other IDE drivers.
If it shows up, the problem is often software related, and you might have to check your favourite hd prepping tutorial.
If it does not show up in hdtoolbox, or does not work reliably, It might be one of those cards that are simply not compatible with the old drivers, and I have no suggestions at this point. (Suggestions please!)

[fgh]

Hdtoolbox says 'Unit is not a disk (Type 7)'   Your card is mistaken for an optical removable media (CDR, etc) by hdtoolbox.
As far as I know this occurs with all cards that report as removable media (848A), and only when you go to 'change drive type' and select 'read configuration'.
If the important values (Cylinders, Heads, Blocks, etc) are filled out, you may ignore the message, fill out your own manufacturer and model info and proceed to partition the card.
(Unpatched hdtoolbox will show negative capacity for large drives (>4gb?). Ignore it, don't worry unless the partition page shows the wrong size.)
Failing that, there are patches for hdtoolbox available to fix this, and HDInstTool will also work. (Read the notes above about HDInstTool first).
Also, if you use WinUAE to prepare the cards you should not have this issue (even if you mount the CF using IDE0).



Reasons for incompatibility

Background   The ATA standard supports lots of different types of media, both fixed and removable types.
IDE drivers communicate with devices using the ATA protocol, and require devices to identify as removable media or not.
For the CF card itself this type info does not change anything, but it has to give an answer.
Many CF cards are set to report as removable media, but not all, and it's hard to know in advance. ('Industrial' level cards are usually fixed type)
The information is stored on CF card's CIS (Central information structure) and communicated by the ATA controller inside the card when replying to the ATA IDENTIFY DEVICE command.


The problem with certain amiga IDE drivers and 'removable' media   As mentioned above, original scsi.device IDE drivers have no problem with removable type CF cards or SD cards.
However, drivers by Elaborate Bytes - idefix97 and individual computers product drivers - do not work with removable type devices.

The drivers do not comply with the ATA-4 (or newer) specification.
IDE drivers use the ATA command IDENTIFY DEVICE to get all kinds of information from the device.

The first word of the device reply (word 0, 'General configuration') is 848Ah for removable type CF cards, and 044Ah or 0040h for fixed type CF cards.



First, bit 15 set in the reply (cards reporting 848A) can cause drivers to fail.
Drivers not written in accordance to ATA-4 (or newer) do not expect bit 15 set for ATA devices.

• ATA-4 standard ( 1998 ) says that bit 15 in word 0 shall be 0 for ATA devices (page 95), but can be 1 (848Ah) for CF compatible cards (page 52).

It also says that ATAPI devices shall not respond to IDENTIFY DEVICE, but reply to IDENTIFY PACKET DEVICE instead (page 41)

• ATA-3 standard ( 1997 ) says it must be 0 for ATA devices and 1 for ATAPI devices (page 49)
  
• ATA-2 standard ( 1996 ) says it is 'reserved for non-magnetic devices' (page 37)
• ATA-1 standard ( 1994 ) says it is 'reserved for non-magnetic devices' (page 44)
• CF specification v3.0 ( 2004 ) says word 0 standard is 848Ah, but can alternatively be 044Ah or 0040h to "turn off removable media". ( page 130 )

Second, bit 7 set indicates that a device is removable media. This did not change since the original ATA-1 specification.
(Additionally bit 6 could be set to specifically mark a disk as fixed, but this was removed in ATA-6 standard (2002) so I guess modern drivers use bit 7.)

See attachments below
Attached are ATA specifications 1-5, CF spec 3, and the entire IDENTIFY DEVICE replies from my four CF cards and one SD card.

Amiga IDE drivers that fail with removable type CF cards probably expect bit 15 to be 0 for ATA devices, and therefore either give up on the device, or regard it as an ATAPI device and fail.
It's confirmed by Doobrey that Elaborate Bytes drivers simply stop processing the device if bit 15 is set in the reply.
It's possible that other drivers might accept bit 15 set but still fail with removable type CF cards because bit 7 is set.

A CF card's reply can not be changed manually except on a few old cards where the manufacturer (Sandisk, Lexar) provided tools to ‘flip the removable bit’.
However, I've read that some modern CF cards sense if they are connected to a proper IDE bus, and change their type to fixed.
My Transcend 4GB 133x, Kingston 4GB (flower img) and Kingston 16GB 266x cards report as fixed (word 0 = 044A)

The ATA Identify Device replies can be read with the BusTrace program.

J.Schönfeld made the TrueIDE dual CF-IDE adapter to work around the driver problem.
This adapter modifies the CF cards IDENTIFY DEVICE reply to allow removable type cards on drivers that only work with fixed CF cards.


Regarding PCMCIA operation and incompatibility
   Compact Flash storage devices operate in three modes:

• PC Card Memory Mode
• PC Card I/O Mode
• True IDE Mode

Compact Flash devices must support operation in all three modes, but operate only in a single mode at any given time.
The CF card senses (using pin #9) if it's connected to IDE or PCMCIA, and switches modes accordingly.
Connected to IDE, it uses True IDE mode, and becomes a hardware-level PATA IDE device.
Connected to PCMCIA, things are getting more complex, as it takes the role of both IDE controller and IDE device, using PC Card I/O mode.
CF cards needs to support several different PCMCIA access modes, different register locations, etc.

Some CF cards (Kingston 4Gb etc) work with IDE/scsi.device but not with pcmcia/compactflash.device.
This can theoretically be due to the card, the driver or the amiga pcmcia implementation.
It is confirmed (by T.Wilen) that some cards do not support all mandatory register locations, causing problems with amiga pcmcia operation.

Let me know if you have more information


Regarding SD cards   SD cards are not ATA devices and do not have integrated controllers like CF cards.
Therefore there are far less differences between SD cards than between CF cards.
For the same reason SD-IDE adapters need to be active adapters, not just passive like most CF-IDE or CF-PCMCIA adapters.
As far as I know SD-IDE adapter manufacturers face the same options for ATA identification as a CF manufacturer, so compatibility might vary between SD-IDE adapters in a similar way as it does between CF cards.

As I wanted to experiment with SD cards, I have successfully tested 12 different 8GB SDHC cards on my a600 (PFS3 direct-scsi and ROM resident scsi.device) with a £6 SD-IDE adapter.

The test went like this

• Put three partitions on each card (300mb | 4gb | The rest)
• Copy 35mb file to dh1 in winuae
• Copy file from dh1 to dh2 on a600
• Copy file from dh2 to PC in winuae
• Confirm that files are binary identical (Using 'Beyond Compare')

Result
All cards work perfectly on the a600 and the the files were binary identical.

Notes about IDE-SD adapters
When connected to a PC (over IDE), the adapters reports like a removable type CF card (word 0 = 848Ah).
Therefore drivers having problems with removable type CF cards will also have problems with such adapters.

I have not tested many SD adapters, but it is possible that adapters reporting as fixed devices exist.
After all this is not more complex, as the SD adapter needs to come up with the ATA response anyway.


Relevance of PCMCIA / CF card voltage ratings?   The PCMCIA specification allows 3,3 or 5 volt devices, and the amiga pcmcia implementation only works with 5v devices.
The ‘low voltage key’ (physical incompatibility) on 3.3v pcmcia devices prevent using a 3.3v device in an amiga.
But if a 3.3v only CF card existed, and was used in a generic PCMCIA adapter, this could be a problem…
However: According to this document on the Compact Flash Association, all CF cards are able to operate on both 3.3v and 5v operation.
So voltage rating is not relevant.





I started this table with compatibility information for the different IDE drivers.
There are lots of holes, but I will not have time to finish it myself, so please let me know if you have something to add!


That’s it for now, well done if you are still reading!
There certainly are incompatible cards, but I hope some will find their card is compatible after all with this little guide.


Attachments
Only tiny attachments allowed here. Have a look at amibay or EAB for ATA and CF specifications etc..


Topics to be revisited

• Drive and partition size limitations and solutions
• Using two devices (master/slave) on Gayle IDE

Additions and suggestions are welcome! Thanks :)

[fgh]

(Reserved)