GeneralSaver (was: Access to compiler's variable types possible ?)

[Jose]

Just a crazy idea I had to make an abstract piece of code that would take any variable (including arrays and linked lists) and save it to disk. To do this one would need to know what a variable's type is in the 1st place. For example to write a struct like:
 

Code: [Select]

struct MaVar
{ char *Name;
  int a;
}
one would need to know that MaVar.Name is actually an array so that we save the actuall array and not the pointer to it.
Possible ? 8-)

[Jose]

@Cymric
It's a start:) I guess one would have to store what the actuall types are in the HD anyway, to be able to load and parse stuff later, BUT the big advantage here is that it wouldn't be necessary to make custom structures for every variable one want's to save.
BTW aren't the hunks debuggers use written down after the code is actually compiled ? That would rule out that option.

[Jose]

Hey.
"If you try to access, for example, an integer as though it were a string, would you not receive an error code?"
In C I think it depends on the compiler settings but normally you'll only get a warning. It's a "feature" of C that allows some code to be faster but can be disastrous if you mess up.

" Couldn't you capture the error and use it to determine whether or not the variable can be accessed in that way? For example, say you have an integer variable containing the number "1". You try to append the string "test" to it, and receive an error. You therefore know that it's not a string variable and move on to the next variable type: integer. You attempt to increase the variable by one. This is successful, so you know it's an integer."
No, cause in C it only get's a warning and even if it did get an error your code won't know anything about it (even the C preprocessor stuff AFAIK), the compiler just stops and prints an error message to the screen.

[Jose]

@Cymric
I've put down an example, I think it clarifies what I wanted.
I'm writing the programs myself so I know how the structs I want to save to the HD are organized. Put simple I just wanted to avoid making a saver/loader for each and do a general piece of code. If it's impossible to get that info from the compiler then then only way I see it would be to put it in the structs themselves. For example the variable I used above could become:

Code: [Select]


/* Possible types */
#define 1 PTR /* A pointer */
#dfine 2 NONPTR /* Non pointer. In this case the var's 1st
field is a WORD with it's size. IMPORTANT: A sequence of non
pointer variables would only need one Type and Size
definition in the beginning */

struct MaVar
{ WORD Type; /* Allways initialized to 1 */
  char *Name; /* See note bellow */
  WORD Type; / Allways initialized to 2 */
  WORD Size; /* Allways initialized to sizeof (int) */
  int a;
};
/* IMPORTANT: to keep it consistent the 1st 2 bytes in the
MaVar.Name string would have to be a WORD defining the
variable is not a pointer (a 2). The next 2 bytes (3 and 4)
would make up a WORD with the strings size. Then the actual
strings starts. */


So like this the code would then save things to HD with the instructions given in the structs themselves. Like I said I wanted to do this without adding anything to the variables so the usefullness of this would depend on if it's more troublesome or not than the "conventional" custom saver/loader way.

What do you think ? :idea:

[Jose]

@Karlos

Hmm, hadn't thought about portability between platforms, that would screw things up. But if the code was to run on 68k or PPC I guess there would be no problem. Even if it was to run on x86 it should work provided the user used files saved in the same platform (not ideal).
Regarding the alignment thing, didn't thought about that either :-) But I thought alignment of structures was made by padding bytes at the end or not ? If so, my method above would still work, though if there is a better one bring it on.

As to the (un)serialization functions that's cool but it sort of defeats the whole purpose I think,  wich is to minimize the needed work to save data to disk (e.g. if I have to write a serialization function for each struct it will be more or less the same trouble as writtin a custom loader/saver for each).

BTW I liked Sigler's idea of having a separate description outside of each struct, that makes it unecessary to repeat it inside each one, though a pointer to it is still needed of course.

[Jose]

@Cymric
" Not a bad idea, but it needs extension: you must store that type string in the dump too, and that requires some indication as to its length..."
Well, if one establishes a format the loader already knows what data it's dealing with, unless we're to make a general loader too :-)

"... Second, you need to be able to distinguish between various structs (and unions)"
Each struct begins with a pointer to it's description.

[Jose]

@sigler
Thanks for taking the time but I never bothered with C++ :-) Maybe one of these days...

@All

I've been thinking about the better way to implement a description of data and came up with the framework bellow. But I'd still like your suggestions/critics on it before setting down and try to write the code that uses it.

Another important doubt is that a string would apparently be better cause unecessary fields for some options could simply be omitted thus making the thing simpler.

But at the same time if I'm to use nested data descriptions how am I to put a pointer in them if they're inside the string ? Even if it's possible it probably won't be very practical. Another problem of strings is that they're numbers have to be parsed, wich slows down things. I could parse them on startup but that would be a bit weird..

Here it is: :-D

Code: [Select]

/* Description of a block of data in memory,
description of a struct is a zero terminated array
(Dscrpt->Type == 0) of these */
struct Dscrpt
{ WORD Type;
  UWORD Size; /* Size of block of variables with this Type,
or variable lenght block definition (see bellow) */
  LONG AddtInfo;
};
/* Some considerations about struct Dscrpt */
/* - At 1st it was to be a one value approach but it's
changed for the following reasons:
     - The size field allows descriptions to comprise very
big sequence of variables with similar attributes
       so it effectively renders descriptions smaller in
most cases
     - The AddInfo field was added to allow for usage of
pointers in description nesting, wich couldn't be
       done any other way, but it now also provides other
features
   - All in all, this is probably one of the (or just the ?
;)) best possible compromises between size and simplicity
versus features, powerfullness and flexibility
*/

/* Type definition field */
#define TYPEDEFFLD 0x0000ffff
/* Type extension field */
#define TYPEFLGFLD 0xffff0000
/* Possible values in Type. If not mentioned, Size and
AddtInfo do not apply (are ignored) */
#define NS 0 /* Do not save following block of data */
#define NP 1 /* Block of non pointer variables */

/* Block of pointer variables. Indirection level given by c
*/
#define PO(c) 2|(c<<8) /* Block of pointer variables (used
if != NULL), whose object Dscrpt is at the object pointed to
+ offset given in AddtInfo */
#define PP(c) 3|(c<<8) /* Block of pointer variables (used
if != NULL), whose object Dscrpt is pointed to by AddtInfo
*/

/* Block of structs */
#define SO 4 /* Struct inside this one whose Dscrpt is at
it's 1st element + offset given in AddtInfo */
#define SP 5 /* Struct inside this one whose Dscrpt is
pointed to by AddtInfo */

/** Block of arrays (fixed or variable lenght) **/
/* IMPORTANT: For these upper 8 bytes in Size == variable's
block size, lower 24 bytes == extra info (see type
descriptions) */
/*      Following macro must be used to define Size
(a- this variable's block size, b- extra info): */
#define SZ(a, b) (a<<24)|b

/* Fixed lenght arrays. Nr. of elements given in b */

#define AO 6 /* Array's base type description pointed to by
AddtInfo */
#define AP 7 /* Array's base type description pointed to by
each element at the offset given in AddtInfo */

/* Variable lenght arrays */
#define AMO 8 /* Size memo based. Same as AO but array's
size is given by value (size memo) in address of 1st element
+ offset in b */
#define AMP 9 /* Size memo based. Same as AP but &quot;&quot; */
#define AZO(c) 10|(c<<8) /* Zero terminated. Same as AO but
array is zero terminated by type given in c (type
definitions bellow). If base type is complex, b is offset to
the terminating field on each element */
#define AZP(c) 11|(c<<8) /* Zero terminated. Same as AP but
&quot;&quot; */
  /* Possible types in c */
  #define 1: BT /* BYTE */
  #define 2: WR /* WORD */
  #define 3: LN /* LONG */
  #define 4: FL /* float */
  #define 5: DB /* double */


[Jose]

These are the features I tried to keep in mind (my notes before I made the actuall "standard" sketch.

- Fast and relatively easy to use:
  - No need to type too many or fancy time consuming multiflaged masked definitions. When needed, last ones are done by a macro
  - Descriptions are written only once, then refered using pointers
- Flexible:
  - Support external or internal (pointed to inside the data itself) data block descriptions in any place, to best fit any particular needs
  - Description nesting (support for nested complex variables)
  - Support for multi indirection pointers
  - 2 ways to support variable lenght arrays: size memo and zero terminated
  - Support for multidimensional arrays (using nested descriptions, see provided example)
  - Support for multiple unpredictable datatype trees and arrays using internal descriptions
- General, practically any complex/variable lenght data type combinations should be described

[Jose]

A drawback of this is that a simple NULL terminated string description would be:
struct Dscrpt NullTermStr_Desc[] = {AZO(1), 1, 0};

Maybe trying to make it powerfull and with features, it ended up too complicated/counterproductive :-?  :-(
If there's a big sequence of strings in a struct say 20, I think it rules though, it would simple become {AZO(1), 20, 0}; :-D

A string based description could be more simple for simple pieces of data but due to what I said above, I don't know if it would be adequate for these features.
If you have a good idea to implement these features using a string let me know ...

Waiting for comments...
.....

 8-)

[Jose]

Hi there:-D. I've made the saver part of the code and it's compiling well for about a week now. I'm now correcting the remaining bugs that make the file save incorrectly. Most types are saving well though, but I've stumbled into something that I hope doesn't make the whole thing useless: Arrays (inside structs or not). It seems that sasc (will try with VBCC at the end cause I need the sasc debugger for now) saves fixed size arrays inside the struct itself. Can I assume this to be the standard or are there some compiler/compiler settings that just add the array's pointer and keep the elements in other place ? If the answer is yes then we're screwed, it won't be "general saver" anymore.

Finally, if this turns out to work, do you guys think it's worth it to make a library of it ?

:pint: