GeneralSaver (was: Access to compiler's variable types possible ?)

[Jose]

Just a crazy idea I had to make an abstract piece of code that would take any variable (including arrays and linked lists) and save it to disk. To do this one would need to know what a variable's type is in the 1st place. For example to write a struct like:
 

Code: [Select]

struct MaVar
{ char *Name;
  int a;
}
one would need to know that MaVar.Name is actually an array so that we save the actuall array and not the pointer to it.
Possible ? 8-)

[Cymric]

Not within standard ANSI-C, and not without severe hacking of the C++ runtime library by the looks of things. I have a strong hunch that that various debugging hunks written out by the compiler can be parsed to derive the information you're after: good debuggers allow you to look at variables and the like, and for that they must know the type---which is what you want to know. That's as far as my knowledge can take you, unfortunately.

[Jose]

@Cymric
It's a start:) I guess one would have to store what the actuall types are in the HD anyway, to be able to load and parse stuff later, BUT the big advantage here is that it wouldn't be necessary to make custom structures for every variable one want's to save.
BTW aren't the hunks debuggers use written down after the code is actually compiled ? That would rule out that option.

[motorollin]

If you try to access, for example, an integer as though it were a string, would you not receive an error code? Couldn't you capture the error and use it to determine whether or not the variable can be accessed in that way? For example, say you have an integer variable containing the number "1". You try to append the string "test" to it, and receive an error. You therefore know that it's not a string variable and move on to the next variable type: integer. You attempt to increase the variable by one. This is successful, so you know it's an integer.

I don't know anything about C so please tell me if this is gibberish :-)

--
moto

[Jose]

Hey.
"If you try to access, for example, an integer as though it were a string, would you not receive an error code?"
In C I think it depends on the compiler settings but normally you'll only get a warning. It's a "feature" of C that allows some code to be faster but can be disastrous if you mess up.

" Couldn't you capture the error and use it to determine whether or not the variable can be accessed in that way? For example, say you have an integer variable containing the number "1". You try to append the string "test" to it, and receive an error. You therefore know that it's not a string variable and move on to the next variable type: integer. You attempt to increase the variable by one. This is successful, so you know it's an integer."
No, cause in C it only get's a warning and even if it did get an error your code won't know anything about it (even the C preprocessor stuff AFAIK), the compiler just stops and prints an error message to the screen.

[Cymric]

BTW aren't the hunks debuggers use written down after the code is actually compiled ? That would rule out that option.

Well, it depends on what you were after, which wasn't very clear from my point of view :-). If you write all programs yourself, then you know the type of variable at compile time---there's no need to ask the program or the compiler. If you want you can abstract away that information so that the program itself can determine at runtime the type of objects it's dealing with (in essence providing the rudimentaries of  a runtime library found in C++ or ObjectiveC), but this would be a pain in programming, as everything is a struct or pointer to a struct. Your code would be littered with #define's and the like to hide the fact that you're not accessing a variable directly, but through a struct.

In C, there is no magical recipe which dereferences a pointer and says 'oh, that's an integer', or 'oh, that's an union'. You can with some clever heuristics determine whether you are dealing with a pointer, but those tricks rely heavily on the machine's architecture. And whether it is a pointer or not is not of much use unless you are into automated memory management.

If you haven't written the program, then what you're after is much, much harder, and relies on the presence of those debugging hunks, or a so-called cooperative language which stores that information in the executable. Java is an example of such a language.

[SamuraiCrow]

In Python there is an add-on known as "pickle" that will serialize any non-cyclic linked structure into a file.

[Jose]

@Cymric
I've put down an example, I think it clarifies what I wanted.
I'm writing the programs myself so I know how the structs I want to save to the HD are organized. Put simple I just wanted to avoid making a saver/loader for each and do a general piece of code. If it's impossible to get that info from the compiler then then only way I see it would be to put it in the structs themselves. For example the variable I used above could become:

Code: [Select]


/* Possible types */
#define 1 PTR /* A pointer */
#dfine 2 NONPTR /* Non pointer. In this case the var's 1st
field is a WORD with it's size. IMPORTANT: A sequence of non
pointer variables would only need one Type and Size
definition in the beginning */

struct MaVar
{ WORD Type; /* Allways initialized to 1 */
  char *Name; /* See note bellow */
  WORD Type; / Allways initialized to 2 */
  WORD Size; /* Allways initialized to sizeof (int) */
  int a;
};
/* IMPORTANT: to keep it consistent the 1st 2 bytes in the
MaVar.Name string would have to be a WORD defining the
variable is not a pointer (a 2). The next 2 bytes (3 and 4)
would make up a WORD with the strings size. Then the actual
strings starts. */


So like this the code would then save things to HD with the instructions given in the structs themselves. Like I said I wanted to do this without adding anything to the variables so the usefullness of this would depend on if it's more troublesome or not than the "conventional" custom saver/loader way.

What do you think ? :idea:

[Cymric]

It's probably the only way to do it, and not much more work than a 'conventional' custom loader / saver. This does pretty much the same thing, but then dumps the variables straight to non-volatile memory, without bothering with type information and what not.

Generalisation can be cumbersome depending on whether you want to be able to simply 'plug & go'. With unions you can code this quite elegantly; with pointers you can 'shadow' variables throughout the program with but a single assignment. Nevertheless, if you want to be able to dump N-dimensional arrays, strings, and complex structs, things can become tricky. You will need to specify some sort of loader format too: is it write-as-you-ask, or will you group all strings near the end of the file; will you provide error checking or not; etcetera.

However, you cannot make it completely general, as structs are not completely general. You will need some program code to deal with the specific nature of the structs you're dumping, although the building blocks they are made up of (ints, chars, pointers, floats, ...) are general.

In my experience though, such little blocks of code are extremely useful and reusable if you've coded them as general and robust as you can. You might spend a week getting it to work, but the effort is worth it. I created a small set of routines to set all kinds of parameters in the program in a standardised way (complete with input checking against preset bounds) and various function hooks to do some pre- or postprocessing, and it has worked marvellously in any program I've linked it to. Write down the question, provide some glue code to link the input values to program parameters, and forget about the rest.

[sigler]

Damn, I just wrote a lengthy post which I lost, here's the short version:

struct mystruct
{
char* typeinfo;
short x;
float y;
char* name;
mystruct* linkto;
};

s = short
f = float
p = pointer to another struct
P = pointer to nullterminated string

char* mystruct_typeinfo = "sfPp";


void main()
{
struct mystruct* instance = (struct mystruct*)malloc(sizeof(struct mystruct);
instance->typeinfo = mystruct_typeinfo;
}

--
Sigurd Lerstad

[Cymric]

Not a bad idea, but it needs extension: you must store that type string in the dump too, and that requires some indication as to its length. Second, you need to be able to distinguish between various structs (and unions). That's what makes such a state-dumping routine somewhat less-than-easy to write, although it is atill all perfectly straightforward C, of course.

[Karlos]

Don't forget that the size of each primitive will vary from architecture to architecture, as will alignment requirements and the endian representation of multibyte data, if platform portability is important.

In short, never rely on simply dumping a struct as binary data even if you are careful when handling pointer data. Even changing the compiler settings on the same target could cause changes in alignment which in turn could seriously alter the topology of your structure.

A much better approach, IMHO for plain C is to add a function pointer to your struct that points to the appropriate serialization functions to load/save that structure to an open filestream. A factory function that creates the structure would ensure that this function pointer is set:

Code: [Select]



typedef Foo struct {
  int someInteger;
  float someFloat;
  char someFixedCharData[8];
  Foo* next;
  Foo* prev;

  /* serialisation handlers, point to proper function at instansiation */
  int (*serialize)(const Foo* foo, FILE* fp);
  int (*unserialize) (Foo* foo, FILE* fp);

} Foo_t;


/* implementation detail */

int serializeFooInstance(const Foo* foo, FILE *fp)
{
/* serialization code specific to Foo here */
}

int unserializeFooInstance(Foo* foo, FILE *fp)
{
/* unserialization code specific to Foo here */
}

Foo* makeFoo(void)
{
  Foo* foo = (Foo*) malloc(sizeof(Foo));
  if (foo) {
    /* initialise all the members to safe defaults... */

    foo->serialize = serializeFooInstance;
    foo->unserialize = unserializeFooInstance;
    return foo;
  }
  return NULL;
}




You can then do a poor man's method call with the struct, rather like this:

Code: [Select]


FILE* fp;
Foo* myFoo;

....

myFoo = makeFoo();

...

if (fp = fopen(......)) {
  myFoo->serialise(myFoo, fp);
}


and so on.

The main point about this is that each structure type knows how to un/serialize an instance of itself.

Of course, if you follow this idea to it's logical conclusion, you'll ditch C and start C++ which gives you much more elegant ways of doing it ;-)

[Karlos]

I should further add that in all non-trivial cases, the physical cost of perfoming IO will be the performance limiting case, not the overhead in calling code to perform the appropriate serialisation of data.

[Cymric]

*Ponders a little*. No, I don't think that switching to C++ is very useful. Of course, the OO-ness does provide a nice, consistent interface (which is a Good Thing, by the way), but it still doesn't easily tell you what variables need to be serialised---and that's what this discussion is all about. With C++ all classes still would need their own serialisation function, and those would still need manual updates if the variable structure of that class changes. (Thus rendering the entire class / inheritance idea more or less superfluous again.) The main problem Jose is facing is that there is no keyword (not in C, not in C++) which gives you back some structure outlining the variables and their type in a given scope.

[Jose]

@Karlos

Hmm, hadn't thought about portability between platforms, that would screw things up. But if the code was to run on 68k or PPC I guess there would be no problem. Even if it was to run on x86 it should work provided the user used files saved in the same platform (not ideal).
Regarding the alignment thing, didn't thought about that either :-) But I thought alignment of structures was made by padding bytes at the end or not ? If so, my method above would still work, though if there is a better one bring it on.

As to the (un)serialization functions that's cool but it sort of defeats the whole purpose I think,  wich is to minimize the needed work to save data to disk (e.g. if I have to write a serialization function for each struct it will be more or less the same trouble as writtin a custom loader/saver for each).

BTW I liked Sigler's idea of having a separate description outside of each struct, that makes it unecessary to repeat it inside each one, though a pointer to it is still needed of course.