Serialization / persistance library for C

[Jose]

About an year ago I posted a thread about making what's normally called serialization or persistance (I didn't even knew they called it that at that time)a more confortable thing, in C. After much thought, changes and huge stupid amounts of time coding macros(which if it was today I'm sure I wouldn't even start) I can say that the solution I found is not like what other guys proposed at all, but I think it's much, much better;) The bad news is that only the macros are now completed, code to process the data descriptions they make still has to be written or partially updated, since parts of previous version can be used. However that's more or less what are the syntax possibilities I've manages to achieve.

An incomplete early version of the manual can be found HERE

BTW, I also posted about this on utilitybase cause it's more dev oriented but so far there non gives a rat's ass about it...

Your comments are highly appreciated of course:)

[Trev]

To start, I think the documentation would benefit from a concise description of the problem you're trying to solve and how you propose to solve it--in plain language, of course. Perhaps a few examples of serialization common in other languages would be appropriate, as you're trying to fix a problem in C for which  many languages have intrinsic or at least commonly accepted solutions. As part of that, you might compare and contrast serialization and persistence as implemented in COM, Corba, or Java.

I have to agree with some of the past comments on the library: the names of the macros are quite confusing. If you must use abbreviations, I would use the accepted (or de facto standard) English abbreviations, e.g. "ser" for serial and serialize and "ptr" for pointer. That's still making an assumption about your audience, though. You may choose to use abbreviations accepted by the computer science community or some other group of programmers. In any case, after skimming the document, it's not at all clear to me what each macro does. Perhaps it should be? That could also indicate a lack of knowledge on my part, so that particular comment may not hold any weight. ;-)

Are the library and serialized objects portable? i.e. Can I serialize an object on a big-endian system and deserialize the output on a little-endian system without corrupting data? How about interoperability with other serialization schemes?

Have you looked at tpl <http://tpl.sourceforge.net/>? (I imagine it's pronounced tuple, but I can't confirm that.) Many of the concepts appear similar to yours. Perhaps your project could add value to tpl or vice versa.

Trev

[Jose]

@Trev
I'll try to update the doc to make it more readable, thanks for the tips...

A "def" is much like format string in tpl.

Regarding tpl. Must confess I did not do the extensive reasearch on other available solutions I should've done before losing so much time with this. After taking a look at tpl docs some of the differences that pop up are:
- My "defs" can be reused (i.e.) if you have already defined one for certain struct elements and later want to serialize the others you can define only the diference and reference the 1st def on the 2nd (or any others).
Mine supports getting the size of C basic types automatically (through sizeof()), which sometimes makes defs simpler or no need for defs at all, on tpl you seem to need to allways provide a format string with the right types.
Mine supports nested complex defs even when pointers are involved (i.e. array of pointers to array of pointers to structs which might have arrays inside them etc.)
Another strong point is beacuse of being able to follow pointers it shouldn't be dificult to use defs to deallocate memory too, I still haven't looked deep into this though.
I'm also sure tpl also has many advantages, maybe later we can do a better comparison, I need to sleep now..

Yes, it will be endian agnostic.

I've allways wondered how the printf function can compare the variables one passes to it with the the format string (which in a way seems to be kind of like tpl works) so that the compiler can issue warnings, I've allways thought this was embedded in the C language itself and not accessible to the user ?

Regarding open source, if I'm not allowed to make money with it I think I'll leave it (and this is something I'd like you guys opinion too), I currently need it, not that I'm a greedy person or anything, really.

:pint:

[Trev]

I've allways wondered how the printf function can compare the variables one passes to it with the the format string (which in a way seems to be kind of like tpl works) so that the compiler can issue warnings, I've allways thought this was embedded in the C language itself and not accessible to the user ?

The warnings are compiler-specific. For example, Microsoft exposes Visual C++'s checks in sal.h. See SAL Annotations <http://msdn2.microsoft.com/en-us/library/ms235402.aspx> for more information. (Actually, this is the first time I've bothered to check how Visual C++ does this.) Vbcc exposes similar funcitonality via the printflike pragma, and GCC exposes it via the printf attribute.

The implementation of stdarg.h is platform-specific.

tpl uses the BSD license, which means you can do whatever you want with the source code, including keeping changes to yourself, as long as you include the tpl copyright notice somewhere in your documentation. How you distribute your own software is, of course, entirely up to you. Personally, I think there are far too many closed source Amiga libraries. I find myself constantly reinventing the wheel because I don't want to trust a library on Aminet that was written 15 years ago by an author that's nowhere to be found. Source code would have allowed bug fixes. Apart from that, no one's going to make money selling development libraries for Amiga-like operating systems. There just isn't a market for them. :-(

Trev

[Jose]

@Trev

I took a look at the VBCC doc and saw how the printflike pragma works. I don't think that you can use this method without having to have a format string and then repeating the variable names (i.e. elements in a struct) all over again. Looking at the tpl doc there is a shorthand struct syntax that allows just passing the format string specifying the types to serialize and a pointer to the struct.
BUT: It also says this works by calculating what the padding will have to be for all the elements to be aligned. From what I've read around sometimes through the years this is compiler dependent, what if compiler A aligns to 16 bit and compiler B to 32 bit ?
Worse what happends when you won't want to serialize all elements in a struct, since it's counting on the format string to "guess" the alignment of the elements.

Another weak pointer I found. The user has to make sure types are the same size when exchanging data between apps compiled with different compilers. My library uses sizeof()...

Enouph of my ranting, I'm sure mine has flaws too, but I'm happy to see I have something unique:)


Regarding the closed source thing. I can make a statement in the license compromising to release as source code / allowing someone else to continue it if I decide to stop development, maybe that could make a difference ?

On the make money aspect, yes, true for Amiga, but this is the kind of thing that can be ported to various platforms more or less easily, there's no GUI and the concept/algorithms will be the same. So I could make (or invite some of you guys in here to) ports for Amiga68k, AOS4, MOS, AROS, OpenBeOS, Linux, Windows, MacOS....
Only dragback will be that I've actually made it multithreaded, i.e. serialization actually happends at the same time the stream is sent. This is tested and working(on 68k), it's possible cause I put marks in buffer subsets which are accessed by a semaphore. So implementing this last thing will be platform specific, if I don't decide to make the windows version synchrounous that is..