char *buf2=new char[size]; problem

[Karlos]

I haven't read all of this thread, so I might be missing the obvious:

Why grow an array in tiny increments?

Make a linked list of allocation buffers, each of which contains space for a a few kb of data.

Then simply traverse the list in an outer loop, filling the current buffer and keeping track of how full it is. When it is full, add a new buffer to the list and start from there. You only need to keep track of the size of the current buffer being filled and the overall amount of data recieved.

Disadvantages:

Your data is stored in a noncontiguous way, divided into blocks of several KB. Random access to the data is therefore not efficient when scattered accesses span several blocks.

You can, however traverse the list relatively easily and sequential access would not be a huge performance hit.

Advantages:

No limit to the amount of data that can  be recieved (until memory fills, basically). You never copy anything at any point. Your blocksize, inclusive of link pointers can be carefully chosen to give a single block that matches the system page size. This can be an effective optimisation and help reduce framgentation.


If you really need random read access, there is an alternative.

Allocate an array of pointers that point to sequentially allocated blocks of data. These would be all the same size, again several KB each.

You simply fill the first buffer (pointed to by buffers[0]), once that is full, you allocate another, pointed to by buffers[1] etc.

Random access can then be achieved by simply breaking down the supplied index into its block number and offset into the block.

Advantages:

Likely even faster than the linked list for sequential access and dramatically faster for random access.

Disadvantages:

Unless you periodically grow your buffers array, it has an implicit limit to the amount it can hold.

[PhatBoiCollier]

Its broken and crashes horribly?  A bit harsh matey.  I was working from memory (no pun intended).

You are correct though a check on the return value of realloc should be made.  If realloc fails then you are out of memory and so are screwed anyway!

Oh, and dont forget to free buf2, or it may break and crash horribly taking the world with it etc..

Code: [Select]


// Could do char buf1 here and pass &buf1 to function
char buf1[1];
// Initialise to 0 elements
int size=0;
// Initialise to NULL so realloc works properly!
// Could do "char *buf2 = NULL;", its the same thing!
char buf2[] = NULL;

while (recv(new_fd,buf1,sizeof(buf1),0) && (size>0))
{
    // Increase size of array
    size++;
    // Reallocate buffer (+1 for terminator!)
    if realloc(buf2,(size+1)*(sizeof(char))>0)
    {
        // Copy character (-1 because array starts at 0)
        buf2[size-1] = buf1[0];
        // Reset terminator
        buf2[size] = '\0';
        // Print status
        printf("size: %d\nbuf2: %s\n",size,buf2);
    }
    else
    {
        // realloc failed
        printf("No more memory!\n");
        // force loop exit
        size = -1;
    }
}
free(buf2);


[Piru]

@PhatBoiCollier
The code is still broken and crashes horribly.

If realloc() cannot expand the current memory allocation, it returns different pointer than the one passed in. In that case, you poke already free memory -> boom. If that doesn't nuke, it will crash at double free when calling realloc/free for the wrong ptr.

[PiR]

Ok, boys, calm down.
This isn't helping the thread owner.

More advanced memory management is really important thing, but as you can see it is now little bit beyond the current skills.

buffer2 can be or to small or to large

The only important thing is that it is not too small. If it is too large you just simply have some memory allocated that is not used. However from the nature of your code this memory is not used yet, but most probabily will be needed very soon. So every program which tries to be efficient, will allocate more memory than single byte and remember how much is used. When more memory is needed then the proper action is taken (I don't want to describe the possibilities of those actions now).

on win32 this work good. and i try to change on win32/bcc

This works 'by mistake'.
In every compiler, the memory allocating functions internally asks Operating System the number of bytes that simply has to be NOT LESS then requested from the programmer. However the programmer doesn't know that exact value and is not allowed to use more than he requested.

Simply win32 asks internally for bigger buffer than SAS/C. this buffer by coincidence was big enough for your example.
(I know I simplified the problem, Piru.)
But this is not a rule and you never should do this.

The idea of PhatBoiCollier to use realloc() is really very good in your case (I can explain why, but this can be hard to understand for you).

Being you I would do it this way:

Code: [Select]



char single_char; // same meaning as char signle_char[1];
// Initialise to 0 elements
unsigned allocated=0;
unsigned used=0;
#define BYTES_TO_INCREMENT 256

int OutOfMemory = 0;

char *current_buff = NULL;
char *next_buff;

//prepare the initial buffer
allocated += BYTES_TO_INCREMENT;
current_buff = malloc( allocated );
if( !current_buff )
{
    // NO MEMORY!
    OutOfMemory = 1;
    return;
}

while (recv(new_fd, &single_char, sizeof(single_char), 0) == sizeof(single_char))
{
    if( (used + 1) == allocated ) { // we used up all allocated memory (we reserve 1 character for terminating '\0' at the end)
        allocated += BYTES_TO_INCREMENT;
        next_buff = realloc( current_buff, allocated );
        if( next_buffer )
        { // we have new bigger buffer, the old one does not exist any more
          current_buff = next_buff;
        }
        else
        { // there was no memory left, old buffer still exists to allow us to use what we have until now
          OutOfMemory = 1;
          break;
        }
    }
    current_buff[ used ] = single_char;
    ++used;
}
current_buff[ used ] = '\0';
printf("allocated: %u\nused: %u\ncontent: %s\n", allocated, used, current_buff );
if( OutOfMemory )
{
    puts( "We run out of memory!" );
}
free( current_buff );



Feel free to use it.
Cheers

[kas1e]

to PiR:

big thnx, your sample is was good. i wrote all what i want.

"The idea of PhatBoiCollier to use realloc() is really very good in your case (I can explain why, but this can be hard to understand for you)."

plz, if you have some time, i try understand.

btw, maybe anyone have link on good docs about work with memory managment in advance ? amiga/sasc orientation will be good.

[PiR]

Hi

So I didn't discourage you... damn... ;-)

It is mayby not that hard to explain, but it surely takes time to describe all the details. I'll try to be short and clear:

realloc() solves you two problems:

1. it manages the memory for you (allocates new buffer, frees the old one, whenever needed).
2. if its needed, it performs a copy of the old buffer to the new one.

From the logical point of view you may think that realloc() always allocates new buffer, performs the copy and frees the old buffer.
But do you remember what I've written earlier about internal calls to OS? The allocated buffer is almost always bigger than requested by the programmer, but you don't know how much bigger.

You do not know. But realloc() can find it out. If really allocated buffer is big enough to serve the new value requested by the programmer (as second argument to realloc()), it is simply not needed to allocate, copy and free. So it happens - if only possible, the old buffer is returned as the new one and no more actions are made. What a great speedup!

You must know that memory allocation is quite difficult task for OS - it must check all free buffers and find the best one (and how to define the best one? the first found that is big enough? and how much of it should be returned?). So for that reasons you shouldn't allocate memory too often (not every single byte).

I have also to mention that the code I wrote is quite effective for what is needed in your code. I mean for transfering of some strings you don't need more complicated memory management. Notice that I defined relatively big INCREMENT (256 bytes), so I guess most of cases it would be enough to allocate only once. More complicated memory management would be needed if you had heavy transfers - a lot of data to read and write, and in unpredictable order.
This solution has also one more advantage dedicated for strings - it keeps all the content in single block, so it is always ready to be printed directly by simply using stdio functions without any tricks.

So for that purposes even professional programs would go this way.

If you're interested in more complicated memory management - this is not that much related with the OS at this level. You just have to learn about basic programistinc structures like lists, trees and hash-tables. Then practice, practice and practice to be able to use them intuitively and without mistakes (which are quite easy to be done and very hard to find afterwards in the working code). Then you'll find always the good answer for every programistic question you'll have while programing.


To explain those data structures it is always better to draw the pictures than to explain everything with plain words - it just simply doesn't work too well on the imagination that way.

All the best on your quest. :-)