Welcome, Guest. Please login or register.

Author Topic: Example of C source code for getting web page.  (Read 8421 times)

Description:

0 Members and 1 Guest are viewing this topic.

Offline koaftder

  • Hero Member
  • *****
  • Join Date: Apr 2004
  • Posts: 2116
    • Show only replies by koaftder
    • http://koft.net
Re: Example of C source code for getting web page.
« Reply #14 on: January 21, 2006, 06:36:49 AM »
@patrik

You are right about the extraneous stuff i had put in there. I just wanted to demonstrate the connection and retrieving some stuff, no need to confuse people.

That document you pointed out is a wonderful read.

@AmigaEd

Sorry to point you off into a wrong direction. I grabbed the package and sure enough, doesnt support amiga ): It's a really nice easy lib to work with. It supported dos/windows/os2/unix/linux/vms, etc. I guess i just thought it ran on amiga cause the guy who wrote it is a nasa geek and everybody seems to mention about how much amiga was used in that organisation.
 

Offline Jose

  • Hero Member
  • *****
  • Join Date: Feb 2002
  • Posts: 2871
    • Show only replies by Jose
Re: Example of C source code for getting web page.
« Reply #15 on: January 21, 2006, 12:37:44 PM »
"If targetting for the Amiga, you should take a look at the AmiTCP-SDK which gives you the necessary headers to work with bsdsocket.library (also link-libraries that can do some misc stuff for you, but they are not needed) which is the standard implementation of the BSD sockets API amongst Amiga TCP/IP stacks."

Even Roadshow ?
I'm asking this cause I'm planning on learning some network stuff too. About a year I got a book from Uni's library on networking but got completely let down by the amount of all the different protocols. I guess one doesn't need to know TCP/IP to do some basic network coding, only if I was to do my own stack. But since many hacks one see desbribed on the net imply some TCP/IP knowledge I'd like to learn it to understand it:) 8-)
\\"We made Amiga, they {bleep}ed it up\\"
 

Offline AmigaEdTopic starter

  • His Dudeness, El Duderino
  • Hero Member
  • *****
  • Join Date: Jan 2005
  • Posts: 512
    • Show only replies by AmigaEd
Re: Example of C source code for getting web page.
« Reply #16 on: January 21, 2006, 12:45:16 PM »
@koaftder,
Hey no problem, no need to be sorry. All of the information that you have posted has been very helpful and I'm sure that eventually I will also give this a try on windoze or unix (for work purposes of course ) ;-)

Best Regards,
AmigaEd
"Pretty soon they will have numbers tattooed on our foreheads." - Jay Miner 1990

La Familia...
A1K - La Primera Dama -1987
A1K - La Princesa- January 2005
A2K - La Reina - February 2005
A2K - Doomy - March 2005
A500 - El Gran Jugador - April 2005
A1200 - La Hermosa Vista - May 2005
A2KHD - El Duro Grande - May 2005
A600 - Prístino - May 2005
A1200 - El Trueno Grande - July 2005
CDTV - El Misterioso - August 2005
C64 - El Gran Lebows
 

Offline patrik

Re: Example of C source code for getting web page.
« Reply #17 on: January 21, 2006, 01:08:14 PM »
@Jose:

Yes, all TCP/IP-stacks on the Amiga, except the obselete AS225, gives you access through bsdsocket.library, even Roadshow - else no existing network applications would work with it.

It is always good to have atleast some knowledge about the protocols you are going to use. Not necessary all the nitty gritty details, but it will give you a much better understanding of how to create applications that are suited for them if you have a fair amount of knowledge about how they work, what their strongpoints and drawbacks are, etc.


/Patrik
 

Offline Piru

  • \' union select name,pwd--
  • Hero Member
  • *****
  • Join Date: Aug 2002
  • Posts: 6946
    • Show only replies by Piru
    • http://www.iki.fi/sintonen/
Re: Example of C source code for getting web page.
« Reply #18 on: January 21, 2006, 03:08:49 PM »
@koaftder

Bugs:

- You don't check if malloc() fails, but just crash if it does.
- You don't check if socket() fails, but just continue instead.
- bzero ( &(socket_detials.sin_zero), 8 ) ; is wrong. It assumes knowlege of the struct sockaddr_in, which can be different between platforms. Typecally it is 8 though, but there is no guarantee of this.
- You don't bail out if connect() fails, but just continue.
- You don't check if send() succeeds.
- You don't check how much data you manage to recv().
- You limit the recv size to 20000 bytes. If more data would be available you just truncate input.
- There is no guarantee single recv() will get all the input at once. You might get just the header for the 1st call, or part of the header. You should call recv() till -1 (error) or 0 (eof) is returned.
- You printf %s the input buffer, even though it is not '\0' terminated.
- Sending fixed cookies will not work. Esp PHPSESSID will just fail once the session id has expired.
 

Offline ChaosLord

  • Hero Member
  • *****
  • Join Date: Nov 2003
  • Posts: 2608
    • Show only replies by ChaosLord
    • http://totalchaoseng.dbv.pl/news.php
Re: Example of C source code for getting web page.
« Reply #19 on: January 21, 2006, 04:03:35 PM »
Nothing gets past Piru.   :-D
Wanna try a wonderfull strategy game with lots of handdrawn anims,
Magic Spells and Monsters, Incredible playability and lastability,
English speech, etc. Total Chaos AGA
 

Offline AmigaEdTopic starter

  • His Dudeness, El Duderino
  • Hero Member
  • *****
  • Join Date: Jan 2005
  • Posts: 512
    • Show only replies by AmigaEd
Re: Example of C source code for getting web page.
« Reply #20 on: January 21, 2006, 04:08:29 PM »
Hi Piru,
We know you are the Amiga Code master.
How about showing us the correct way to do it.

Thank you,
AmigaEd
"Pretty soon they will have numbers tattooed on our foreheads." - Jay Miner 1990

La Familia...
A1K - La Primera Dama -1987
A1K - La Princesa- January 2005
A2K - La Reina - February 2005
A2K - Doomy - March 2005
A500 - El Gran Jugador - April 2005
A1200 - La Hermosa Vista - May 2005
A2KHD - El Duro Grande - May 2005
A600 - Prístino - May 2005
A1200 - El Trueno Grande - July 2005
CDTV - El Misterioso - August 2005
C64 - El Gran Lebows
 

Offline Piru

  • \' union select name,pwd--
  • Hero Member
  • *****
  • Join Date: Aug 2002
  • Posts: 6946
    • Show only replies by Piru
    • http://www.iki.fi/sintonen/
Re: Example of C source code for getting web page.
« Reply #21 on: January 21, 2006, 05:31:13 PM »
Code: [Select]

;/*
gcc -noixemul -Wall -O2 httpget.c -o httpget
quit
*/

#include <sys/types.h>
#include <sys/socket.h>
#include <netdb.h>
#include <netinet/in.h>
#include <string.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#ifndef STDOUT_FILENO
#define STDOUT_FILENO 1
#endif


#include <proto/exec.h>
#ifdef __SASC
#include <proto/socket.h>
#endif
#include <clib/alib_protos.h>

#define RECV_BUFSIZE 16384

struct MinList *http_get(const char *host, int port, const char *path);
struct MinList *dorecv(int s);
void dumplist(struct MinList *list);
void freelist(struct MinList *list);

struct datanode
{
  struct MinNode node;
  int            len;
};

int main(void)
{
  struct MinList *res;

  res = http_get(&quot;www.amiga.org&quot;, 80, &quot;/&quot;);
  if (res)
  {
    dumplist(res);
    freelist(res);
  }
  else
  {
    fprintf(stderr, &quot;http_get failed\n&quot;);
  }

  return 0;
}

/*
   FUNCTION

   http_get - HTTP GET a location off a web server

   struct MinList *http_get(const char *host, int port, const char *path)
   

   INPUT

   The http-request must be split into valid components for this function.

   host: hostname
   port: port number
   path: the path of object to http get. spaces and special chars should
         be encoded to %<hex>

   RESULT

   struct MinList *

     NULL if error, else list filled with 'struct datanode' nodes. Note
     that the output includes the full header returned by the server, and
     it's left for the caller to parse it (separate header and actual
     data).

   NOTE

   This function blocks, and it can potentially take hours to complete;
   for example if the file is long, or the connection is very slow.
*/

struct MinList *http_get(const char *host, int port, const char *path)
{
  struct MinList *list = NULL;
  int s;

  if (host && host[0] && port > 0 && path)
  {
    struct sockaddr_in saddr;
    struct hostent *he;

    bzero(&saddr, sizeof(saddr));

    he = gethostbyname(host);
    if (he)
    {
      memcpy(&saddr.sin_addr, he->h_addr, he->h_length);
      saddr.sin_family = he->h_addrtype;
    }
    else
    {
      saddr.sin_addr.s_addr = inet_addr(host);
      saddr.sin_family = AF_INET;
    }

    if (saddr.sin_addr.s_addr != INADDR_NONE)
    {
      s = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
      if (s != -1)
      {
        saddr.sin_port = htons(port);

        if (connect(s, (struct sockaddr *) &saddr, sizeof(saddr)) != -1)
        {
          const char *fmt =
            &quot;GET %s HTTP/1.0\r\n&quot;
            &quot;Host: %s\r\n&quot;
            &quot;User-Agent: httpget_test_app/1.0\r\n&quot;
            &quot;\r\n&quot;;
          char *req;

          if (path[0] == '\0')
          {
            path = &quot;/&quot;;
          }

          req = malloc(strlen(fmt) +
                       strlen(path) - 2 +
                       strlen(host) - 2 + 1);
          if (req)
          {
            int reqlen;

            sprintf(req, fmt, path, host);
            reqlen = strlen(req);

            if (send(s, req, reqlen, 0) == reqlen)
            {
              list = dorecv(s);
            }

            free(req);
          }

          close(s);
        }
      }
    }
  }

  return list;
}

struct MinList *dorecv(int s)
{
  struct MinList *list = NULL;
  UBYTE *buf;

  buf = malloc(RECV_BUFSIZE);
  if (buf)
  {
    int ok = 0;

    for (;;)
    {
      int actual;

      actual = recv(s, buf, RECV_BUFSIZE, 0);
      if (actual == -1)
      {
        /* error */
        break;
      }
      else if (actual == 0)
      {
        /* eof */
        ok = 1;
        break;
      }
      else
      {
        struct datanode *node;

        if (!list)
        {
          list = malloc(sizeof(*list));
          if (!list)
          {
            break;
          }
          NewList((struct List *) list);
        }

        node = malloc(sizeof(*node) + actual);
        if (!node)
        {
          break;
        }
        node->len = actual;
        memcpy(node + 1, buf, actual);

        AddTail((struct List *) list, (struct Node *) node);
      }
    }

    if (!ok)
    {
      freelist(list);
      list = NULL;
    }

    free(buf);
  }

  return list;
}

void dumplist(struct MinList *list)
{
  if (list)
  {
    struct datanode *node;

    fflush(stdout);

    for (node = (APTR) list->mlh_Head;
         node->node.mln_Succ;
         node = (APTR) node->node.mln_Succ)
    {
      write(STDOUT_FILENO, node + 1, node->len);
    }

    fflush(stdout);
  }
}

void freelist(struct MinList *list)
{
  if (list)
  {
    struct datanode *node, *nextnode;

    for (node = (APTR) list->mlh_Head;
         (nextnode = (APTR) node->node.mln_Succ);
         node = nextnode)
    {
      free(node);
    }

    free(list);
  }
}
 

Offline Piru

  • \' union select name,pwd--
  • Hero Member
  • *****
  • Join Date: Aug 2002
  • Posts: 6946
    • Show only replies by Piru
    • http://www.iki.fi/sintonen/
Re: Example of C source code for getting web page.
« Reply #22 on: January 21, 2006, 05:39:25 PM »
Oh, I used exec lists there.

If this thing must be portable you can use dlist instead.
 

Offline ChaosLord

  • Hero Member
  • *****
  • Join Date: Nov 2003
  • Posts: 2608
    • Show only replies by ChaosLord
    • http://totalchaoseng.dbv.pl/news.php
Re: Example of C source code for getting web page.
« Reply #23 on: January 21, 2006, 06:03:27 PM »
I am trying to slowly convert my braincells over from SASC to gcc.


What does -Wall do?

What does -noixemul do?

Wanna try a wonderfull strategy game with lots of handdrawn anims,
Magic Spells and Monsters, Incredible playability and lastability,
English speech, etc. Total Chaos AGA
 

Offline Piru

  • \' union select name,pwd--
  • Hero Member
  • *****
  • Join Date: Aug 2002
  • Posts: 6946
    • Show only replies by Piru
    • http://www.iki.fi/sintonen/
Re: Example of C source code for getting web page.
« Reply #24 on: January 21, 2006, 06:09:49 PM »
@ChaosLord

-W is warning option. -Wall means "enable all standard warnings"

-noixemul means 'do not use ixemul.library' (this is stricly gcc thing).

BTW: In this program m68k gcc users might need to drop the -noixemul option, that is to use ixemul.library. Also, dependin on the gcc version -lamiga migth be required to get the thing to link.
 

Offline Dr_Righteous

  • Hero Member
  • *****
  • Join Date: Feb 2002
  • Posts: 1345
    • Show only replies by Dr_Righteous
Re: Example of C source code for getting web page.
« Reply #25 on: January 21, 2006, 06:26:08 PM »
@Piru
BAH! Who needs error checking?!  :-D
- Doc

A4000D, A3640 OC-36.3MHz, custom tower, Mediator A4000D. Diamond Banshee 16M, Indivision AGA 4000, GVP HC+8.

Mac Mini 1.5GHz, that might run MorphOS someday, when the fools who own it come to the realization that 30 minutes just isn\'t enough time to play with it enough to decide whether or not you like it enough to cough up $200.

 - Someone please design SOME kind of DIY accelerator for the A4000. :D -
 

Offline Piru

  • \' union select name,pwd--
  • Hero Member
  • *****
  • Join Date: Aug 2002
  • Posts: 6946
    • Show only replies by Piru
    • http://www.iki.fi/sintonen/
Re: Example of C source code for getting web page.
« Reply #26 on: January 21, 2006, 06:30:48 PM »
Note: It builds with SAS/C aswell now (at least with MiamiSDK).

sc resopt incdir Path:to/MiamiSDK/netinclude/ link httpget.c lib Path:to/MiamiSDK/netlib/miami.lib

Anyway, if someone has any more questions about the src, just ask. I'm happy to explain it.
 

Offline koaftder

  • Hero Member
  • *****
  • Join Date: Apr 2004
  • Posts: 2116
    • Show only replies by koaftder
    • http://koft.net
Re: Example of C source code for getting web page.
« Reply #27 on: January 21, 2006, 08:42:27 PM »
Quote

Piru wrote:
@koaftder

Bugs:

- You don't check if malloc() fails, but just crash if it does.
- You don't check if socket() fails, but just continue instead.
- bzero ( &(socket_detials.sin_zero), 8 ) ; is wrong. It assumes knowlege of the struct sockaddr_in, which can be different between platforms. Typecally it is 8 though, but there is no guarantee of this.
- You don't bail out if connect() fails, but just continue.
- You don't check if send() succeeds.
- You don't check how much data you manage to recv().
- You limit the recv size to 20000 bytes. If more data would be available you just truncate input.
- There is no guarantee single recv() will get all the input at once. You might get just the header for the 1st call, or part of the header. You should call recv() till -1 (error) or 0 (eof) is returned.
- You printf %s the input buffer, even though it is not '\0' terminated.
- Sending fixed cookies will not work. Esp PHPSESSID will just fail once the session id has expired.


haha, i'm supprised you took the time to list all the unresponsible things i did. Since you took the time to review it, i'll take the time to fix it. Here it is:

Code: [Select]

#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <string.h>
#include <stdio.h>
#include <stdlib.h>

int main ( void ) {
int socket_handle ;
struct sockaddr_in socket_detials ;
char * input_buffer;
char * pinput_buffer ;
ssize_t bytes_received ;
ssize_t bytes_sent ;
char * phttpget ;
char * httpget =
 &quot;GET / HTTP/1.0\r\n&quot;
 &quot;Host: www.amiga.org\r\n&quot;
 &quot;\r\n&quot;;

phttpget = httpget ;
bytes_sent = 0 ;

input_buffer = malloc(1024);
if ( input_buffer == NULL ) {
printf ( &quot;Sorry, couldnt allocate memory for input buffer\n&quot; );
return -1 ;
}
memset ( input_buffer, 0, 1024 ) ;

memset ( &socket_detials , 0 , sizeof(struct sockaddr_in) );

socket_handle = socket ( AF_INET, SOCK_STREAM, 0) ;
if ( socket_handle == -1 ) {
printf ( &quot;Could not create socket\n&quot; ) ;
return -1 ;
}
socket_detials.sin_family = AF_INET ;
socket_detials.sin_addr.s_addr=inet_addr(&quot;68.90.68.66&quot;);
socket_detials.sin_port = htons(80);

if ( connect (socket_handle,(struct sockaddr*)&socket_detials, sizeof ( struct sockaddr)) == -1 ){
printf ( &quot;Couldnt connect to server\n&quot; ) ;
return -1 ;
}

printf ( &quot;Attempting to send %d bytes to server\n&quot; , strlen ( httpget ) );
for(;;){
bytes_sent = send ( socket_handle , phttpget, strlen(phttpget), 0 ) ;
if ( bytes_sent == -1 ) {
printf ( &quot;An error occured sending data\n&quot; );
return -1 ;
}
if ( httpget+strlen(httpget) == phttpget )
break ;
phttpget += bytes_sent ;
}

        for (;;) {
bytes_received = recv ( socket_handle , input_buffer , 1023, 0 ) ;
if ( bytes_received == -1 ) {
printf ( &quot;An error occured during the receive procedure \n&quot; ) ;
return 0 ;
}
if ( bytes_received == 0 )
break ;
pinput_buffer = input_buffer + bytes_received ;
*pinput_buffer = 0 ;
printf ( &quot;%s&quot; , input_buffer ) ;
}

printf ( &quot;\nFinished receiving data\n&quot; ) ;
return 0 ;
}
 

Offline AmigaEdTopic starter

  • His Dudeness, El Duderino
  • Hero Member
  • *****
  • Join Date: Jan 2005
  • Posts: 512
    • Show only replies by AmigaEd
Re: Example of C source code for getting web page.
« Reply #28 on: January 22, 2006, 03:57:36 AM »
@Piru,
Thank you for posting the code.

When I try to compile the code I get an error. Here is the compiler log....

Compiler: m68k-AmigaOS
Executing  m68k-amigaos-gcc.exe...
m68k-amigaos-gcc.exe "C:\CrossCompiler\AmiDevCpp\MyAmigaCProjects\GetWebPageExamples\Piru's example\PirusExample.c" -o "C:\CrossCompiler\AmiDevCpp\MyAmigaCProjects\GetWebPageExamples\Piru's example\PirusExample.exe"    -I"C:\CrossCompiler\AmiDevCpp\usr\local\amiga\m68k-amigaos\sys-include"  -I"C:\CrossCompiler\AmiDevCpp\usr\local\amiga\m68k-amigaos\include"  -I"C:\CrossCompiler\AmiDevCpp\usr\local\amiga\include\g++-3"   -L"C:\CrossCompiler\AmiDevCpp\usr\local\amiga\m68k-amigaos\lib" -L"C:\CrossCompiler\AmiDevCpp\usr\local\amiga\m68k-amigaos\lib\libnix"
In file included from /usr/local/amiga/lib/gcc/m68k-amigaos/3.4.0/../../../../m68k-amigaos/sys-include/clib/alib_protos.h:22,
                 from C:\CrossCompiler\AmiDevCpp\MyAmigaCProjects\GetWebPageExamples\Piru's example\PirusExample.c:23:
/usr/local/amiga/lib/gcc/m68k-amigaos/3.4.0/../../../../m68k-amigaos/sys-include/devices/timer.h:30: error: redefinition of `struct timeval'
Execution terminated



I can't quite figure out what is going on.

IF I comment out the...
#include
then I can get it to compile and it seems to work.

Thank you,
AmigaEd
"Pretty soon they will have numbers tattooed on our foreheads." - Jay Miner 1990

La Familia...
A1K - La Primera Dama -1987
A1K - La Princesa- January 2005
A2K - La Reina - February 2005
A2K - Doomy - March 2005
A500 - El Gran Jugador - April 2005
A1200 - La Hermosa Vista - May 2005
A2KHD - El Duro Grande - May 2005
A600 - Prístino - May 2005
A1200 - El Trueno Grande - July 2005
CDTV - El Misterioso - August 2005
C64 - El Gran Lebows
 

Offline Piru

  • \' union select name,pwd--
  • Hero Member
  • *****
  • Join Date: Aug 2002
  • Posts: 6946
    • Show only replies by Piru
    • http://www.iki.fi/sintonen/
Re: Example of C source code for getting web page.
« Reply #29 from previous page: January 22, 2006, 04:09:00 AM »
@AmigaEd

I believe it's include conflict between sys/time.h and devices/timer.h. I didn't see it as MorphOS includes account for the problem.

Anyway, clib/alib_protos.h isn't really required here as it only declares one of the prototypes (NewList()). As you noticed it works without.