Motivation

Since the original release of Netscape 6, the third-party vendor that originally authored the SmartDownload library on windows and mac stopped supporting it. There were significant defects that warranted the need for a maintainable download library. There were two choices:

• get the source code for the SmartDownload libraries on mac and windows and then expand the original linux FTP implementation to be at par with the SmartDownload libraries
• expand the original linux FTP implementation to be at par with the SmartDownload feature set and make the library cross-platform

• it allowed us to have an easily maintainable codebase we were already familiar with (SmartDownload source was to be provided with minimal to no documentation which was not deemed acceptable by engineering).
• we could implement our own integrated user-interface (the SmartDownload library forced the use of their own user-interface which didn't comply with interface guidelines)
• we could tune the network library at will to work with our server farm from a functional stand point and from an efficiency standpoint
• mozilla could now take advantage of the download functionality and architecture of the stub installers: previously the SmartDownload libraries being binary were not acceptable for use by mozilla.org since source is required for code used in any mozilla project (mostly)

Feature Set

Example and Building the Library

See TestLibxpnet for a sample usage of these APIs. The library can easily be built on any platform by building the three objects first: nsSocket.o, nsFTPConn.o, nsHTTPConn.o and linking them into a static library. Note that all three classes, i.e., nsSocket, nsFTPConn, nsHTTPConn, have self-tests built into them if the appropriate macros are defined at compile time: TEST_NSSOCKET for nsSocket, TEST_NSHTTPCONN for nsHTTPConn, and TEST_NSFTPCONN for nsFTPConn. Each class can be compiled into a standalone program when these test macros are turned on. Alternatively, if you have built mozilla, you will find the resultant static library in dist or under the libxpnet/src directory.

Architecture

API Overview

1. nsHTTPConn

   Constructors
   

   • nsHTTPConn(char *aHost, int aPort, char *aPath);
     Prepare to create an HTTP connection to the supplied host at a possibly non-default port and download the resultant response into the file whose path is specified.
     aHost: a DNS-resolvable hostname or an IP address string (only IPv4 currently supported)
     aPort: a possibly non-default integer port number less than 65536
     aPath: a path to the destination file on the local system that will be created with the HTTP response contents
     
     
   • nsHTTPConn(char *aURL);
     Prepare to create an HTTP connection to get the file at the supplied URL. The destination file on the local system will be named after the leaf name of the HTTP URL supplied. If no leaf name is detected then the default file name will be "index.html". Note that the downloaded file will be created in the current working directory.
     aURL: an HTTP URL per the semantics specified by RFC 1738
     
     
   • nsHTTPConn(char *aHost, int aPort, char *aPath, int (*aEventPumpCB)(void));
     Prepare to create an HTTP connection to the supplied host at a possibly non-default port and download the resultant response into the file whose path is specified. An event pump callback can be supplied to receive notification during the connection phase.
     aHost: a DNS-resolvable hostname or an IP address string (only IPv4 currently supported)
     aPort: a possibly non-default integer port number less than 65536
     aPath: a path to the source file on the HTTP server (http://<aHost>:<aPort>/<aPath>)
     aEventPumpCB: a callback function to be implemented by the client which will be called during the connection phase (optional and can be NULL)
     
     
   • nsHTTPConn(char *aURL, int (*aEventPumpCB)(void));
     Prepare to create an HTTP connection to get the file at the supplied URL. The destination file on the local system will be named after the leaf name of the HTTP URL supplied. If no leaf name is detected then the default file name will be "index.html". Note that the downloaded file will be created in the current working directory. An event pump callback can be supplied to receive notification during the connection phase.
     aURL: an HTTP URL per the semantics specified by RFC 1738
     aEventPumpCB: a callback function to be implemented by the client which will be called during the connection phase (optional and can be NULL)
     

   Methods

   • int Open();
     Opens a new HTTP connection using the host and port specification passed in the constructor.
     
     
   • int ResumeOrGet(HTTPGetCB aCallback, char *aDestFile);
     Resumes a mid-file download if an old destination file is found or starts the download afresh if no destination file is found on the local file system. Uses the URL or server path passed in the constructor. Automatically starts from the end of the incompletely downloaded file.
     aCallback:the optional download progress callback to be implemented by the client (can be NULL) that is updated once every 1KB (can be tuned in the libxpnet source)
     aDestFile: the path to the destination file to be downloaded
     
     
   • int Get(HTTPGetCB aCallback, char *aDestFile);
     Performs an HTTP GET of the URL or server path passed in the constructor.
     aCallback: the optional download progress callback to be implemented by the client (can be NULL) that is updated once every 1KB (can be tuned in the libxpnet source)
     aDestFile: the path to the destination file to be downloaded
     
     
   • int Get(HTTPGetCB aCallback, char *aDestFile, int aResumePos);
     Resumes a mid-file download from the resume position of the file being downloaded. Uses the URL or server path passed in the constructor.
     aCallback: the optional download progress callback to be implemented by the client (can be NULL) that is updated once every 1KB (can be tuned in the libxpnet source)
     aDestFile: the path to the destination file to be downloaded
     aResumePos: the position in the middle of the file to start downloading: usually the size of the incompletely downloaded file
     
     
   • int Close();
     Closes the HTTP connection to the server specified in the constructor if originally opened.
     
     
   • void SetProxyInfo(char *aProxiedURL, char *aProxyUser, char *aProxyPswd);
     Use this API before performing an HTTP GET. If you need to download through an HTTP proxy server pass the proxy server hostname and port in the constructor and the actual URL to download in this API, i.e., the "proxied URL." In addition, we optionally support a username/password specification for the proxy server. The password is base64 encoded and sent over the network for authentication purposes if specified.
     aProxiedURL: the URL of the actual file to be downloaded (can be an FTP or HTTP URL)
     aProxyUser: optional proxy server username
     aProxyPswd: optional proxy server password that is base64 encoded and sent to the proxy server
     
     
   • static int ParseURL(const char *aProto, char *aURL, char **aHost, int *aPort, char **aPath);
     A helper static method to parse an FTP or HTTP URL depending on which protocol this URL is expected to be. It returns the host, port and server path. Useful for use with the protocol implementations' constructors.
     aProto: the protocol this URL is expected to be in (kHTTPProto and kFTPProto defined in nsHTTPConn.cpp
     aURL: the source URL to parse
     aHost: the resultant hostname parsed from the supplied URL
     aPort: the resultant port integer parsed from the supplied URL
     aPath: the resultant server path of the file to download parsed from the supplied URL
     
   
   

2. nsFTPConn

   Constructors

   • nsFTPConn(char *aHost);
     Prepare to make an FTP connection to the supplied host.
     aHost: a DNS-resolvable hostname or an IP address string (only IPv4 currently supported)
     
     
   • nsFTPConn(char *aHost, int (*aEventPumpCB)(void));
     Prepare to make an FTP connection to the supplied host. An event pump callback can be supplied to receive notification during the connection phase.
     aHost: a DNS-resolvable hostname or an IP address string (only IPv4 currently supported)
     aEventPumpCB: a callback function to be implemented by the client which will be called during the connection phase (optional and can be NULL)
     

   Methods

   • int Open();
     Open a connection to the FTP host on default FTP port 21 using the host specified in the constructor.
     
     
   • int Open(char *aHost);
     Open a connection to the FTP host on default FTP port 21 using the host specified explicitly by the client of this API.
     aHost: a DNS-resolvable hostname or an IP address string (only IPv4 currently supported)
     
     
   • int ResumeOrGet(char *aSrvPath, char *aLoclPath, int aType, int aOvWrite, FTPGetCB aCBFunc);
     Resumes a mid-file download if an old destination file is found or starts the download afresh if no destination file is found on the local file system. Uses the server path explicitly passed in by the client of this API. Automatically starts from the end of the incompletely downloaded file.
     aSrvPath: the path to the file on the server (ftp://<aHost>:21/<aPath>)
     aLoclPath: the path to the destination file on the local filesystem
     aType: the FTP transfer type, i.e., binary or ASCII
     aOvWrite: whether or not to overwrite the destination file if it already exists
     aCBFunc: the optional download progress callback to be implemented by the client (can be NULL) that is updated once every 1KB (can be tuned in the libxpnet source)
     
     
   • int Get(char *aSrvPath, char *aLoclPath, int aType, int aOvWrite, FTPGetCB aCBFunc);
     Performs an FTP retrieve command (RETR) of the server path explicitly passed in by the client of this API.
     aSrvPath: the path to the file on the server (ftp://<aHost>:21/<aPath>)
     aLoclPath: the path to the destination file on the local filesystem
     aType: the FTP transfer type, i.e., binary or ASCII
     aOvWrite: whether or not to overwrite the destination file if it already exists
     aCBFunc: the optional download progress callback to be implemented by the client (can be NULL) that is updated once every 1KB (can be tuned in the libxpnet source)
     
     
   • int Get(char *aSrvPath, char *aLoclPath, int aType, int aResumePos, int aOvWrite, FTPGetCB aCBFunc);
     Resumes a mid-file download from the resume position of the file being downloaded. Uses the server path explicitly passed in by the client of this API. Resumption is indicated to the FTP server by sending a restart (REST) command before retrieving the file.
     aSrvPath: the path to the file on the server (ftp://<aHost>:21/<aPath>
     aLoclPath: the path to the destination file on the local filesystem
     aType: the FTP transfer type, i.e., binary or ASCII
     aResumePos: the position in the middle of the file to start downloading: usually the size of the incompletely downloaded file
     aOvWrite: whether or not to overwrite the destination file if it already exists
     aCBFunc: the optional download progress callback to be implemented by the client (can be NULL) that is updated once every 1KB (can be tuned in the libxpnet source)
     
     
   • int Close();
     Closes the FTP connection to the server specified in the constructor if originally opened by sending the QUIT command.
     
   
   

3. nsSocket

   Constructors

   • nsSocket(char *aHost, int aPort);
     Prepare to open a socket to the specified host on the specified port using a local ephemeral port.
     aHost: a DNS-resolvable hostname or an IP address string (only IPv4 currently supported)
     aPort: a possibly non-default integer port number less than 65536
     
     
   • nsSocket(char *aHost, int aPort, int (*aEventPumpCB)(void) );
     Prepare to open a socket to the specified host on the specified port using a local ephemeral port. An event pump callback can be supplied to receive notification during the connection phase.
     aHost: a DNS-resolvable hostname or an IP address string (only IPv4 currently supported)
     aPort: a possibly non-default integer port number less than 65536
     aEventPumpCB: a callback function to be implemented by the client which will be called during the connection phase (optional and can be NULL)
     

   Methods

   • int Open();
     Open the socket to the host/port passed in the constructor.
     
     
   • int SrvOpen(); // server alternate to client Open()
     Open a listening socket on an ephemeral port.  Useful for FTP passive mode.
     
     
   • int SrvAccept(); // must be called after SrvOpen()
     Accept a client connection on the listening socket.
     
     
   • int Send(unsigned char *aBuf, int *aBufSize);
     Send a buffer of the specified size to the host/port open already. Requires Open() to have been called first.
     aBuf: the buffer to send
     aBufSize: the size of the buffer to send (on return indicates how much of the buffer was sent)
     
     
   • int Recv(unsigned char *aBuf, int *aBufSize);
     Receive a buffer of an expected size: keep flushing the socket till the buffer is filled.
     aBuf: the buffer to stuff with the received data
     aBufSize: the expected number of bytes to receive (on return the actual number of bytes received)
     
     
   • int Recv(unsigned char *aBuf, int *aBufSize, int aTimeoutThresholdUsecs);
     Receive a buffer of an expected size: keep flushing the socket till the buffer is filled. Accepts a configurable timeout threshold in microseconds passed to select() when in a non-blocking loop.
     aBuf: the buffer to stuff with the received data
     aBufSize: the expected number of bytes to receive (on return the actual number of bytes received)
     aTimeoutThresholdUsecs: the total timeout specified in microseconds
     
     
   • int Close();
     Close this socket by closing the platform-native descriptor.
     
     
   • int GetHostPortString(char **aHostPort);
     Returns the host-port string representation of the IP address and port of the listening socket descriptor, i.e., "<AA1>,<AA2>,<AA3>,<AA4>,<PP1>,<PP2>" where <AAx> is the string value of integer in the xth position of an IP address and <PPx> is the high 8 bits and low 8 bits of the port.
     aHostPort: the formatted host-port string on return
     
     
   • static float CalcRate(struct timeval *aPre, struct timeval *aPost, int aBytes);
     Helper utility to calculate the download rate. Can be used by clients of the FTP and HTTP implementations in their callbacks.
     aPre: time before download
     aPost: time now or time when download completed
     aBytes: number of bytes downloaded
   
   

Future Work