Code recipe : Build a Download Manager

Ok lets begin the blog with something interesting to build, a download manager that boosts up the download speed. After that stated, lets begin with some basic information about the program we are going to build. Basically a download manager is a software that tries to boost up the download speed. Talking about the speed boost up there are many ways to make that happen. Some of known mechanism are Jumbo Grams, UDP transfer and multi-stream TCP transfer. First two options are a bit difficult to implement i.e. you may need your own server and switches to implement those. The most commonly used method to boost up the speed of data transfer is third one "Multi-Stream Data transfer in TCP". Most download managers use this technique to boost download speed including IDM. What IDM does is, it breaks file to be downloaded into multiple chunks, fetch each chunks in separate thread and finally after all the threads have completed their portion of download, merge those parts to a single file. The program we will be discussing about in this post will be using similar technique (Multi-Stream TCP) as IDM. Now after the background review lets jump right into coding part. Lets start with normal method of downloading the file from a server.

public void normal(String link) throws MalformedURLException, IOException{
        BufferedInputStream in = null;
        BufferedOutputStream out = null;
 
        // connect to the server
        URL url = new URL(link);
        HttpURLConnection conn = (HttpURLConnection)url.openConnection();
        conn.connect();
 
        // prepare for read from server
        in = new BufferedInputStream(conn.getInputStream());
         
        // prepare for write to local disk       
        String fileName = "TempFile";
        out = new BufferedOutputStream(new FileOutputStream(fileName));
        int n;
         
        // read from the server and write to local storage
        byte data[] = new byte[1024];
        while( (n = in.read(data)) != -1){
            out.write(data, 0, n);
        }
        in.close();
        out.close();
    }

What we are doing in this code is, we are first opening up a connection to the url as provided by the user. Then assign an input stream to the connection just for reading the incomings. Next step is to read from the input stream assigned to the connection and write the data to output stream that points to a local file output. Simple enough huh. Now lets move for implementing parallel streams. As mentioned in the review part firstly we are going to chop the total size of the file into different chunks, use the partial content indication method to notify the server, get those content in separate threads and finally merge those chunks into single.

package workingonnetworking;
 
import java.io.*;
import java.net.*;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import java.util.logging.Level;
import java.util.logging.Logger;
 
public class DownloadManager {
 
    private OutputStream[] r;
    private final int totalChunks = 5;
    private File[] tempFile;
    public DownloadManager() throws FileNotFoundException{
        r = new OutputStream[totalChunks];
        tempFile = new File[totalChunks];
         
        for (int i = 0 ; i < totalChunks ; i++){
            tempFile[i] = new File("temp"+i);
            r[i] = new BufferedOutputStream(new FileOutputStream(tempFile[i]));
        }
         
    }
    public static void main(String[] args) throws IOException {
        DownloadManager cc = new DownloadManager();
//        cc.normal(http://localhost:31415/public_html/img/textile.jpg);
        cc.parallel("http://localhost:31415/public_html/img/textile.jpg");
    }
    public void normal(String link) throws MalformedURLException, IOException{
        BufferedInputStream in = null;
        BufferedOutputStream out = null;
 
        // connect to the server
        URL url = new URL(link);
        HttpURLConnection conn = (HttpURLConnection)url.openConnection();
        conn.connect();
 
        // prepare for read from server
        in = new BufferedInputStream(conn.getInputStream());
         
        // prepare for write to local disk       
        String fileName = getFileName(link);
        out = new BufferedOutputStream(new FileOutputStream(fileName));
        int n;
         
        // read from the server and write to local storage
        byte data[] = new byte[1024];
        while( (n = in.read(data)) != -1){
            out.write(data, 0, n);
        }
        in.close();
        out.close();
    }
     
     
    public void parallel(String link){
        try {
            // open up a connection to get total size of the file
            URL url = new URL(link);
            HttpURLConnection conn = (HttpURLConnection)url.openConnection();
            int total = conn.getContentLength();
             
            // connection array
            HttpURLConnection[] con = new HttpURLConnection[totalChunks];
            Minions[] m = new Minions[totalChunks];
             
            // calculate bytes each stream has to download
            int eachDownload = total/totalChunks;                        
            String byteRange = "";
            ExecutorService tE = Executors.newCachedThreadPool();
             
            for (int i = 0 ; i < totalChunks ; i++){
                // open the connection streams with byte range
                byteRange = (eachDownload*i)+"-"+((i!=totalChunks-1)?(eachDownload*(i+1)-1):total);
                con[i] = (HttpURLConnection) url.openConnection();
                con[i].setRequestProperty("Range", "bytes="+byteRange);
                con[i].connect();
                 
                // if server doesn't support partial content
                if (con[i].getResponseCode() != 206){
                    System.out.println("Parallel Stream is not supported by the server");
                    this.normal(link);
                    break;
                }
                 
                // Fire the threads to do the job.
                m[i] = new Minions(con[i], r[i]);
                tE.execute(m[i]);
                 
            }
            tE.shutdown();                                  // no new threads are to be created
            tE.awaitTermination(10000, TimeUnit.DAYS);      // wait untill all the threads terminates
                         
            File dest = new File(getFileName(link));
            String filePath = mergeFiles(dest, tempFile);
            System.out.println("Download File Completed.");
            System.out.println("File has been saved to " + filePath);
             
        } catch (InterruptedException ex) {
            Logger.getLogger(DownloadManager.class.getName()).log(Level.SEVERE, null, ex);
        } catch (IOException ex) {
            Logger.getLogger(DownloadManager.class.getName()).log(Level.SEVERE, null, ex);
        }
    }
    private String mergeFiles(File dest, File[] files){
        try {
            byte[] reader = new byte[1024*5];
            int n;
            InputStream is;
            OutputStream os;
             
            os = new BufferedOutputStream(new FileOutputStream(dest));
            for (File iter : files){
                is = new BufferedInputStream(new FileInputStream(iter));
                while((n = is.read(reader)) != -1){
                    os.write(reader, 0, n);
                }
            }
 
            os.flush();
            os.close();
        } catch (IOException ex) {
            Logger.getLogger(DownloadManager.class.getName()).log(Level.SEVERE, null, ex);
        }
        return dest.getAbsolutePath();
    }
     
    // get filename out of the link unable to find one returns 'DownloadedFile' as default
    private String getFileName(String link){
        String fileName = link.substring(link.lastIndexOf("/"), link.length());
        if (fileName.isEmpty()) fileName = "DownloadedFile";
        return fileName;
    }
     
    // these are the threads fetching each of chunks
    private class Minions implements Runnable{
        HttpURLConnection con;
        OutputStream r;
        public Minions(HttpURLConnection acon, OutputStream ar){
            con = acon;
            r = ar;
        }
 
        @Override
        public void run(){
            try {
                BufferedInputStream in = new BufferedInputStream(con.getInputStream());
                int numRead;
         
                byte data[] = new byte[1024];
                while( (numRead = in.read(data, 0, 1024)) != -1){
                    r.write(data, 0, numRead);
                    r.flush();
                }
                 
                r.close();
            } catch (IOException ex) {
                Logger.getLogger(DownloadManager.class.getName()).log(Level.SEVERE, null, ex);
            }
             
        }
    }
}

Firstly, we are opening up a connection to the server just to get the total size of the file. Then divide the size to nearly equal chunks. Each chunks are assigned certain range of data. Then for each chunk, open a connection that is set to fetch his part of file ( range of file). Range property can be used to notify server that we are only interested on certain range of the file, so provide us only those range of bytes. This property can also be used for pause and resume of the download. But not all the servers around, provides support for partial content so we better check the response code provided by the server after we set the Range. If the response code is not 206 ( Partial Content ) then we proceed with the normal download. Else fire the threads and wait for all the threads to complete. Finally merge up all those partial contents. And thats it you have your file on your local machine.