Navigatrix.net - A Voyager's Companion
http://navigatrix.net/

Download Entire Websites In A Single Bound
http://navigatrix.net/viewtopic.php?f=10&t=505
Page 1 of 1

Author:  Moe [ 12 Sep 2013, 00:25 ]
Post subject:  Download Entire Websites In A Single Bound

Say there is a website, or a part of a website, that you want to read...later.

For example, you are going back to the boat in the next 15 minutes and will set sail in the morning, but you want the well written and enjoyable Navigatrix Documentation to read when you have time, but maybe not an internet connection.

Download it and read it off the clock on your own time offline.

It's easy.
Code:
mkdir offline && cd offline
to make a directory to hold your treasure and change to it. (or wherever your hearts desire)
Code:
wget \
     -w 20 --random-wait \
     --recursive \
     --no-clobber \
     --page-requisites \
     --convert-links \
     --domains navigatrix.net \
     --no-parent  navigatrix.net/nxdoc/
This command downloads the Web site http://navigatrix.net/nxdoc.

The options are:

  • -w 20 --random-wait: to avoid getting banned for too many requests in too short a time. The number (20) is time in seconds. This can be reduced, or the line eliminated if you laugh in the face of danger.

  • --recursive: download the entire Web site from entry point on down.

  • --no-clobber: don't overwrite any existing files (used in case the download is interrupted and
    resumed).

  • --page-requisites: get all the elements that compose the page (images, CSS and so on).

  • --convert-links: convert links so that they work locally, off-line.

  • --domains navigatrix.net: don't follow links outside navigatrix.net.

  • --no-parent: don't follow links outside the directory nxdoc. and inadvertantly download the entire site
Fire away and it will
Quote:
FINISHED --2013-09-12 15:21:43--
Total wall clock time: 23s <----no random wait time; 22m 23s with random wait.
Downloaded: 65 files, 2.4M in 7.1s (347 KB/s)
Converting navigatrix.net/nxdoc/index.html... 22-0
Converting navigatrix.net/nxdoc/02_getting_nx.html... 9-0
Converting navigatrix.net/nxdoc/01_nx_what_why.html... 4-0
Converting navigatrix.net/nxdoc/04_applications.html... 17-0
Converting navigatrix.net/nxdoc/05_gps.html... 4-0
Converting navigatrix.net/nxdoc/06_config_nx.html... 38-0
Converting navigatrix.net/nxdoc/08_nx_support_db.html... 3-0
Converting navigatrix.net/nxdoc/03_installing_nx.html... 22-0
Converting navigatrix.net/nxdoc/07_hidden_treasures.html... 4-0
Converting navigatrix.net/nxdoc/nx_doc_stylesheet.css... 2-0
Converted 10 files in 0.003 seconds.
Then you can either navigate to the directory with the File Manager or
Code:
wadda@mini:~/offline$ firefox -new-window file:////home/$USER/offline/navigatrix.net/nxdoc/index.html
Then again you could just use the Scrapbook extension that is pre-installed in Firefox. It's just as effective, but not as much fun.

Page 1 of 1 All times are UTC - 5 hours
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/