Collect is a server to collect & archive websites written for NodeJS.

It is intended for people who want to archive individual websites, eg

https://example.com

Collect stores a static copy of the url on your disk

• Features
  • Screenshots
  • Installation
    • Settings
    • Plugins
  • Updating
• Contributing
• Security considerations
• Warning
• Credits
• License

• General
  • Website archiving
  • Website viewing
• Webinterface
  • Add sites to the archive
  • Browse your archive by domain
  • Edit titles of saved pages
  • Delete sites
  • Updates changes on the server in real time
• API
  • Get all sites / sites by domain
  • Get details of saved content
  • Add a site to the archive
  • Delete a site
  • Edit title of a saved page
  • Download all saved pages as an archive (See Backup)
  • For more, see the API documentation

Before installing Collect, please make sure that git, node and npm are installed.

Note: This server was tested on NodeJS v7.7.3 and NodeJS v6.13.1.

Start by cloning the repository to your computer/server:

git clone https://github.com/xarantolus/Collect.git

Go in the Collect directory

cd Collect/Collect

Install dependencies

npm install

To start in production mode (recommended), type

npm start production

or

node app production

When you open the website in your browser, you will notice that you need to authenticate.

To change settings, edit Collect/config.json. There, you can set a port, username, password, id_length and api_token.

It is recommended to use PhantomJS to process the websites after downloading. This ensures that dynamically loaded content is also saved.

To use this, install the node-website-scraper-phantom module.

npm install website-scraper-phantom

This command must be run in the directory that contains the package.json file.

After installing, the server should output PhantomJS will be used to process websites when started.

If the install fails, you cannot use the module and Collect will fall back to the normal way of saving pages.

If you cannot save any pages after installing, remove the module by running

npm uninstall website-scraper-phantom

If you already have Collect installed on your computer/server and want to update to the latest version, follow these steps.

Go in the directory where Collect is installed.

cd /path/to/Collect

You might want to back up your settings file.

Windows:

move Collect\config.json ..\

Linux/Unix:

mv Collect/config.json ../config.json

Download the latest version:

git fetch --all

Apply all changes (this usually overwrites your cookies file, but not the directory where your sites are saved.)

git reset --hard origin/master

Restore the settings file.

Windows:

move ..\config.json Collect\

Linux/Unix:

mv ../config.json Collect/config.json

Go to the directory that contains package.json.

cd Collect

Install all required packages.

npm install

After restarting your server, the new version should be up & running.

See the contributing file.

This project is being developed in Visual Studio 2017.

The following extension(s) are used:

• Bundler & Minifier

• The login system uses plain text. Anyone with (e.g. SSH) access to your server can read it.
• Any site you download can read & set cookies. A downloaded website could send your login cookie to another server. If you host this software in your private network without outside access, everything should be fine even if a cookie gets stolen, but don't take my word for it.
• The connection does by default not use HTTPS.

You're using this tool at your own risk. I am not responsible for any lost data like passwords or websites.

Website Scraper Module: MIT License. I really want to thank the creators. This server is mostly a user interface to this module and would never have been possible without their work.

Website Scraper Module PhantomJS Plugin: MIT License. Makes processing dynamic pages as easy as pie.

The UIkit library: Copyright YOOtheme GmbH under the MIT License. I really love this UI framework.

ArchiverJS: Mit License. node-archiver is a nice module for generating all kind of archive files. It is used to create backups.

Ionicons: MIT License. The icons are really nice. I used the ion-ios-cloudy-outline icon.

Notification Sound: CC0 1.0 Universal License

See the License file