not downloading the latest version of the website about wayback-machine-downloader HOT 4 CLOSED

It appears that the page between 20110827210108 and 20121007234751 wasn't modified then become a 404 in 20130110230837. Wayback Machine will return the first version of the page it encountered if it wasn't modified after.

Ref: http://web.archive.org/cdx/search/cdx?url=http://tohokinemakan.jp/index.html&gzip=false

We do had a bug with the 404 of 2013. I've pushed a fix to 1.0.0. If you update to 1.0.0, it should show up with the -a/--all flag.

➜  gem git:(master) ✗ bin/wayback_machine_downloader http://tohokinemakan.jp/index.html -l -a
[
{"file_url":"http://tohokinemakan.jp:80/index.html","timestamp":20130110230837,"file_id":"index.html"},
]

from wayback-machine-downloader.

I have upgraded to version 1.0.0. It seems that I need the --all option for 200 pages as well.
For example: http://web.archive.org/cdx/search/cdx?url=http://tohokinemakan.jp/information.html&gzip=false
This page has 2 different version (with different content). Both of them have 200 response code. But without --all the old one will be fetched.

$ wayback_machine_downloader http://tohokinemakan.jp/information.html -a -l
[
{"file_url":"http://tohokinemakan.jp:80/information.html","timestamp":20121008235908,"file_id":"information.html"},
]

$ wayback_machine_downloader http://tohokinemakan.jp/information.html -l
[
{"file_url":"http://tohokinemakan.jp:80/information.html","timestamp":20100815073403,"file_id":"information.html"},
]

from wayback-machine-downloader.

Try to update to 1.1.2, it should download the correct last version of files with or without the --all flag. It's also now capable of parsing more deeply snapshot pages if you have a large website.

from wayback-machine-downloader.

Fixed in 7eedc1a.

from wayback-machine-downloader.