Biological and Chemical Oceanography Data Management Office about datasets HOT 7 OPEN

NGO. Part of Woods Hole Oceanographic Institution.

from datasets.

from datasets.

Okay. I only wanted to note it.

from datasets.

While this is true, it is not automatically the case that an NGO will
have the funds to continue maintaining or maintaining access to such
datasets, even if hostility towards government counterparts is less than
we might fear. All it would take would be a catastrophic cut in funding
for such to these NGOs. This may even apply to some universities.

Agreed, but federally located data is probably at a slightly higher risk and in a nearer timeframe.

from datasets.

currently crawling via

wget     --mirror     --warc-file=www.bvo-dmo.org.warc     --warc-cdx \
  --page-requisites     --html-extension     --convert-links \
  --execute robots=off     --directory-prefix=.     --span-hosts \
  --domains=bco-dmo.org,usjgofs.whoi.edu \
  --exclude-domains=mapservice.bco-dmo.org \
  --user-agent='Mozilla (mailto:[email protected])' \
  --wait=10     --random-wait     http://www.bco-dmo.org/data

from datasets.

@Flyingmana , shouldn't you also include a -N and -m in case your stream is interrupted and you have to restart?

from datasets.

@gabefair Unfortunately -N is not compatible with --warc-file

from datasets.