Code Monkey home page Code Monkey logo

github-grab's Introduction

github-grab

More information about the archiving project can be found on the ArchiveTeam wiki: GitHub

Setup instructions

General instructions

Data integrity is very important in Archive Team projects. Please note the following important rules:

We strongly encourage you to join the IRC channel associated with this project in order to be informed about project updates and other important announcements, as well as to be reachable in the event of an issue. The Archive Team Wiki has more information about IRC. We can be found at hackint IRC #gitgud.

If you have any questions or issues during setup, please review the wiki pages or contact us on IRC for troubleshooting information.

Running the project

Archive Team Warrior (recommended for most users)

This and other archiving projects can easily be run using the Archive Team Warrior virtual machine. Follow the instructions on the Archive Team wiki for installing the Warrior, and from the web interface running at http://localhost:8001/, enter the nickname that you want to be shown as on the tracker. There is no registration, just pick a nickname you like. Then, select the GitHub project in the Warrior interface.

Project-specific Docker container (for more advanced users)

Alternatively, more advanced users can also run projects using Docker. While users of the Warrior can switch between projects using a web interface, Docker containers are specific to each project. However, while the Warrior supports a maximum of 6 concurrent items, a Docker container supports a maximum of 20 concurrent items. The instructions below are a short overview. For more information and detailed explanations of the commands, follow the follow the Docker instructions on the Archive Team wiki.

It is advised to use Watchtower to automatically update the project container:

docker run -d --name watchtower --restart=unless-stopped -v /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower --label-enable --cleanup --interval 3600 --include-restarting

after which the project container can be run:

docker run -d --name archiveteam --label=com.centurylinklabs.watchtower.enable=true --log-driver json-file --log-opt max-size=50m --restart=unless-stopped atdr.meo.ws/archiveteam/github-grab --concurrent 1 YOURNICKHERE

Be sure to replace YOURNICKHERE with the nickname that you want to be shown as on the tracker. There is no registration, just pick a nickname you like.

Supporting Archive Team

Behind the scenes Archive Team has infrastructure to run the projects and process the data with. If you would like to help out with the costs of our infrastructure, a donation on our Open Collective would be very welcome.

Issues in the code

If you notice a bug and want to file a bug report, please use the GitHub issues tracker.

Are you a developer? Help write code for us! Look at our developer documentation for details.

Other problems

Have an issue not listed here? Join us on IRC and ask! We can be found at hackint IRC #gitgud.

github-grab's People

Contributors

arkiver2 avatar fusl avatar imerr avatar km09 avatar tech234a avatar tomglass avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

github-grab's Issues

Downloader variable lost somewhere

At some point, there is "Downloader" variable lost somewhere, and this is preventing rsync upload to happen

Starting CheckIP for Item 
Finished CheckIP for Item 
Starting GetItemFromTracker for Item 
Received item 'web:initial:opfx/gitflow' from tracker
Starting PrepareDirectories for Item web:initial:opfx/gitflow
Finished PrepareDirectories for Item web:initial:opfx/gitflow
Starting WgetDownload for Item web:initial:opfx/gitflow
1=404 https://github.com/opfx/gitflow  
Finished WgetDownload for Item web:initial:opfx/gitflow
Starting PrepareStatsForTracker for Item web:initial:opfx/gitflow
Finished PrepareStatsForTracker for Item web:initial:opfx/gitflow
Starting MoveFiles for Item web:initial:opfx/gitflow
Finished MoveFiles for Item web:initial:opfx/gitflow
Starting ChooseTargetAndUpload for Item web:initial:opfx/gitflow
Requesting targets.
Trying target rsync://at-rsync3.phirephly.design:8873/ateam-airsync/:downloader/.
Could not get rsync target.
Failed to upload, retrying...
Requesting targets.
...

Add support for Packages

Feature description: https://github.com/features/packages

Docs: https://docs.github.com/en/packages

Introduction blog post: https://github.blog/2019-05-10-introducing-github-package-registry/

Random examples (all from a search for 'random'):

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.