Light

tuubes / datatractor Goto Github PK

View Code? Open in Web Editor NEW

2.0 2.0 1.0 280 KB

Extract game data from web sources

License: GNU General Public License v3.0

Python 100.00%

tuubes web-scraping extractor minecraft-tool

datatractor's Introduction

DataTractor

Extract game data from web sources.

How to use

Setup & Run

You need python version 3. To install the package, run:

python setup.py install

Tip: you can add the --user option to install it for you only, without root permissions.

Then, in the repo's main directory, simply run the xtract script:

python xtract.py -v game_version -o output_directory

-o might be omitted, see below.

Options

Type	Option	Description
Mandatory	`-v version`	Sets the game version
Optional	`-o dir`	Sets the output directory, default is `./out/generated_$v` where `$v` is the game version
Optional	`-p` or `--packets`	Enables the packets extractor
Optional	`-b` or `--blocks`	Enables the blocks extractor
Optional	`--nocache`	Disables the HTTP cache
Optional	`--cachetime seconds`	Sets the cache timeout in seconds, default is 300s (5 minutes)

If no extractor is specified, all the available extractors will run.

Example

python xtract.py -v 1.12.2 --packets

This will generate all the packets classes for Minecraft version 1.12.2.

datatractor's People

Contributors

Stargazers

Watchers

Forkers

extrasaucestudio

datatractor's Issues

Handle compact position fields

Location: Long: 26 bits for x, 12 bits for y, 26 bits for z

It would be nice to provide a construction method and a getter that converts the long to the location.

Support NBT tags

Assume that TuubesCraft use flow nbt with CompoundTag objects?

Headings: Close sub-section when its parent ends

<h1>Section 1</h1>
<h2>Section 1.1</h2>
<h1>Section 2</h1> <!-- This should end both section 1 and 1.1 -->

Extract block data

Possible data sources:

For each block:

unique, cross-version Tuubes name
minecraft id, including "metadata"
minecraft name

Add a nice main program

Update the readme

Once the main program is available (see #5), explain how to use it.

Construct packet data from wiki.vg

http://wiki.vg/Protocol

Each packet is described in a table with 3 columns : field, type, description.

Rewrite the Scala generator to make good use of the new extracted information

See #13

Handle angle fields

See http://wiki.vg/Protocol#Data_types

Add an option to cache requests

With https://realpython.com/caching-external-api-requests/

Parse html tables

Scala generator should register the packets to a Tuubes Protocol

See Protocol.scala

Extract compound fields

For instance the arrays that contain more than one element, as in http://wiki.vg/Protocol#Player_List_Item

Ability to generate scala classes from packet data

Detect the length of strings and arrays

Often, the strings and arrays are preceded by a vvvLength field, where vvv is the name of the array/string field. This could be detected by DataTractor.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.