Code Monkey home page Code Monkey logo

gitmesh's Introduction

gitmesh

What's this?

This is an experimental project to make a distributed git server cluster. The main goal of this project is to find a reasonable way to add scalability and redundancy to git repositories. Basic idea is locating git repositories on multiple nodes, and proxy requests from git clients to appropriate nodes. This approach is similar to GitHub's DGit (it has been renamed to Spokes).

The distributed gitserver cluster consists of following two kinds of servers:

  • Controller server

    This is a front server of the cluster. It manages repository servers and proxy requests from git clients to appropriate repository servers. We can make redundant it by setup multiple instances with a load balancer.

  • Repository server

    This is a storage server of the cluster. Git repositories are located on this kind of servers actually. We can add any number of repository server instances to the cluster.

Architecture

This project is still under development phase, but if you are interested, please try it. Any feedback is welcome!

Setup

You can run a minimal gitmesh cluster on docker and docker-compose.

Prerequisites

  • Java 8
  • sbt
  • Docker

Build docker images

First, run the following command at the root of this repository to build docker images.

$ ./build-docker.sh

This script builds following docker images:

  • gitmesh-console
  • gitmesh-controller-server
  • gitmesh-repository-server

Run a cluster using docker-compose

Next, run the following command to start a minimal gitmesh cluster (console x 1, controller x 1, mysql x 1, repository x 2).

$ ./run-docker-compose.sh

This command executes docker-compose internally. So you can run a cluster with another structure by modifying docker-compose.yml.

Web console is available at http://localhost:8080, and endpoints are available at http://localhost:8081. See this section to know how you can use a gitmesh cluster as a Git server.

Build and run from source code

You can also build and run gitmesh from source code for now. This guide shows how to run the cluster with minimum configuration (one controller server and one repository server on a single machine).

Prerequisites

  • Java 8
  • sbt
  • MySQL

You have to create an empty database before run the controller server. If you use docker, run a MySQL container as follows:

$ docker pull mysql
$ docker run --name mysql -e MYSQL_ROOT_PASSWORD=mysql MYSQL_DATABASE=gitmesh -d -p 3306:3306 mysql

Start the controller server

Modify gitmesh-controller-server/src/main/resources/application.conf for your environment, and run the controller server as follows:

$ cd gitmesh-controller-server
$ sbt ~jetty:start

The controller server is started on port 8081 in default. Tables are created automatically in the database configured in application.conf.

Start the repository server

Modify gitmesh-repository-server/src/main/resources/application.conf for your environment, and run the repository server as follows:

$ cd gitmesh-repository-server
$ sbt ~jetty:start

The repository server is started on port 8082 in default.

Check the cluster operation

Let's create a new repository and push a commit using git command to check the cluster operation.

You can create a new repository via Web API. In this case, a repository url is http://localhost:8081/git/test.git.

$ curl -XPOST http://localhost:8081/api/repos/test

Create a local repository and push a first commit to the remote repository on the cluster:

$ mkdir test
$ cd test
$ git init
$ touch README.md
$ git add .
$ git commit -m 'first commit'
$ git remote add origin http://localhost:8081/git/test.git
$ git push origin master

The remote repository is created under the directory configured in the repository server's application.conf.

Web console

gitmesh also offers a web console for administrators.

Web console

Run followng commands to start the web console.

$ cd gitmesh-console
$ npm install
$ npm run dev

The web console is available at http://localhost:8080/. You can check repositories and nodes status, and create or delete repositories on this console.

gitmesh's People

Contributors

maaaato avatar takezoe avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

gitmesh's Issues

Stop to use repository name as directory name

It causes troubles when a process attempts to delete a directory of repository but another process is grabbing files in the directory. In particular, we often saw this trouble in Windows.

Using a unique id (like UUID) as a directory name, it makes possible to delete a repository by deleting a record from the database. We can able to delete a physical directory later like a garbage collector.

Load balancing for read requests

While current gitmesh implementation directs all read requests to the primary node, when a repository has replicas, we can decrease load of the primary node by directing these requests to replicas.

Repo not created when node has large disk space

I tried with two node. Ubuntu 16.10 x64 and Raspberry pi 3.
But Ubuntu node doesn't create repo.

kounoike@myserver:~/distributed-git-server/dgit-controller-server$ cur -d "" http://localhost:8080/api/repos/dgit_test
kounoike@myserver:~/distributed-git-server/dgit-controller-server$ curl http://localhost:8080/api/nodes
[{"node":"http://localhost:8081","diskUsage":0.9790734247221924,"repos":[]},{"node":"http://raspberrypi.local:8081","diskUsage":0.5193819272268632,"repos":["dgit_test"]}]
kounoike@myserver:~/distributed-git-server/dgit-controller-server$ df /tmp
Filesystem      1K-blocks     Used  Available Use% Mounted on
/dev/sda2      2859026248 59829632 2653943344   3% /

This result says "Ubuntu node(localhost) has large diskUsage". Actually, myserver has large free space.

by this line, diskUsage calculated by free / total. It means "Free %". It maybe bug.

https://github.com/takezoe/distributed-git-server/blob/95fcce4e340dd089601bcd97d42253e80f08825f/dgit-repository-server/src/main/scala/com/github/takezoe/dgit/repository/InitializeListener.scala#L47

Docker support

Offering docker images would make possible to experiment gitmesh cluster on Kubernetes.

Can't create replica of empty repository

Can't create replica of empty repository when repository servers are added after repository creation. It tries to git clone to create replica even if a source repository is empty, and it fails.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.