Self Hosted Git Repo Support about gitness HOT 41 CLOSED

Hey Sebastian, this is going to be a fun feature to code!

Task 1 we need to add a hook URL for custom git repositories, /hook/git

Task 2 how do we know which branch or sha to build? should this get passed as a url parameter /hook/git?branch=xxx?

Task 3 can we extract the commit hash, branch, author and commit message without cloning the entire repository? we need this information when we create a commit record in our database, prior to sending the build to our queue

Task 4 can we extract the .drone.yml file without cloning the entire repository? This is important because we clone our repository into a virtual machine. We need the yaml in order to know which virtual machine type to create. we have a chicken and egg issue.

The good news is that, based on this stack overflow answer, it looks like this is possible:
http://stackoverflow.com/questions/14405782/git-fetch-single-file-from-remote-repository-programatically

Task 5 we need a form to create a new generic git repository. we need to assign a slug value to the repository that matches our routing logic /:host/:owner/:name. for github repos this looks like this: github.com/drone/drone. For custom git repositories this could look like custom/git/{{repo name}}

If we can figure out items 2 and 3 this should be easy enough to integrate

from gitness.

I'd like to chime in!
I think for self-host provider like gitlab, this is doable, they do have push web hook with all necessary information available as json via http post.

I'm more concerned on how drone going to manage its repo handler abstraction, since I see handler.RepoAdd (albeit the name) is still coupled specifically to GitHub. Can't wait to see bitbucket support getting integrated here ;)

As for the bare repo, we may have to let user install custom post-receive hook (script) manually, that script would then gather all necessary info before posting it to drone's endpoint, that is, presuming we can provide such script. ;)

Also for the slug, I think it will be easier to manage if we can assign its handler roughly like this:

m.Post("/new/:provider", handler.UserHandler(handler.RepoCreate))
m.Post("/hook/:provider", handler.ErrorHandler(handler.Hook))

But I don't know how the priority goes, so this may be deferred for later. :D

Anyway, kudos for @bradrydzewski and the whole drone.io team for sharing their awesome works! 👍

from gitness.

GitBucket could be of an interest as a first pass at supporting self-hosted repos. I believe their webhook format mirrors GitHub's, so that piece would be much easier.

from gitness.

+1 GitLab support.

GitLab Post-Receive Hook json example:
--> https://github.com/gitlabhq/gitlabhq/blob/5-4-stable/app/views/hooks/_data_ex.html.erb

from gitness.

Seems like a great opportunity to start pushing for a starndard webhook payload for this sort of thing.

from gitness.

👍

from gitness.

+1 for GitLab support, would be very useful.

This could be useful: https://github.com/plouc/go-gitlab-client

from gitness.

+1 GitLab support.

from gitness.

@ajcrowe looks like a good start to the API. Anyone willing to add post-commit hooks and ssh key endpoints?

from gitness.

+1 GitLab support

from gitness.

sounds like GitLab support is a must have feature! I created an issue specifically to track progress on GitLab support. Let's move all related discussion to issue #53

from gitness.

👍 on this!
Actually, I'm currently working on refactoring repository handler at my repo-handling branch It's more like an implementation of what I said above

So, this is basically extending @FooBarWidget's work at #34 by using more generic approach to add repository, it will check the host parameter and then select appropriate repository type from the presets, and fallback to custom repo if it's not there. Currently the html templates were mostly copy-pasted though.

I think this could be a precursor to help #2, and #53. But I would like to know if this is favorable or not. And maybe some thoughts about the caveats if we have way too generic method to add repository, since with this patch you may call /new/<anything> and you can expect it to work correctly. Afterall, we only need correct repository url to clone it.

But maybe I missed something, so some inputs would certainly appreciated.

Thanks!

from gitness.

@fudanchii I definitely think it makes sense to do /new/:host

The questions I have are more around long term maintenance. Based on the popularity of this feature request, will we end up supporting 5 different hosts? 10? 15? Are there some things we can do to simplify maintenance and allow people to more easily add hosts?

• should we create some sort of type Host interface?
• should we register them in a map[string]Host? kind of like plugins?
• should we separate them into a separate package?

I don't have the answers. Just some food for thought

from gitness.

Yep, I think I want our Host interface to cover API provided by each host, How about normalizing the endpoints as:

• Check for private / public repo, currently implemented at go-github, and being used at handler.RepoCreateGithub. We need this to decide repository's url scheme, which in turn used to decide whether we must inject our pubkey or not (see below), this may be a noop for custom repo.
• Pubkey injection, add drone's pubkey to the host via API, or we can just direct user to <slug>/keys page and tell user to add it manually, given the host doesn't have its API implemented.
• Hook injection, it is known that github, gitlab, and bitbucket has API for hook addition. For custom host we can just display our hook endpoint and let user install it manually.
• Payload parser, I haven't check with go-github, but apparently GitHub let us choose its hook payload mime-type. But I believe gitlab only support json, haven't check bitbucket though. We may have to follow one established format (either github, gitlab, bitbucket ?) for any other custom repo.
• Event notification, success build, failed build, etc. May be a noop for custom repo.

Thoughts?

I'll try to tinker some more on this,

from gitness.

I like that! It could be great if we turned this feature into a plugin so it could be simple to add custom Host plugin

from gitness.

@fudanchii we might want to delegate the entire request to the Host. Each host may want to render the page slightly differently. Also linking (via OAuth) is very different in GitHub as opposed to Bitbucket.

type Host struct {
    // returns an html form used to create a new repository
    func New(w http.ResponseWriter, r *http.Request, u *User) error

    // saves a new repository.
    func Save(w http.ResponseWriter, r *http.Request, u *User) error

    // links to the host using something like OAuth, and also
    // processes the host redirect response.
    func Link(w http.ResponseWriter, r *http.Request, u *User) error

    // processes a build request hook
    func Hook(w http.ResponseWriter, r *http.Request) error
}

var hosts = map[string]Hook{}

func Hook(w http.ResponseWriter, r *http.Request) error {
    name := r.FormValue("host")
    host, ok := hosts[name]
    if !ok {
        return fmt.Errorf("Unable to find host %s", name) 
    }

    return host.Hook(w, r)
}

thoughts?

from gitness.

Ah, I see.
It still on the top off my head, but I was thinking not to delegate responsibility upon requests,
and hopefully we can minimize duplicated code between Host implementation, and it's easier to implement the hosts.

For hook,
It's something along this line:

// subtyping model.Repo
type GithubRepo Repo
type BitbucketRepo Repo

type HostRepo interface {
    ParseHook() hookStruct
    FetchBuildScript() string
}

func (g *GithubRepo) ParseHook() string { 
    // GitHub specific Hook
}

func NewGitHubRepo() { }

var HostMap map[string]HostRepo

// Hook router
func Hook(w http.ResponseWriter, r *http.Request) error {
    var hook hookStruct
    var rawScript string

    hostName := r.FormValue(":host")
    id := r.FormValue("id")
    payload := r.FormValue("payload")

    repo, err := database.GetRepoSlug(id)
    //...

    if host, ok := HostMap[hostName]; ok {
        hook = host.ParseHook(payload)
        rawScript = host.FetchBuildScript(repo, hook.Head_id)
    } else {
        // error
    }

    commit := &Commit{}
    commit.RepoID = repo.ID
    commit.Branch = hook.Branch()
    commit.Hash = hook.Head.Id
    commit.Status = "Pending"
    commit.Created = time.Now().UTC()
    // ...

    buildscript, err := script.ParseBuild([]bye(rawScript))
    // ...

    // database.SaveCommit

    // ...

    // database.SaveBuild

    // add to queue

}

Is it possible to extract go-github hook struct format and use it as a reference for hookStruct?

from gitness.

I definitely agree that we should minimize code duplication. Initially I considered the exact same approach that you proposed. I almost started coding it :) but I was worried about the variation in data structures.

there is variation in the data input:

• adding a generic git repository requires url, and possibly username and password
• adding a github repository requires owner and name
• adding a bitbucket repository requires owner and name
• no clue how gitlab works yet

there is variation when linking accounts:

• linking a github account uses oauth2
• linking a bitbucket account uses oauth1a
• not implemented for generic git
• no clue how gitlab works yet

there is variation in hook payloads

• the bitbucket hook payload is completely different from the github hook payload
• the github payload is complex, does it make sense for generic git?

We may end up integrating with 10 different hosting providers in the next few months. I'm just worried that we don't know enough about them yet to properly abstract all the behavior.

from gitness.

Okay, I already have something in mind, will try to tinker with this some more.
Maybe integrating github enterprise and the planned bitbucket support should take more priority than this? At the very least we have something functional, and actually we may have more information after they get integrated.

We can always get this rebased from there. :)

On the other note, I still believe that given we have consistent model.Repo and model.Commit struct, we can put all implementation details for Host under single interface, and use transitional struct to map between internal drone's data-structure and each Host implementation data-structure.

from gitness.

Yes, good idea 👍 We'll try to get Bitbucket integrated asap, without any abstraction, so that we have some concrete examples to work from. That will make it much easier to define the single interface, as you mentioned, and make sure we got it right

from gitness.

• adding a gitlab repository requires owner (via auth_token) and project id (see https://github.com/gitlabhq/gitlabhq/blob/master/doc/api/projects.md#add-project-hook)
• linking a gitlab account required a username or email and a password (see https://github.com/gitlabhq/gitlabhq/blob/master/doc/api/session.md) which provide a auth_token when connected

from gitness.

This would be very cool. I was thinking about using Drone for verifying patches submitted into Gerrit. Doesn't mean Drone must support Gerrit, I don't mind putting something between to convert Gerrit event stream into Drone API calls or something.

In any case, keep up this great work 👍

from gitness.

Hey guys! Any update on this? :)

from gitness.

Any chance of getting generic (i.e. non-hosted / bitbucket / github / stash / gitlab, etc) Git repo support?

edit: asking since I'd like to use Drone in our company, and we're using a self-hosted generic git service. The lack of support for generic solutions in Drone is preventing me from getting it adopted.

from gitness.

I suppose this could be done if you're able to install a post-receive hook that would ping the Drone server to run tests. Seems like this wouldn't be too crazy to implement on the Drone side, but there should probably be a new issue for this.

from gitness.

i saw there was someone trying to add stash into exp branch
but now it has been merged and i can't find it anymore
so is this dropped now or not?

EDIT:
nevermind found the pull request already #567

from gitness.

The latest version of Drone completely relies on 3rd party version control systems (ie GitHub, Bitbucket, etc) for user accounts, user login and repository permissions. These capabilities were completely removed from Drone. This means that Drone will not support generic Git repositories.

from gitness.

I will add that all is not lost. We are planning on adding dynamic plugins (using JavaScript) that you may be able to use to hack something together. It will just take a bit of creativity to fully implement the required methods:

type Remote interface {
    // Authorize handles authentication with thrid party remote systems,
    // such as github or bitbucket, and returns user data.
    Authorize(w http.ResponseWriter, r *http.Request) (*model.Login, error)

    // GetRepos fetches all repositories that the specified
    // user has access to in the remote system.
    GetRepos(user *model.User) ([]*model.Repo, error)

    // GetScript fetches the build script (.drone.yml) from the remote
    // repository and returns in string format.
    GetScript(user *model.User, repo *model.Repo, hook *model.Hook) ([]byte, error)

    // Activate activates a repository by creating the post-commit hook and
    // adding the SSH deploy key, if applicable.
    Activate(user *model.User, repo *model.Repo, link string) error

    // ParseHook parses the post-commit hook from the Request body
    // and returns the required data in a standard format.
    ParseHook(r *http.Request) (*model.Hook, error)

from gitness.

Sad, sad day indeed.... 😢 😭

from gitness.

well there go my dream of moving away from bamboo

from gitness.

@ramonski FYI, I have written a specific extension to use drone with our self-hosted mercurial repositories:
https://github.com/cedk/drone/tree/trypod
http://hg.tryton.org/trypod
It should not be too difficult to adapt it for other custom usage.

from gitness.

well there go my dream of moving away from bamboo

I think someone could easily create a deamon that implements the minimum required REST endpoints (login, get repository, upload ssh key) to hook into Drone. This daemon could sit on your Git server, or it could probably even sit on your Drone server and use ssh to communicate with your Git server.

So your dream isn't dead :)

It will just require an extra module, which won't be included in the core Drone codebase.

If nobody wants to create this daemon, I will do it. I just have a pretty long backlog at the moment so it would take a few months. If someone wants to tackle this now I'm happy to offer my assistance.

from gitness.

@bradrydzewski @cedk - I've made a fork of the Drone project in which I'm starting a plugin for Stash. I'll definitely be taking up your offers for help. I'll see how far I get in the next few days before I call for help though. I'm using the BitBucket plugin and the Trypod plugins as a base reference for now.

from gitness.

Really cool that you are trying it! Thanks and good luck! :)

from gitness.

Have there been any developments since this issue was closed? I'm interested in using drone for our deployments where I work, but all our repos are self-hosted.

from gitness.

@ChiperSoft the upcoming 0.4 release of Drone will feature a plugin model for hooking up custom remote systems. One just needs to satisfy the following interface:
https://github.com/drone/drone/blob/0.4.0/pkg/remote/remote.go#L9

Self-hosted raw git repositories won't be officially supported, however, I'm confident they can be added through an external plugin once 0.4 is ready.

from gitness.

#10 (comment)

Now it's 0.8 and I can't find how to make a external plugin to support my custom remote.

from gitness.

@spacemeowx2 ypu should implement the Remote interface described in https://github.com/drone/drone/blob/master/remote/remote.go (see other implementation in the same folder).

from gitness.

@mavimo So after I writing it how can I plugin it into drone without compile again?

from gitness.

Currently it's not possible without recompilation, but maybe there will follow some API comparable to secrets and registry.

from gitness.

Got it, thank you.

from gitness.