vcs-python / vcspull Goto Github PK
View Code? Open in Web Editor NEW๐ Synchronize projects via yaml/json manifest. Built using `libvcs`.
Home Page: https://vcspull.git-pull.com/
License: MIT License
๐ Synchronize projects via yaml/json manifest. Built using `libvcs`.
Home Page: https://vcspull.git-pull.com/
License: MIT License
By default
~/path/to/dirs:
project_name:
repo: 'git+repo_url'
The directory checked out would be ~/path/to/dirs/project_name
.
Allow overriding the dir, so project_name
could download to ~/path/to/dirs/custom_dir
~/path/to/dirs:
project_name:
dir: custom_dir
repo: 'git+repo_url'
The hackable repo manager, built on libvcs
. With great defaults, and presets if those don't work
Finds configs (based on template loaders in django) and processes them them
CONFIG_LOADERS = [
{
"BACKEND": "vcspull.configs.loaders.json.JSONLoader",
"XDG_DIRS": True,
# optionally
"DIRS": [
"/custom/lookup/path",
],
},
{"BACKEND": "vcspull.configs.loaders.json.YAMLLoader", "XDG_DIRS": True},
{"BACKEND": "vcspull.configs.loaders.json.TOMLLoader", "XDG_DIRS": True},
{"BACKEND": "vcspull.configs.loaders.json.PythonLoader", "XDG_DIRS": True},
]
(inspired by context processors / middleware):
helpers that expand config, e.g.
repo: 'git+https://github.com/vcs-python/vcspull'
repo:
remotes:
origin:
push: 'git+https://github.com/vcs-python/vcspull'
pull: 'git+https://github.com/vcs-python/vcspull'
CONFIG_PROCESSORS = [
# before:
# repo: 'git+https://github.com/vcs-python/vcspull'
# after:
# repo:
# remotes:
# origin:
# push: 'git+https://github.com/vcs-python/vcspull'
# pull: 'git+https://github.com/vcs-python/vcspull'
"vcspull.configs.processors.simple_repo",
# before:
# repo:
# origin: 'git+https://github.com/vcs-python/vcspull'
# after:
# repo:
# origin:
# push: 'git+https://github.com/vcs-python/vcspull'
# pull: 'git+https://github.com/vcs-python/vcspull'
"vcspull.configs.processors.simple_remote",
]
e.g. pip
, npm
/yarn
, normal vcs URLs
Function to handle merge behavior when same repo name exists. `
'MERGE_BEHAVIOR: 'vcspull.config.merge.default'
default = deep_extend
def overwrite(a, b, config_a, config_b):
return b
def extend(a, b, config_a, config_b):
a.extend(b)
def deep_extend(a, b, config_a, config_b):
# deep merge code
above: hopefully if we pass config_a and config_b, if at all, they'd be immutable
see:https://github.com/TehShrike/deepmerge#custommerge
VCS_BACKENDS = {
'git': 'libvcs.git.Git',
'svn': 'libvcs.svn.Svn'
'hg': 'libvcs.hg.Mercurial'
}
Allows people to override with custom behavior, esp. overriding sync()
{
"extends": "vcspull.settings.default",
"CONFIG_LOADERS": [],
"CONFIG_PROCESSORS": []
}
json
wrapper for jsoncStrip removing trailing commas and comments
VCSPull(settings='str_or_dict', init=True)
vp = VCSPull(settings='vcspull.config.default')
vp = VCSPull(settings='~/.config/vcspull/settings.py')
vp = VCSPull(settings='~/.config/vcspull/settings.json')
Without initializing / finding configs:
vp = VCSPull(settings='~/.config/vcspull/settings.json', init=False)
vp.settings
vp.init() # Finds configs via loaders, parses via backends, builds index of repos
use case:
when creating a general system backup, i feed it with a file tilling it what folders to ignore (like steam games, caches
i would like to extend that file with the names of the vcs repos, because those i "backup" with the actual dvcs
https://github.com/vcs-python/vcspull/issues/new
Perhaps a possibility in the future, also, may be a vcspull run-script <script>
command
This could make it possible to interact with vcspull
and libvcs
based on the environmental context.
vcspull run <script>
This allows running against your environment settings.
System configs: XDG_CONFIG_DIRS/scripts/my_script.py
, e.g. /etc/tmuxp/scripts/my_script.py
or vcspull run my_script
User configs: You can also have $XDG_CONFIG_DIR/scripts/my_script.py
and use vcspull run my_script
Also support XDG_CONFIG_DIRS
for global configs on systems
def run(repos, configs, *args, **kwargs):
# list of repo objects that can synced
blender =
cd to repository by name
Because 'str'
can't be decoded.
git+https://github.com/i3/i3.git
isn't chopping off the git+
when adding remotes.
Conflicts
Catch conflicts of same name and:
Without this, merging can't be done #231)
Catch same repo, same remote name, different URL
VERSION: vcspull 0.2.2
Using a relative path instead of an absolute path for the workspace root-dir seems to lead to infinite recursion after checkout of first project.
EXAMPLE: vcspull.yaml (needs: real Git repository)
projects:
foo: "git+https://github.com/the_user/foo.git"
$ vcspull -c vcspull.cfg co
NOTE:
It would be nice to use the HERE
idiom (the directory where the config-file is located).
https://github.com/vcs-python/vcspull/blob/master/docs/config-generation.md?plain=1#L112
The link in this line leads me to a "404 Not Found"
Deprecate support for python 2.x
re: #333 (comment)
<folder>:
<repoName>: '<repoUrl>'
GitRepo(path=pathlib.Path('~/work') / pathlib.Path('libtmux'), **options)
SVNRepo(path=pathlib.Path('~/work') / pathlib.Path('MySVNProject'), **options)
~/work/:
libtmux:
vcs: 'git'
options:
origin:
url: 'https://github.com/vcs-python/vcspull.git'
libtmux_mirrored_remotes:
vcs: 'git'
options:
origin:
fetch: 'https://github.com/vcs-python/vcspull.git'
push: 'git+ssh//[email protected]/vcs-python/vcspull.git'
MySVNProject:
vcs: 'svn'
options:
url: http://unladen-swallow.googlecode.com/svn/trunk/
search repos
To simplify things, lets make the format verbose.
/path/to/parent_dir:
- name: 'default to vcs default' # optional
- url: raw_vcs_url # no more ambiguity over how to handle auth # required
- vcs: 'git' # or 'hg', 'svn' # required
- prompt_auth: true/false (whether to prompt or skip if not authenticated) # default true
- rev: afafaf # changeset, revision number or branch, defaults to head
- username: u # optional
- password: p # optional
- externals: True # checkout submodules/svn externals
|django-rq-dashboard| (git) Adding remote origin <git+https://github.com/brutasse/django-rq-dashboard.git>
fatal: remote origin already exists.
Traceback (most recent call last):
File "/home/t/.local/bin/vcspull", line 8, in <module>
sys.exit(cli.cli())
File "/home/t/.local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/home/t/.local/lib/python3.7/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/home/t/.local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/t/.local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/t/.local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/t/.local/lib/python3.7/site-packages/vcspull/cli.py", line 117, in update
list(map(update_repo, found_repos))
File "/home/t/.local/lib/python3.7/site-packages/vcspull/cli.py", line 159, in update_repo
r.update_repo() # Creates repo if not exists and fetches
File "/home/t/.local/lib/python3.7/site-packages/libvcs/git.py", line 139, in update_repo
self.obtain()
File "/home/t/.local/lib/python3.7/site-packages/libvcs/git.py", line 127, in obtain
self.remote_set(name=r['remote_name'], url=r['url'])
File "/home/t/.local/lib/python3.7/site-packages/libvcs/git.py", line 331, in remote_set
self.run(['remote', 'add', name, url])
File "/home/t/.local/lib/python3.7/site-packages/libvcs/base.py", line 108, in run
cwd=cwd,
File "/home/t/.local/lib/python3.7/site-packages/libvcs/util.py", line 178, in run
raise exc.CommandError(output=output, returncode=code, cmd=cmd)
libvcs.exc.CommandError: Command failed with code 128: git remote add origin https://github.com/brutasse/django-rq-dashboard.git
Hi ,
I have this problem :
$ vcspull
Traceback (most recent call last):
File "/usr/lib64/python2.7/logging/init.py", line 851, in emit
msg = self.format(record)
File "/usr/lib64/python2.7/logging/init.py", line 724, in format
return fmt.format(record)
File "/usr/lib64/python2.7/site-packages/vcspull/log.py", line 76, in format
prefix = self.template(record) % record.dict
File "/usr/lib64/python2.7/site-packages/vcspull/log.py", line 115, in debug_log_template
reset + levelname + asctime + name + module_funcName + lineno + reset
TypeError: cannot concatenate 'str' and 'list' objects
Logged from file cli.py, line 115
Thanks.
Currently when using this tool one has to edit the config file and then run the vcspull
command (maybe with filter on the new repo name) to get the new checkout.
I would propose to add some sort of command where you can do something like vcspull add <repoUrl>
which will then generate an entry similar to:
<folder>:
<repoName>: '<repoUrl>'
in the existing config. would be the folder from which the call is executed Obviously if the already exist it should just add an entry to that block. The should be extractable from the , if not it needs to be passed in the call.
Once this config update was done vcspull
should be able to run its normal "checkout/update" logic on that new entry to create it.
That way you can add a new entry in the vcspull config and get the checkout right away.
Any thoughts on this?
Best regards
Segaja
Move instances of tmpdir
aka py.path.local
to tmp_path
pathlib.Path
from stdlib
companion: vcs-python/libvcs#304
โฏ vcspull vcspull tmuxp cihai cli unihan-etl unihan-db libvcs libtmux alagitpull alagitpull
|vcspull| (git) Updating to 'master'.
|tmuxp| (git) Updating to 'master'.
|cihai| (git) Updating to 'master'.
|cli| (git) Updating to 'master'.
|unihan-etl| (git) Updating to 'master'.
|unihan-db| (git) Updating to 'master'.
|libvcs| (git) Updating to 'master'.
|libtmux| (git) Updating to 'master'.
|alagitpull| (git) Updating to 'master'.
Traceback (most recent call last):
File "/home/x/.local/bin/vcspull", line 11, in <module>
sys.exit(cli.cli())
File "/home/x/.local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/home/x/.local/lib/python3.6/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/home/x/.local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/x/.local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/x/.local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/home/x/.local/lib/python3.6/site-packages/vcspull/cli.py", line 107, in update
list(map(update_repo, found_repos))
File "/home/x/.local/lib/python3.6/site-packages/vcspull/cli.py", line 120, in update_repo
repo_dict['pip_url'] = repo_dict.pop('url')
KeyError: 'url'
โฏ vcspull --version
vcspull 1.1.0
~
โฏ python --version
Python 2.7.15rc1
Hey @tony ,
how willing would you be to include some contribution scripts in the repository to help generate an initial config file?
I use vcspull to manage ~70 repos in my work place all under one gitlab group. I wrote a small shell script to scan this group and generate an initial config file out of it.
Initially I thought to just share this internally in my company, but if you are willing we could add this to the repository (after i made it a bit more abstract).
Please let me know your thoughts about this.
Best regards
Segaja
gevent, threading vcs updates
Assuming two repos with the same name, same URL, merge remotes
This way work/mycompany.yaml's remote repo will merge with the generic one at vcspull.yaml
@jcfr Are you ok with licensing your contributions to this project and libvcs as MIT?
The current license for vcspull and libvcs is BSD 3-Clause.
If you agree to licensing your contribution(s) to vcspull and libvcs under MIT, please reply if its ok.
(doing this for another project too tmux-python/libtmux#46 tmux-python/tmuxp#264)
It would be nice to have a --dry-run
option to see what would happen if I used this configuration. I see two reasons why this would be nice:
|volo| (git) Already up-to-date. Already up-to-date.
Traceback (most recent call last):
File "/usr/lib/python2.7/logging/__init__.py", line 851, in emit
msg = self.format(record)
File "/usr/lib/python2.7/logging/__init__.py", line 724, in format
return fmt.format(record)
File "/usr/local/lib/python2.7/dist-packages/pullv/log.py", line 60, in format
prefix = self.template(record) % record.__dict__
ValueError: unsupported format character ')' (0x29) at index 529
Logged from file git.py, line 168
|bower| (git) Updated
Updating 747cbc1..d24da6
Is there a test module that needs to be in place that pip does not import?
$ pip install vcspull
$ vcspull
from test import support
ImportError: No module named 'test'
$ pip install test
$ vcspull
from test import support
ImportError: cannot import name 'support'
For #38's additions
~/study/rust/:
gitui: 'git+https://github.com/extrawurst/gitui.git'
vcspull gitui
noop
$ pullv
$ pullv -t git
$ pullv -t git hg
$ pullv -r django*
$ pullv -r django hi*
To only update repos in ~/workspace
$ pullv -p ~/workspace/
$ pullv
determine automatically if -p
['./', '/', '~', '../']
determine automatically if -t
['git', 'hg', 'svn']
determine automatically if -r
'[a-z]'
After digging into the code, i've found a reference to a shell_command_after
params but it seems never used. Do I miss something?
Python 3.9 benefits vcspull and libvcs could use:
typing.TypedDict
as of 3.8 (no need for backport package)typing.Protocol
typing.Literal
List[str]
-> list[str]
(Type Hinting Generics in Standard Collections)Debian python 3.9
Bullseye is the codename for Debian 11, released on 2021-08-14.
https://wiki.debian.org/DebianBullseye
Python 3, 3.9.1
RedHat RHEL uses 3.6 by default, has up to 3.9
You can install Python 3.8 and Python 3.9, including packages built for either version, in parallel with Python 3.6 on the same system, with the exception of the mod_wsgi module. Due to a limitation of the Apache HTTP Server, only one of the python3-mod_wsgi, python38-mod_wsgi, or python39-mod_wsgi packages can be installed on a system.
source
Hello @tony, it would be very nice if vcspull could update remotes in existing repos. What do you think? For example to add additional remotes to submodules.
โฏ vcspull django
fatal: ref HEAD is not a symbolic ref
Traceback (most recent call last):
File "/Users/me/Library/Python/2.7/bin/vcspull", line 11, in <module>
sys.exit(cli.cli())
File "/Users/me/Library/Python/2.7/lib/python/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/Users/me/Library/Python/2.7/lib/python/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/Users/me/Library/Python/2.7/lib/python/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/me/Library/Python/2.7/lib/python/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/me/Library/Python/2.7/lib/python/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/Users/me/Library/Python/2.7/lib/python/site-packages/vcspull/cli.py", line 107, in update
list(map(update_repo, found_repos))
File "/Users/me/Library/Python/2.7/lib/python/site-packages/vcspull/cli.py", line 125, in update_repo
r.update_repo()
File "/Users/me/Library/Python/2.7/lib/python/site-packages/libvcs/git.py", line 157, in update_repo
symref = self.run(['symbolic-ref', '--short', 'HEAD'])
File "/Users/me/Library/Python/2.7/lib/python/site-packages/libvcs/base.py", line 105, in run
cwd=cwd
File "/Users/me/Library/Python/2.7/lib/python/site-packages/libvcs/util.py", line 178, in run
raise exc.CommandError(output=output, returncode=code, cmd=cmd)
libvcs.exc.CommandError: Command failed with code 128: git symbolic-ref --short HEAD
~/study/django/django tags/2.0
command output:
โฐโ vcspull -c ./vcspull.yaml
Traceback (most recent call last):
File "/home/aschleifer/.local/bin/vcspull", line 8, in <module>
sys.exit(cli.cli())
File "/usr/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/usr/lib/python3.10/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/usr/lib/python3.10/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib/python3.10/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/home/aschleifer/.local/lib/python3.10/site-packages/vcspull/cli.py", line 111, in update
list(map(update_repo, found_repos))
File "/home/aschleifer/.local/lib/python3.10/site-packages/vcspull/cli.py", line 147, in update_repo
current_remote = r.remote(config_remote_name)
AttributeError: 'SubversionRepo' object has no attribute 'remote'
config:
/home/aschleifer/tmp/vcspull:
config: 'svn+https://<svn_host>/svn/<repo_path>/trunk/config'
It does make the actual new svn checkout, but the error shows up.
On a side note I also noticed that for git repos you see the output of the git command when it is executed, but for svn you see nothing, which makes it difficult to see the progress when you pull big/slow repositories.
XDG_USER_CONFIG/vcspull:
repos/*{json,yaml}
settings.{yaml,json,toml}
Moving the config files ~/.vcspull/.{json,yaml} -> ~/.config/vcspull/repos/.{json,yaml}
~/study/c++/:
uMario:
repo: "git+https://github.com/jakowskidev/uMario_Jakowski"
remotes:
tony: "[email protected]:tony/uMario_Jakowski.git"
~/study/c++/:
sdl2-playproject: "git+ssh://[email protected]/tony/sdl2-playproject.git"
aseprite:
repo: 'git+https://github.com/aseprite/aseprite.git'
remotes:
tony: 'git+ssh://[email protected]/tony/aseprite.git'
won't merge repos. fix
It would be great if vcspull supported relative paths for git+ssh remotes by using the colon character as a separator between host and path.
--continue-on-error
/ etc. (see what other apps use for flags to continue on error)
When pulling repos with remotes:
{u'url':'git+ssh://[email protected]/tony/tmuxp.git', u'remote_name': 'tony'}
https://github.com/tony/vcspull/tree/click
Consider allowing starting with/
, ./
, ~
or any string with a /
within to be treated as pulling by file path.
$ vcspull ./.config
$ vcspull ./.config/*
$ vcspull ~/*
If starting with http
, git
, file
, try to find match with vcs url
$ vcspull https://github.com/tony/vcspull.git
(done) Default, match repo name
$ vcspull vim
(done) Pull multiple repos by vcs name delimited by space
vcspull neovim vim
If I use a simple config entry like:
~/test/cvspull:
mypackage: 'git+ssh://[email protected]/mygroup/mypackage.git'
I always see the line about Updating remote origin
:
Updating remote origin (ssh://[email protected]/mygroup/mypackage.git) with git+ssh://[email protected]/mygroup/mypackage.git
which then in turn causes a retry of the update.
To fix this issue I have to use a config like this:
~/test/cvspull:
mypackage:
url: 'git+ssh://[email protected]/mygroup/mypackage.git'
remotes:
origin: 'ssh://[email protected]/mygroup/mypackage.git'
This then works, but I would like to have a more compact/simplified config with only one line per repo.
(repogroup and name have been abstracted, as the used gitlab is internal ;) )
Best regards
Segaja
vcspull 0.0.10
$ vcspull dotnet
Traceback (most recent call last):
File "/usr/local/bin/vcspull", line 9, in <module>
load_entry_point('vcspull==0.0.10', 'console_scripts', 'vcspull')()
File "/usr/local/lib/python2.7/site-packages/vcspull/cli.py", line 119, in main
command_load(args)
File "/usr/local/lib/python2.7/site-packages/vcspull/cli.py", line 144, in command_load
raise exc.NoConfigsFound(
NameError: global name 'exc' is not defined
~/study/llvm:
llvm: 'git+http://llvm.org/git/llvm.git'
~/study/llvm/llvm/tools:
clang: 'git+http://llvm.org/git/clang.git'
lldb: 'git+http://llvm.org/git/lldb.git'
Grabbing clang/lldb before llvm will cause a race condition. Git cannot clone into an empty directory, but tools/{clang,lldb}
are where they are housed by default.
fatal: destination path '/home/tony/study/llvm/llvm' already exists and is not an empty directory.
|llvm| (git) Cloning.
|llvm| (git) git clone --progress http://llvm.org/git/llvm.git /home/tony/study/llvm/llvm
fatal: destination path '/home/tony/study/llvm/llvm' already exists and is not an empty directory.
|llvm| (git) Cloning.
|llvm| (git) git clone --progress http://llvm.org/git/llvm.git /home/tony/study/llvm/llvm
fatal: destination path '/home/tony/study/llvm/llvm' already exists and is not an empty directory.
|llvm| (git) Cloning.
|llvm| (git) git clone --progress http://llvm.org/git/llvm.git /home/tony/study/llvm/llvm
fatal: destination path '/home/tony/study/llvm/llvm' already exists and is not an empty directory.
|llvm| (git) Cloning.
|llvm| (git) git clone --progress http://llvm.org/git/llvm.git /home/tony/study/llvm/llvm
fatal: destination path '/home/tony/study/llvm/llvm' already exists and is not an empty directory.
|llvm| (git) Cloning.
Potential workaround could be:
git init
git remote add origin <path>
git fetch
git checkout -t origin/master
vcspull sqlalchemy clr
Allow scanning for repos with a find/ag.
$ vcspull scan [path] # prompt to merge into config (creating backup first) or create new
Should allow ignore dirs/patterns. Be fast, leverage https://github.com/ggreer/the_silver_searcher if in PATH
.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.