Code Monkey home page Code Monkey logo

gitproj's Introduction

git-proj

Description

git-proj implements git sub-commands for managing large binary files. Its setup and management is more straightforward than "git LFS" because it doesn't require a git server.

test package

Why git-proj?

Have you ever had a developer commit their whole home directory into your project's git repository (on purpose or accidentally)? I have seen this. That includes their .ssh/ dir with their ssh keys. They better have passwords on their private keys! Someone making this mistake will likely have password-less keys. Yikes, what a security mess--all their keys, and shared keys, will need to be changed on all the systems they access.

Sure, you can "delete" the mistake, but if it were pushed to your main git server, it would be in the repo forever unless you do the work needed to rip it out. Yeah, some tools will do it, but they are not trivial, and your repo should be "locked" for a few hours until it is repaired. Maybe a faster solution: create a new repo from a cleaned-up HEAD on all active branches, and delete the compromised repo. Yeah, faster, but you've lost all your change history.

That is one scenario. This tool won't remove large binary files from a git repo, but it can help with a more common scenario. Beginning developers don't realize they shouldn't be committing object files, exe files, log files, video files, or generated PDFs to the repository. Your snappy repo will start to become huge and slow. There are some good reasons for saving and tracking those kinds of files, but a git repo is not the place. So git-lfs was created. You can skip this tool if you are OK with that solution and don't mind paying for the storage. I wanted something simpler to set up and with a more distributed approach. This tool is a start down that path.

The main inspiration for this git-proj tool came from my desire to use git to version video projects. But video files are HUGE, yet they don't change much. Since most video files are rendered from video editing tools, it is only important to version the video editor files to recreate a particular video file. Since rendering takes time, you will want to save some rendered files. But there is usually no need to save every rendered version.

Currently, this tool only supports git and large files on a local system. The "remotes" can be set up on mounted drives, and they can be used to clone the files to other systems.

See the issues, milestone 2.0 tagged issues for future enhancements that will support remotes across a network.

git proj sub-commands

  • git proj - general help.

  • git proj init - initialize a local git-proj repo.

  • git proj-remote - initialize a new "remote repo" from the local one.

  • git proj push - push files in raw/ to the remote repo (and do a regular git push).

  • git proj pull - pull raw/ files from the remote repo (and do a regular git pull).

  • git proj status - git status and status of binary files in raw/ compared to the remote's raw/.

  • git proj clone - create a git workspace and raw/ workspace from a remote.

  • git proj config - Fix or change the git-proj configuration files.

  • git proj add - TBD. Add files and symlinks to the top raw/ directory.

Click on the image above to see more details.

User Docs

The user docs can be browsed online at: user-doc

I recommend you start with: git-proj

Then the Create_a_git-proj_Repo tutorial shows how to get started with the git-proj commands.

For a complete understanding of all the configuration variables and their "levels," see gitproj Configuration Documentation.

Installing

System requirements:

  • OS: Any Debian-based OS. For example, Ubuntu, Debian, Mint, MX. For other OS, the "portable" installer, which ends with ".tar.gz", might work. (Or see the Quick Start developer section for building a native package for your OS. This is a harder path. I would recommend trying the portable installer first.)

  • Packages: git (>=2.17), bash, coreutils (fmt, tr), gawk (awk), git-flow, libpod-markdown-perl (pod2markdown), markdown, openssh-client, openssh-server, perl (pod2html, pod2man, pod2usage), rsync, sed, tidy. (If you use the apt-get package manager to install, all of these packages will be installed when git-proj is installed.)

  • The git sub-command directory /usr/lib/git-core exists.

User requirements:

  • You have used more than just git clone

  • You have used the git CLI a lot (not just git in an IDE)

Download Locations:

test package

Select the latest version (any OS). The ones with 'RC' or 'test' (and timestamps) in the names are not "stable."

Download example:

tVer=0.7.1-1
tPkg=git-proj-$tVer-mx-x86_64.deb
tUrl=https://moria.whyayh.com/rel/released/software/own/git-proj/deb
tOpt="--user=guest --password=guest"
wget $tOpt $tUrl/$tPkg

Install Example:

sudo apt-get install ./$tPkg

Getting more usage help

The user docs can be browsed online at: user-doc

After installation, html and markdown help files and tutorials can also be found at: /usr/share/doc/git-proj/user-doc/

Also, you can get help with these commands:

man git-proj - this will give you an overview of git-proj

git proj -h - output the overview AND usage help for ALL of the git-proj sub-commands.

git proj [CMD] -h - output the help for "git proj CMD". For example: "git proj init -h"

git proj [CMD] -H usage - this outputs just the short usage help for "git proj CMD." For example: "git proj init -H usage." If a command has required options, you can type the command for the usage help.

Source

https://github.com/TurtleEngr/gitproj

For Developers

The developer docs can be found at: test/dev-doc

gitproj's People

Contributors

sofeel avatar turtleengr avatar

Watchers

 avatar  avatar

gitproj's Issues

Auto clean the test packages

Create a cron job on Moria to remove all except the latest 5 test packages
from /rel/development/software/own/git-proj/deb/

Document the testing structure

Getting the tests set up, so they can be run on a remote server, has made the structure a bit more complex. Mainly the management of the tar files needs to be described, because they are the most fragile part to the tests.

DevOps: Describe typical version system expectations

High-level description of how version systems are used by professional engineers.

Basic

  • Only version source code (exceptions...)
  • Do NOT version generated code (exceptions...)
  • Have places for pulling and pushing generated files (or packages). (dev and prod areas)
  • Version build scripts, and tool configurations
  • Follow a branching model

Branches

  • Use the git-flow branching model. It works for large and small products.
  • main - only stable releases are committed on this branch
  • develop - new code requires test, and all tests pass (process for if tests fail...)
  • feature/branches - code can be checked in at any time, even if tests fail. This branch can be skipped if you are the only developer.
  • release/branches - same requirements as "develop" plus all is cleaned up and packaged. The release/branch part can be skipped if you have a small team and small product. Just merge develop to main when it is stable and packaged.

Move away from travis-ci

Travis-CI's pricing plans are too confusing, and they could lead to surprising costs. I would like to do the "pay by use" plan that they describe in lots of places, but there is no plan shown for that!

From the comments in the community forum, it is clear they only want Enterprise customers, now that open-source products have hardened their product and shown it can handle large loads.

Try

  • GitHub Actions - Pro: managed by GitHub. Con: there may be limits, and a future where they want $
  • Jenkins - Pro: self-hosted, so always free. Con: self-hosted, and security could be an issue

Requirements:

  • Run tests with changes to develop or main branches.
  • Build a test package.
  • Install the package.
  • Verify the install.
  • Push passing packages to a known location
  • Remove old test packages.
  • How can a RC or final package build be "signaled"?

GitHub Actions

usage-limits-billing-and-administration

  • Free for public repos (with limits)
  • < 20 concurrent jobs
  • < 6 hours run time per job
  • < 72 hours for workflows
  • artifacts and log files are deleted after 90 days
  • after 60 days of no activity in a repo, any scheduled workflows will be disabled

10-github-actions-resources-basics-ci-cd

Cleanup: Tests for yes/no etc. can be cleaned up

This parameter syntax is cleaner than using multiple steps.

Make first letter lower: ${ans,}. Return first char: ${ans::1}

Input: ans="Yes" or ans="NO"

declare -l ans
ans=${ans:-n}             # default 'n'
if [ "${ans::1}" = "y" ]   # ::1 only returns the first letter

ans=${ans:-n}
ans=${ans,}                # return lower case, first letter
if [ "${ans::1}" = "y" ]

git init has problems if current dir had files with spaces in their names

Handle files with spaces in the names.

Prompt user to change the file names?

Example problem:

/usr/lib/git-core/gitproj-init.inc: line 191: edit/obs-config/obs-studio/logs/2022-03-24: No such file or directory
/usr/lib/git-core/gitproj-init.inc: line 191: 21-18-45.txt: No such file or directory

ls edit/obs-config/obs-studio/logs
'2022-03-24 21-18-45.txt'  '2022-03-25 10-39-51.txt'  '2022-03-25 11-23-22.txt'

Remove absolute paths from product code and tests

The absolute paths of bin and doc makes the tests to sensitive to where they are run. Remove them from the config files. The gpBin can be set to where a git-proj script is run and gpDoc can be set based on the gpBin location.

The test scripts need to have paths set to be relative to the top-git-dir for the cloned git-proj project. Test files (in the tar fiiles), should have the paths set to be relative to the directory just above the product's top-git-dir, or relative to the test's home or project dir.

Code git-proj-config

Review configs looking for issues or suggested fixes.

Implement processes for a user to copy git-proj vars between
PROJ/.gitproj, PROJ/.git/config, and ~/.gitconfig, using update or
force.

Cleanup: fixup gpVar settings to follow the precedence rules

See git-proj doc, and config.md.

Implement

  • if env. var not set, read var from ~/.gitconfig, PROJ/.git/config,
    giving a "default" if var is not defined.

  • CLI option will override env. var.

  • Validate the settings, if any error, exit

  • Copy the test-env*.tgz files to a "public" place. And implement the
    Makefile targets for managing them.

Refactor pre-commit into separate function files

Refactor pre-commit to include functions from files in ~/.pre-commit.d/ and/or PROJ/.pre-commit.d/ The functions can get their options from "git config" variables. The PROJ/.pre-commit.d/ and files would be versioned, so the PROJ/.git/hooks/pre-commit can always call them.

For example:

.pre-commit.d/
    CheckFileNames
    CheckWhiteSpace
    CheckForTabs
    CheckNotRaw
    CheckBigFiles

git config vars and section names are case-insensitive

Describe the bug

Upper-case in config var names could break the code with unexpected names.

Desktop (please complete the following information)

  • OS and OS Version: MX 19

  • Version: 0.5.4

Additional context

Fix the 'git config' related functions to lowercase the section and key names.

Get pre-commit from more than one place

Have git-proj-init create .git/hooks/pre-commit and PROJ/.pre-commit
from ~/.pre-commit or $gpDoc/hooks/pre-commit (the first one found, wins).

And have git-proj-clone create .git/hooks/pre-commit from
PROJ/.pre-commit, which should have been versioned, or use $gpDoc/hooks/pre-commit (the first one found wins).

Code git-proj-add

Add single files outside of raw/ to raw/, creating symlinks.

Support adding whole directory trees looking for binary files to be
moved to "raw" files). See function in pre-commit hook script, for
identifying those files.

If cvs is used to version raw/ files (a future enhancement), this would
be the logical place for adding raw/ files.

Absolute paths in tar files break Travis-CI testing

The "setup" tar files have absolute paths in many places. Their output needs to be "patched" at run-time, just after being unpacked.

The hard part: the tar files could have been built on different systems, with different locations. So the starting path part will need to be put in each tar file, so that the patch process knows what needs to be changed.

Maintaining relative paths is really hard for an application, so absolute paths are used for things like the "origin" locations.

Maintaining absolute paths is hard for testing processes, because tests can be installed anywhere.

Add option to git init, for it to skip adding files

Ver 0.7.6
git init could make mistakes in adding too many files to the new git repo. So, provide an option to skip adding files to the new repo. That way the user can add the files they want in git.

This is needed because git doesn't always ignore the files are dirs specified in .gitignore

Show the git-proj in action

Is your feature request related to a problem? Please describe.

  • More example scenarios are needed.

  • Why would you want this sub-command?

  • What does it solve?

Cleanup vars that are required, and copied between the different 'levels'

The vars in the config files are copied between the different 'levels' (Product, User, Project, and Local). But it is not always consistent.

Cleanup: init, clone, and config

This also requires more tests to verify what is copied.

Also, the git-proj-config tests have different results between local and Action.

Don't hard code "origin" for remote names

To support a fully distributed git reop along with a distribute raw repo, allow the user to define more than one remote.

Keep the raw remote paired with the git remote.

git init should not include files in an existing raw/ dir, in its search for large binary files

If a raw/ dir already exists, then skip looking at those files.
Ver 0.7.6

Files that are in ignorefiles should also be ignored in the search for large files, because they will not be added to the initial git repo (i.e. they better not be added).

"mkdir raw" should be skipped if it exists.

Existing files in raw/ should not be replaced with any moves. An error should be thrown.

Cleanup: change true/false integer tests to be internally "safer".

Use "0" and "1", for input values, and do string tests:

if [ "${flag:-0}" = "0" ]
if [ "${flag:-0}" != "0" ]
if [ "$?" != "0" ]  # this is optional, since $? will always be an int

This is a common typo:

If "flag" is not set, or it is set to a non-integer, the following if stmt fails with a syntax error. If "flag" is not set it would be better if it was assumed to be "false"

if [ $flag -ne 0 ]; then

or this would be safer style (always define a "good" default value), but it will still have a syntax error if flag is not an integer.

if [ ${flag:-1} -ne 0 ]; then

Notice also the style for testing true/false, only "0" or "not 0" is tested. There should be no tests looking for "1" or any other value. Exception handling using error return codes would be the only exception.

A Better Solution

  • Validate ALL of the vars that could be user-settable.

  • Vars that are boolean values could accept, t, f, true, false, yes, no, 0, 1, and a validator would echo "true" or "false" (and return the corresponding codes), throw an error, or use a default value (with a warning). Note: Bash confuses things with return codes that are reversed from the usual 0/1 expectation: 0=OK, true, !0=problem, false.

  • Vars that require specific values or a range of integers could also include some transformations, throw an error, or use a default value (with a warning).

Cleanup: Remove "internal" function tests

For good TDD, the tests should only test at the topmost interface or function inputs and outputs.

Tests for internal functions should be moved to a different test area. I.e. they are optional; they are only used when the functions are first created. Refactoring could remove or change internal functions. The important part is testing the interface inputs and the expected outputs, not how the internal functions work.

Cleanup config var handling with PROJ/.gitproj

git-proj-clone will initially copy all of PROJ/.gitproj (after git
clone). Then update local config: local-status, local-host,
proj-name. If PROG/.gitproj is missing, recreated it from ~/.gitconfig

After cloning remote, update local vars remote-status, remote-raw-dir

When in a git or git-proj dir, config vars s/b first saved to .git/config

Do this, at the end of the command, update PROJ/.gitprog with the
gitproj values from .git/config. But these vars must stay set to TBD,
in .gitproj: local-status, local-host, proj-name, remote-status,
remote-raw-dir

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.