Code Monkey home page Code Monkey logo

go-billy's Introduction

WE CONTINUE THE DEVELOPMENT AT go-git/go-billy. This repository is abandoned, and no further updates will be done on the code base, nor issue/prs will be answered or attended.

go-billy GoDoc Build Status Build status codecov

The missing interface filesystem abstraction for Go. Billy implements an interface based on the os standard library, allowing to develop applications without dependency on the underlying storage. Makes it virtually free to implement mocks and testing over filesystem operations.

Billy was born as part of src-d/go-git project.

Installation

go get -u gopkg.in/src-d/go-billy.v4/...

Usage

Billy exposes filesystems using the Filesystem interface. Each filesystem implementation gives you a New method, whose arguments depend on the implementation itself, that returns a new Filesystem.

The following example caches in memory all readable files in a directory from any billy's filesystem implementation.

func LoadToMemory(origin billy.Filesystem, path string) (*memory.Memory, error) {
	memory := memory.New()

	files, err := origin.ReadDir("/")
	if err != nil {
		return nil, err
	}

	for _, file := range files {
		if file.IsDir() {
			continue
		}

		src, err := origin.Open(file.Name())
		if err != nil {
			return nil, err
		}

		dst, err := memory.Create(file.Name())
		if err != nil {
			return nil, err
		}

		if _, err = io.Copy(dst, src); err != nil {
			return nil, err
		}

		if err := dst.Close(); err != nil {
			return nil, err
		}

		if err := src.Close(); err != nil {
			return nil, err
		}
	}

	return memory, nil
}

Why billy?

The library billy deals with storage systems and Billy is the name of a well-known, IKEA bookcase. That's it.

License

Apache License Version 2.0, see LICENSE

go-billy's People

Contributors

ajnavarro avatar akavel avatar alcortesm avatar fkorotkov avatar jfontan avatar jinroh avatar jmalloc avatar mcuadros avatar serabe avatar smola avatar taruti avatar toasterson avatar whickman avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

go-billy's Issues

osfs should implement billy.Changer

I have been loving this abstraction that you guys have created so kudos to that! Trying to change the permissions on a file using the osfs but it seems that it doesn't implement the billy.Changer interface. I was looking into making some changes for this but the relationship between the osfs.OS type and the chroot seems to mess it up a little since I guess we don't want the ChrootHelper to assume the given underlying fs is a changer... Any insight into how I might go about this would be greatly appreciated.

memfs OpenFile does not work on directories

After finally getting a wrapper basically working, I run into an issue because memfs's OpenFile won't open directories. That's not how anything works.

I would strongly encourage you to replace all usage of go-billy with afero, as it fully and properly implements an os "compatible" filesystem abstraction. That said, thank you for your time and effort in this library (though more so with go-git).

memfs doesn't handle renaming files which share a prefix (e.g foobar.txt and foobar.txt.asc)

Whenever I try to rename files that share the same prefix, I get an error "Received unexpected error: file does not exist" on the second file.

I cannot reproduce the same behavior on osfs, which works as expected in this case.

I expect to be able to rename multiple files in a row even if they have part of their name in common.

Here is a test to reproduce the error:

import (
	"testing"

	"github.com/stretchr/testify/assert"
	"github.com/stretchr/testify/require"
	"gopkg.in/src-d/go-billy.v4/memfs"
)

func TestBilly_memfsRename(t *testing.T) {
	mfs := memfs.New()
	fd, err := mfs.Create("/foobar.txt")
	require.NoError(t, err)
	require.NoError(t, fd.Close())
	fd, err = mfs.Create("/barbaz.txt")
	require.NoError(t, err)
	require.NoError(t, fd.Close())
	fd, err = mfs.Create("/foobar.txt.asc")
	require.NoError(t, err)
	require.NoError(t, fd.Close())

	assert.NoError(t, mfs.Rename("/foobar.txt", "foobar-tmp.txt"))
	assert.NoError(t, mfs.Rename("/barbaz.txt", "barbaz-tmp.txt"))
	assert.NoError(t, mfs.Rename("/foobar.txt.asc", "foobar-tmp.txt.asc"))
}

memfs doesn't handle creating and opening files in directory which is a symlink

Whenever I create a symlink to a directory, I cannot create/open files using this symlink. It works perfectly, if I use "osfs".

Here is a test for this bug:

import (
	"testing"

	"github.com/stretchr/testify/assert"
	"github.com/stretchr/testify/require"
	"gopkg.in/src-d/go-billy.v4/memfs"
	"gopkg.in/src-d/go-billy.v4/util"
)

func TestSymlink_Mkdir(t *testing.T) {
	fs := memfs.New()
	//fs := osfs.New("/tmp")
	require.NoError(t, fs.MkdirAll("dir", 0755))
	require.NoError(t, util.WriteFile(fs, "dir/file.txt", nil, 0644))
	require.NoError(t, fs.Symlink("dir", "link"))

	_, err := fs.Stat("dir/file.txt")
	assert.NoError(t, err, "reading directly /dir/file.txt")
	_, err = fs.Stat("link/file.txt")
	assert.NoError(t, err, "reading /dir/file.txt through /link")

	fd, err := fs.Create("link/file2.txt")
	assert.NoError(t, err)
	assert.NoError(t, fd.Close())
	_, err = fs.Stat("dir/file2.txt")
	assert.NoError(t, err, "reading file2.txt created into the /link")
}

Let me know if I have made a mistake.

Github or GitHub

For cached github, the name is GitHub; but for uncached github, it is Github. Officially, github is GitHub, but we need to choose one way and stick to it.

memfs: fatal error: concurrent map writes

Hi, I was trying to run memfs create inside go routine, and faced this error:

Error:

goroutine 1035 [running]:
runtime.throw(0x21e1a74, 0x15)
	/usr/local/go/src/runtime/panic.go:1112 +0x72 fp=0xc0006aba50 sp=0xc0006aba20 pc=0x1034a22
runtime.mapassign_faststr(0x1f6cf80, 0xc000c7d680, 0xc00003f4d7, 0xa, 0xc000a5ed88)
	/usr/local/go/src/runtime/map_faststr.go:211 +0x3f7 fp=0xc0006abab8 sp=0xc0006aba50 pc=0x1015377
github.com/go-git/go-billy/v5/memfs.(*storage).createParent(0xc0003866e0, 0xc00003f4a0, 0x41, 0x1b6, 0xc000422030, 0x16366d5, 0xc0003866e0)
	/Users/myuser/go/pkg/mod/github.com/go-git/go-billy/[email protected]/memfs/storage.go:69 +0x189 fp=0xc0006abb30 sp=0xc0006abab8 pc=0x1636289
github.com/go-git/go-billy/v5/memfs.(*storage).New(0xc0003866e0, 0xc00003f4a0, 0x41, 0x1b6, 0x602, 0xc00003f4a0, 0x41, 0x1)
	/Users/myuser/go/pkg/mod/github.com/go-git/go-billy/[email protected]/memfs/storage.go:50 +0x29f fp=0xc0006abbb0 sp=0xc0006abb30 pc=0x163608f
github.com/go-git/go-billy/v5/memfs.(*Memory).OpenFile(0xc000386700, 0xc00003f4a0, 0x41, 0x602, 0x1b6, 0x2, 0xc00003f4a0, 0x41, 0xc0006abcc8)
	/Users/myuser/go/pkg/mod/github.com/go-git/go-billy/[email protected]/memfs/memory.go:49 +0xaf fp=0xc0006abc38 sp=0xc0006abbb0 pc=0x1633fff
github.com/go-git/go-billy/v5/memfs.(*Memory).Create(0xc000386700, 0xc00003f4a0, 0x41, 0xc00003f4a0, 0x41, 0x1, 0x21cdcd6)
	/Users/myuser/go/pkg/mod/github.com/go-git/go-billy/[email protected]/memfs/memory.go:34 +0x50 fp=0xc0006abc90 sp=0xc0006abc38 pc=0x1633e60
github.com/go-git/go-billy/v5/helper/polyfill.(*Polyfill).Create(0xc00012c8a0, 0xc00003f4a0, 0x41, 0xc00003f4a0, 0x41, 0x0, 0x0)
	<autogenerated>:1 +0x50 fp=0xc0006abcd8 sp=0xc0006abc90 pc=0x13b9670
github.com/go-git/go-billy/v5/helper/chroot.(*ChrootHelper).Create(0xc00012c8c0, 0xc00079ab80, 0x40, 0xc00079ab80, 0x40, 0xc000391e80, 0x6)
	/Users/myuser/go/pkg/mod/github.com/go-git/go-billy/[email protected]/helper/chroot/chroot.go:49 +0x80 fp=0xc0006abd28 sp=0xc0006abcd8 pc=0x13ba210
project/internal/services/generator.(*Engine).GeneratePostgreSQLPrimaryReplica.func1(0x0, 0x0)
	/Users/myuser/Works/myprojects/project/internal/services/generator/template_pg_primary_replica.go:175 +0x14a fp=0xc0006abf78 sp=0xc0006abd28 pc=0x163db8a
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc000c7d5f0, 0xc000e42f40)
	/Users/myuser/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57 +0x59 fp=0xc0006abfd0 sp=0xc0006abf78 pc=0x163a6f9
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0006abfd8 sp=0xc0006abfd0 pc=0x1064cd1
created by golang.org/x/sync/errgroup.(*Group).Go
	/Users/myuser/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:54 +0x66

truncated code:

import "github.com/go-git/go-billy/v5/memfs"
bfs := memfs.New()
eg := errgroup.Group{}

for _, val := range vals {
	eg.Go(func() error {
             f, err := bfs.Create(filepath.Join(dirPath, ns))
             if err != nil {
			return err
	     }
	     defer f.Close()
        }
}

// wait eg

Any Idea how to fix this? or it's not supposed to run inside go routine?

Problem with basic interfaces

We need to choose to either make users of the library (us) take the interfaces directly from go-git or make this public, add a Fuck off with your afero requests at the README and move everything here.

A third option is available in some kind of middle ground: copy and paste (and adapt) the interfaces here and just test that the implementations in go-git satisfy them.

To illustrate the problem, let's take go-git's Filesystem interface. This interface contains a few methods, let's take for this example the one with the signature Create(string) (fs.File, error).

Let us define the interface in billy as either of the following ways:

type Filesystem fs.Filesystem
type File fs.File

type Filesystem interface {
  fs.Filesystem
}
type File interface {
  fs.File
}

In this case, the problem is that implementation cannot use billy.Filesystem in their signatures as Create(string) (billy.File, error) since Go seems to have no notion of variance in the type signature :(

storage.ReadAt doesn't read with empty arrays

I think the line:
l := int64(len(b))
should be
l := int64(cap(b))
i.e. the length of data copied should be based on the capacity of the array and not how much it has already been filled by other data.

Improve documentation

This library is used by go-git and as such many developers get to use it.

Currently the documentation is quite poor. See https://godoc.org/github.com/src-d/go-billy, there's zero package level documentation explaining what this library is for.

It'd be also nice to have a comparison with https://github.com/spf13/afero.

It is also missing a CONTRIBUTING.md and references to Code of Conduct etc as requested by https://github.com/src-d/guide/blob/master/engineering/documentation.md

bring back tmpoverlayfs

While the new mount functionality is a good addition for some use cases, the older tmpoverlayfs provided a more consistent solution to provide temporary files complementary to a archive or network storage. Relying on mount for this leads to this kind of workarounds: src-d/go-billy-siva#9

If this needs clarification, I'll post the full explanation and rationale at some point.

memfs `ReadDir()` doesn't return an error when the directory doesn't exist

EDIT: Oops, filed this against src-d/go-billy instead of go-git/go-billy. Re-filed issue here

Version

v5.4.1

Description

memfs.New().ReadDir("asdf") returns nil, nil instead of nil, <directory-not-found-err>.

Minimal example

func TestMemoryReadDir(t *testing.T) {
	if _, err := osfs.New("/tmp").ReadDir("asdf"); err == nil {
		t.Fatal("osfs: expected error; found <nil>")
	} else {
		t.Logf("osfs: error was correctly returned!")
	}

	if _, err := memfs.New().ReadDir("asdf"); err == nil {
		t.Fatal("memfs: expected error; found <nil>")
	} else {
		t.Logf("memfs: error was correctly returned!")
	}
}

// Output
//   osfs: error was correctly returned!
//   memfs: expected error; found <nil>

Proposed changes

func (fs *Memory) ReadDir(path string) ([]os.FileInfo, error) {
	if f, has := fs.s.Get(path); has {
		if target, isLink := fs.resolveLink(path, f); isLink {
			return fs.ReadDir(target)
		}
+	} else {
+		return nil, &fs.PathError{Op: "open", Path: path, Err: unix.ENOENT}
}

	var entries []os.FileInfo
	for _, f := range fs.s.Children(path) {
		fi, _ := f.Stat()
		entries = append(entries, fi)
	}

	sort.Sort(ByName(entries))

	return entries, nil
}

Plan 9 Support

Getting Billy (osfs) to build on Plan 9 is possible by adding the following:

osfs/os_plan9.go

// +build plan9

package osfs

import (
	"os"
)

func (f *file) Lock() error {
	f.m.Lock()
	defer f.m.Unlock()

	s, err := f.Stat()
	if(err != nil) {
		return err
	}

	// Add exclusive bit
	return (f.File).Chmod(s.Mode() | os.ModeExclusive)
}

func (f *file) Unlock() error {
	f.m.Lock()
	defer f.m.Unlock()

	s, err := f.Stat()
	if(err != nil) {
		return err
	}

	// Remove exclusive bit
	return (f.File).Chmod(s.Mode() & ^os.ModeExclusive)
}

I changed osfs/os_posix.go's build header to the following, but I'm not sure it's necessary:

// +build !windows,!plan9

Constructing a `memfs` from an `osfs`.

I have an interesting use case where I have some functions that run over a billy.Filesystem and I want to run them on a directory while ignoring a specific file. Ideally, I'd be able to do this without modifying the functions and without modifying the underlying filesystem. Instead, I want to construct a memfs from an osfs, and then delete the ignored file from the memfs so I can run the function on the memfs without needing to modify this.

Is this functionality difficult to implement? I couldn't find documentation on the best way to do this.

Use spf13/afero or provide a compatibility layer?

I use https://github.com/spf13/afero/ in a number of ways. I wanted to use https://github.com/src-d/go-git/ with that as the backing filesystem. As a result, I'd have to either switch to this, or implement a compatibility layer. Since the former is not possible, I think the later might be a good approach.

I'm opening this issue to track that progress. Basically we should have a helper function which takes a fileystem from afero and returns something that implements this interface. That should probably be in the afero lib, and the opposite should be here.

there is a maximum global temp file limit on util and memfs

There is currently a global temp file limit on util.TempFile:

if tempCount >= MaxTempFiles {

This is then used by memfs as well as third party code to get temp files.

There should not be a limit of total temporary files. Not global and not per-filesystem. It's very easy to exhaust this maximum on any long-running application. If any, there could be a limit of open files at the same time (as with Linux limits).

The issue can be reproduced adding the following test to test/tempfile.go:

func (s *TempFileSuite) TestTempFileMany(c *C) {
	for i := 0; i < 1024; i++ {
		var fs []billy.File

		for j := 0; j < 100; j++ {
			f, err := s.FS.TempFile("test-dir", "test-prefix")
			c.Assert(err, IsNil)
			fs = append(fs, f)
		}

		for _, f := range fs {
			c.Assert(f.Close(), IsNil)
			c.Assert(s.FS.Remove(f.Name()), IsNil)
		}
	}
}

Then test output is as follows:

$ go test ./...
?   	gopkg.in/src-d/go-billy.v3	[no test files]
# gopkg.in/src-d/go-billy.v3/util
util/util_test.go:10: assert declared and not used
ok  	gopkg.in/src-d/go-billy.v3/helper/chroot	0.010s
ok  	gopkg.in/src-d/go-billy.v3/helper/mount	0.005s
ok  	gopkg.in/src-d/go-billy.v3/helper/polyfill	0.006s

----------------------------------------------------------------------
FAIL: <autogenerated>:40: TemporalSuite.TestTempFileMany

/home/smola/dev/go/src/gopkg.in/src-d/go-billy.v3/test/tempfile.go:59:
    c.Assert(err, IsNil)
... value *errors.errorString = &errors.errorString{s:"max. number of tempfiles reached"} ("max. number of tempfiles reached")


----------------------------------------------------------------------
FAIL: <autogenerated>:41: TemporalSuite.TestTempFileWithPath

/home/smola/dev/go/src/gopkg.in/src-d/go-billy.v3/test/tempfile.go:29:
    c.Assert(err, IsNil)
... value *errors.errorString = &errors.errorString{s:"max. number of tempfiles reached"} ("max. number of tempfiles reached")

OOPS: 4 passed, 2 FAILED
--- FAIL: Test (0.04s)
FAIL
FAIL	gopkg.in/src-d/go-billy.v3/helper/temporal	0.046s

----------------------------------------------------------------------
FAIL: <autogenerated>:122: MemorySuite.TestTempFileMany

/home/smola/dev/go/src/gopkg.in/src-d/go-billy.v3/test/tempfile.go:59:
    c.Assert(err, IsNil)
... value *errors.errorString = &errors.errorString{s:"max. number of tempfiles reached"} ("max. number of tempfiles reached")


----------------------------------------------------------------------
FAIL: <autogenerated>:123: MemorySuite.TestTempFileWithPath

/home/smola/dev/go/src/gopkg.in/src-d/go-billy.v3/test/tempfile.go:29:
    c.Assert(err, IsNil)
... value *errors.errorString = &errors.errorString{s:"max. number of tempfiles reached"} ("max. number of tempfiles reached")

OOPS: 90 passed, 2 FAILED
--- FAIL: Test (0.04s)
FAIL
FAIL	gopkg.in/src-d/go-billy.v3/memfs	0.046s
ok  	gopkg.in/src-d/go-billy.v3/osfs	1.508s
?   	gopkg.in/src-d/go-billy.v3/test	[no test files]
FAIL	gopkg.in/src-d/go-billy.v3/util [build failed]

billy.CapabilityCheck

Looks like billy.CapabilityCheck() doesn't properly identify my linux/amd64 system (and maybe others) as not supporting opening files in RDWD mode. This issue originates from a go-git issue.

Building on Illumos Issue

Illumos doesn't have sys call.Flock. Could we please use syscall.FcntlFlock ?

# code.gitea.io/git/vendor/gopkg.in/src-d/go-billy.v4/osfs
../git/vendor/gopkg.in/src-d/go-billy.v4/osfs/os_posix.go:13:9: undefined: syscall.Flock
../git/vendor/gopkg.in/src-d/go-billy.v4/osfs/os_posix.go:13:41: undefined: syscall.LOCK_EX
../git/vendor/gopkg.in/src-d/go-billy.v4/osfs/os_posix.go:20:9: undefined: syscall.Flock
../git/vendor/gopkg.in/src-d/go-billy.v4/osfs/os_posix.go:20:41: undefined: syscall.LOCK_UN

Issue when go get repository

Hello,

when I run

go get -u srcd.works/go-billy.v1/...

I get an error like that :

# cd /home/miramaze/go/src/srcd.works/go-billy.v1; git pull --ff-only
fatal: protocol error: bad line length character: PACK
package srcd.works/go-billy.v1: exit status 1

Add support for creating symlinks.

This would be needed facilitate src-d/go-git#358.

For the sake of simplicity and predictability, I propose adding a Filesystem.Symlink() method with semantics identical to os.Symlink().

To implement src-d/go-git#358 it's necessary to symlink to files that do not exist (because the checkout is still in progress), which is not possible with os.Symlink(). That said, I think it's safer to work around this in go-git, perhaps with temporary files, rather than trying to implement this functionality atomically in Billy.

If this approach seems reasonable to the project maintainers, I'll make an attempt at an implementation.

/cc @smola

Comparison with afero?

I was working with go-git and saw it used this "billy.Filesystem" abstraction. I was surprised it wasn't afero. Is there much of a difference? I imagine I can write a simple wrapper that fixes any slight differences...

`tar` a directory from memfs

Hey!

I'd like to know if any of you've ever had experience taring a memfs tree. Would that be possible using https://golang.org/pkg/archive/tar/ reading the files directly from memfs? Are there any missing abstractions?

As i see here is where it would rely on specifics of a filesystem: https://golang.org/pkg/archive/tar/#FileInfoHeader as it accepts os.FileInfo, thus it must have the properties properly set. Does memfs comply completely with os.FileInfo? Would i have troubles implementing something like this?

Thx!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.