rofinn / filepathsbase.jl Goto Github PK

Some file types might need to pass around extra keyword arguments,
to all operations.

For example an S3Path might need to pass around some creditials in a config kwarg.

it would be nice if PosixPath and WindowsPath operations would accept this extra kwarg,
and just ignore it.

That way I can write code that works for both the same way,
e.g. a symetrical sync operation that doesn't know if it is syncing up or down,
but is just implemented on top of isfile(path; config=myconfig) and copy(src, dest; config=myconfig) etc

This

julia> filename(p"node-v12.13.1-linux-ppc64le.tar.xz")
"node-v12"

does not strike me as a good answer...

I think my gut reaction here would be to only treat the last . as separating the filename from the extension.

Would be cool to be able to detect that. In unix it's very straight forward. In Windows it's more involved. @oxinabox found these (from slack):

Apparently the way to do it is to call into the win32 API,
and use GetFileAttributes (https://docs.microsoft.com/en-gb/windows/win32/api/fileapi/nf-fileapi-getfileattributesa?redirectedfrom=MSDN)then check the flag (i guess via &
masking) for 0x02
(https://docs.microsoft.com/en-gb/windows/win32/fileio/file-attribute-constants)

This is a feature request, not even that, it's more of a discussion about a possible enhancement.

In a similar way to how / joins paths, it would be cool to have some special operator for adding an extension to a path/string. Just a thought.

Base.join has a clear docstring which is to join an array of strings with delimiters. This seems like a different operation than what the join on types in FilePathsBase provides. Base already has a function that has the intended meaning which is joinpath. So, to me, just removing overloading join and just using joinpath would be a good idea.

Conceptually, under this proposal we seperate the details of a protocal specific path as part of a abolute path (e.g. PosixPath, S3Path, FTPPath etc), from the simpiler relative path which can be more protocol agnostic.

AbsolutePaths would hold the protocol, have roots, hold config etc.
A RelativePath would be agnostic to the details of given types, and it would only have relelvent contraints (if any) imposed upon when joinpath was done.

RelativePaths could be used across protocols for example one could copy a file with its folder structure by splitting out a relative path of a local file, and then joining it to the remote location path.

Logically one probably should only actually be able to open an AbsolutePath.
One option is for some or all operations to take a RelativePath and do join(cwd(), relpath).
Doing this always means probably might mean RelativePath <: SystemPath.

One advantages include that we could automatically know that S3Path("bucket") means "s3://bucket/", since there is no such thing as a relative S3Path.

Further we will get to forbid at dispatch time calls to joinopath(::Any, ::AbsolutePath),
and don't have to worry about how to handle nonsense like joinpath(p"s3://foo/bar/", p"s3://foo/zap/")
In the Haskell path library, the authors seperate abolute and relative paths, and it helped a lot in terms of type-safty. Julia is not a language where that kind of type-safty really works out, since not ahead of time compiled. But I do thinkt it might provide lot of clarity to the code.

I think not having this distinction is what made this #63 and JuliaCloud/AWSS3.jl#74 feel odd

This is related to #71
but is a bit less of a wild idea. It is probably mutually exclusive with that idea.

This would differ from Base.download so we should probably try updating julia first? This was triggered from a conversation in oxinabox/DataDeps.jl#100

Joining paths is not division, so I think this method should be removed.

It would be great if this worked:

julia> using AWSS3, Arrow

julia> path = S3Path("...");

julia> test_table = (; a = rand(10), b = 1:10);

julia> open(path; write=true) do io
           Arrow.write(io, test_table)
       end

ERROR: TaskFailedException:
MethodError: no method matching position(::FilePathsBase.FileBuffer)
Closest candidates are:
  position(::HTTP.Form) at /home/ec2-user/.julia/packages/HTTP/TVRTz/src/multipart.jl:22
  position(::Base.SecretBuffer) at secretbuffer.jl:154
  position(::Base.Filesystem.File) at filesystem.jl:238
  ...
Stacktrace:
 [1] write(::FilePathsBase.FileBuffer, ::Arrow.Message, ::Tuple{Array{Arrow.Block,1},Array{Arrow.Block,1}}, ::Base.RefValue{Tables.Schema}, ::Int64) at /home/ec2-user/.julia/packages/Arrow/Re9EM/src/write.jl:235
 [2] macro expansion at /home/ec2-user/.julia/packages/Arrow/Re9EM/src/write.jl:106 [inlined]
 [3] (::Arrow.var"#103#106"{FilePathsBase.FileBuffer,Int64,Arrow.OrderedChannel{Arrow.Message},Base.RefValue{Tables.Schema},Tuple{Array{Arrow.Block,1},Array{Arrow.Block,1}}})() at ./threadingconstructs.jl:169
Stacktrace:
 [1] wait at ./task.jl:267 [inlined]
 [2] write(::FilePathsBase.FileBuffer, ::NamedTuple{(:a, :b),Tuple{Array{Float64,1},UnitRange{Int64}}}, ::Bool, ::Bool, ::Nothing, ::Bool, ::Bool, ::Bool, ::Int64) at /home/ec2-user/.julia/packages/Arrow/Re9EM/src/write.jl:141
 [3] #write#102 at /home/ec2-user/.julia/packages/Arrow/Re9EM/src/write.jl:83 [inlined]
 [4] write(::FilePathsBase.FileBuffer, ::NamedTuple{(:a, :b),Tuple{Array{Float64,1},UnitRange{Int64}}}) at /home/ec2-user/.julia/packages/Arrow/Re9EM/src/write.jl:83
 [5] (::var"#7#8")(::FilePathsBase.FileBuffer) at ./REPL[18]:2
 [6] open(::var"#7#8", ::S3Path{AWSConfig}; kwargs::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol},NamedTuple{(:write,),Tuple{Bool}}}) at ./io.jl:325
 [7] top-level scope at REPL[18]:1

edit: not sure if this is the only missing method to support Arrow.write though!

The join function currently is there, and I see no reason to change anything about it.

But it would be nice to also have an inplace operator that does the same trick. I'll list some candidates as separate comments, so that folks can like/dislike them individually. Any reasons, explanations, new suggestions etc. would also be most welcome!

We might also look around what other languages do about this.

c.f. https://github.com/JuliaData/CSV.jl/pull/511/files#r332346349

Note that some FilePaths can be mmaps directly, in particular SystemPaths
other cannot, like AWS3.S3Paths
But I guess that can be fairly easily handled by treating SystemPaths like strings,
and all other AbstractPaths like Cmds etc.

This would be a nice usecase for JuliaLang/julia#32732
when I eventually make that work,
since then can just compile-time know if something overload's hasmethod(MMap.mmap, ...)

OTOH, I suspect that only SystemPath's are ever going to be mmap-able.
So maybe htis don't need to be extensable,
and existing differentiation between SystemPaths and all other paths is enough.
If so, we can close this

We should probably have proper documentation published to reference in other path subtypes.

Should there be a function like "within" that tests to see if a path is a logical child of another path? Here's a mockup:

function within(child::AbstractPath, parent::AbstractPath)
	cn = normpath(child).segments
	pn = normpath(parent).segments
	return length(pn) <= length(cn) && all(cn[1:length(pn)] .== pn)
end

@testset "path nesting" begin
	@test true == within(p"/a/b", p"/")
	@test true == within(p"/a/b", p"/a")
	@test true == within(p"/a/b", p"/a/b")
	@test false == within(p"/a/b", p"/a/b/c")
	@test false == within(p"/a/b", p"/c")
end

Currently it errors.

julia> parent(p"file.txt")
ERROR: file.txt has no parents
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] parents at /Users/oxinabox/.julia/packages/FilePathsBase/AeOcX/src/path.jl:143 [inlined]
 [3] parent(::PosixPath) at /Users/oxinabox/.julia/packages/FilePathsBase/AeOcX/src/path.jl:120
 [4] top-level scope at none:0

It is possible to define walkdir in terms of AbstractPath operations.

This would be useful

Currently, joining on an empty string errors even though it should probably just ignore that segment.

julia> cwd() / ""
ERROR: MethodError: no method matching PosixPath(::Tuple{String})
Closest candidates are:
  PosixPath(::Tuple{Vararg{String,N} where N}, ::String) at /Users/rory/.julia/packages/FilePathsBase/oi7XZ/src/posix.jl:8
  PosixPath(::Any, ::Any) at /Users/rory/.julia/packages/FilePathsBase/oi7XZ/src/posix.jl:8
  PosixPath() at /Users/rory/.julia/packages/FilePathsBase/oi7XZ/src/posix.jl:12
  ...
Stacktrace:
 [1] PosixPath(::String) at /Users/rory/.julia/packages/FilePathsBase/oi7XZ/src/posix.jl:19
 [2] #Path#3(::Bool, ::Function, ::String) at /Users/rory/.julia/packages/FilePathsBase/oi7XZ/src/path.jl:34
 [3] Path at /Users/rory/.julia/packages/FilePathsBase/oi7XZ/src/path.jl:23 [inlined]
 [4] join(::PosixPath, ::String) at /Users/rory/.julia/packages/FilePathsBase/oi7XZ/src/path.jl:236
 [5] /(::PosixPath, ::String) at /Users/rory/.julia/packages/FilePathsBase/oi7XZ/src/path.jl:215
 [6] top-level scope at none:0

Moved from rofinn/FilePaths.jl#33

Is there a reason that Tuples are used instead of a Vector{String} for the parts field of WindowsPath and POSIXPath?

The string interface is rather complicated and the only reason we've been subtyping it is to make interop with existing filesystem methods easier. A better solution going forward may be to provide an @filepaths macro that can be applied to methods to provide both a string and filepaths dispatch.

Interpolating a path into a cmd string fails (atleast on Windows)

p = p"testfolder/testfile.txt"

run(`echo $p`)

results in the removal of the file separators.

See some further discussion JuliaAnimators/Javis.jl#116

Same for mktmpdir / mktempdir

As far as I can tell it has the same API/methods.

because they have different behavior when the directory already exists. In Base Julia, mkpath does not error if the path already exists, but mkdir does.

Due to

S3Paths have the issue that calling mkpath twice on the same path errors:
e.g.

using AWSS3
path = S3Path(...)
mkpath(path)
mkpath(path) # errors

vs

path = "test"
mkpath(test)
mkpath(test)

This inconsistency adds difficulties for writing generic code that has abstracted over the path, e.g. Onda's Paths API assumes it can call mkpath on the same path more than once.

mkdir is supposed to return a path.
(I think this was the last PR i got into Base before julia 1.0)
This is useful for things like
path=mkdir(joinpath("a", "b"))

julia> mkdir(p"foo")
p"foo"

julia> mkdir(p"bar"; exist_ok=true)
p"bar"

julia> mkdir(p"bar"; exist_ok=true)
false

On 0.7

julia> using FilePathsBase
julia> path = Path(pwd())
julia> endswith(path, "/")
ERROR: MethodError: no method matching isvalid(::PosixPath, ::Int64)
Closest candidates are:
  isvalid(::AbstractString, ::Integer) at strings/basic.jl:121
  isvalid(::String, ::Int64) at strings/string.jl:301
  isvalid(::SubString, ::Integer) at strings/substring.jl:80
  ...
Stacktrace:
 [1] isvalid at ./strings/basic.jl:121 [inlined]
 [2] getindex(::PosixPath, ::Int64) at ./strings/basic.jl:163
 [3] iterate at ./strings/basic.jl:677 [inlined] (repeats 2 times)
 [4] Type at ./iterators.jl:1038 [inlined]
 [5] endswith(::PosixPath, ::String) at ./strings/util.jl:42
 [6] top-level scope at none:0

I want to be able to broadcast my joins:

julia> p"a" ./ ["x", "y"]

MethodError: no method matching length(::PosixPath)
Closest candidates are:
  length(!Matched::Core.SimpleVector) at essentials.jl:595
  length(!Matched::Base.MethodList) at reflection.jl:849
  length(!Matched::Core.MethodTable) at reflection.jl:923
  ...

Stacktrace:
 [1] _similar_for(::UnitRange{Int64}, ::Type{Any}, ::PosixPath, ::Base.HasLength) at ./array.jl:547
 [2] _collect(::UnitRange{Int64}, ::PosixPath, ::Base.HasEltype, ::Base.HasLength) at ./array.jl:580
 [3] collect(::PosixPath) at ./array.jl:574
 [4] broadcastable(::PosixPath) at ./broadcast.jl:664
 [5] broadcasted(::Function, ::PosixPath, ::Array{String,1}) at ./broadcast.jl:1234
 [6] top-level scope at In[110]:1

I can wrap it in a Ref but I shouldn't have to -- I don't have to for strings.

julia> Ref(p"a") ./ ["x", "y"]
2-element Array{PosixPath,1}:
 p"a/x"
 p"a/y"

Currently, cp and sync should error if the destination parent directory doesn't exist. This is to be consistent with the base cp behaviour. We should probably add test confirm that this remains consistent.

The behavour of not overwriting files by default caught me off-guard.

Base.download(::AbstractString, ::AbstractString) always overwrites.

I assumed this would be the same.

Not overwriting by default is a valid choice,
but it would be useful to log something about it.

No reason that it returns a String, right?

We might want to define a

Base.isless(a::AbstractPath, b::AbstractPath) = string(a) < string(b)

To support things like:

julia> using FilePathsBase
julia> list = [Path(pwd()), Path(pwd())]
julia> sort!(list)
ERROR: MethodError: no method matching isless(::PosixPath, ::PosixPath)
Closest candidates are:
  isless(::Missing, ::Any) at missing.jl:66
  isless(::Any, ::Missing) at missing.jl:67
Stacktrace:
 [1] lt(::Base.Order.ForwardOrdering, ::PosixPath, ::PosixPath) at ./ordering.jl:49
 [2] sort! at ./sort.jl:456 [inlined]
 [3] sort!(::Array{PosixPath,1}, ::Int64, ::Int64, ::Base.Sort.MergeSortAlg, ::Base.Order.ForwardOrdering, ::Array{PosixPath,1}) at ./sort.jl:545
 [4] sort! at ./sort.jl:544 [inlined]
 [5] sort! at ./sort.jl:639 [inlined]
 [6] #sort!#7 at ./sort.jl:699 [inlined]
 [7] sort!(::Array{PosixPath,1}) at ./sort.jl:687
 [8] top-level scope at none:0

Currently it is a Union

But Unions are not friendly to extension (see StridedMatrix for the all-time most annoyinng case)

Can be solved by

abstract type SystemPath <: AbstactPath

struct PosixPath <: SystemPath
   ...etc

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Now that this function is in LinearAlgebra, it seems weird...

I'm using CSV.File to read a csv file where some of the columns contain file paths. So I figured I's do this:

types = Dict(:name => typeof(Path()))
CSV.File(file, types = types)

But I'm getting this error:

ERROR: MethodError: no method matching zero(::Type{FilePathsBase.PosixPath})
Closest candidates are:
  zero(::Type{LibGit2.GitHash}) at /build/julia/src/julia-1.5.0/usr/share/julia/stdlib/v1.5/LibGit2/src/oid.jl:220
  zero(::Type{Missing}) at missing.jl:103
  zero(::Type{Dates.Date}) at /build/julia/src/julia-1.5.0/usr/share/julia/stdlib/v1.5/Dates/src/types.jl:405
  ...
Stacktrace:
 [1] xparse at /home/yakir/.julia/packages/Parsers/DAskp/src/Parsers.jl:752 [inlined]
 [2] parsevalue!(::Type{FilePathsBase.PosixPath}, ::UInt8, ::SentinelArrays.SentinelArray{FilePathsBase.PosixPath,1,UndefInitializer,Missing,Array{FilePathsBase.PosixPath,1}}, ::Array{AbstractArray{T,1} where T,1}, ::Array{UInt8,1}, ::Int64, ::Int64, ::Parsers.Options{false,false,true,false,Missing,UInt8,Nothing}, ::Int64, ::Int64, ::Int64, ::Array{Type,1}, ::Array{UInt8,1}) at /home/yakir/.julia/packages/CSV/MKemC/src/file.jl:914
 [3] macro expansion at /home/yakir/.julia/packages/CSV/MKemC/src/file.jl:634 [inlined]
 [4] parsecustom! at /home/yakir/.julia/packages/CSV/MKemC/src/file.jl:624 [inlined]
 [5] parserow at /home/yakir/.julia/packages/CSV/MKemC/src/file.jl:683 [inlined]
 [6] parsefilechunk!(::Val{false}, ::Int64, ::Dict{Type,Type}, ::Array{AbstractArray{T,1} where T,1}, ::Array{UInt8,1}, ::Int64, ::Int64, ::Int64, ::Array{Int64,1}, ::Float64, ::Array{CSV.RefPool,1}, ::Int64, ::Int64, ::Array{Type,1}, ::Array{UInt8,1}, ::Bool, ::Parsers.Options{false,false,true,false,Missing,UInt8,Nothing}, ::Nothing, ::Type{Tuple{Tuple{SentinelArrays.SentinelArray{FilePathsBase.PosixPath,1,UndefInitializer,Missing,Array{FilePathsBase.PosixPath,1}},FilePathsBase.PosixPath},Tuple{SentinelArrays.SentinelArray{FilePathsBase.PosixPath,1,UndefInitializer,Missing,Array{FilePathsBase.PosixPath,1}},FilePathsBase.PosixPath},Tuple{SentinelArrays.SentinelArray{FilePathsBase.PosixPath,1,UndefInitializer,Missing,Array{FilePathsBase.PosixPath,1}},FilePathsBase.PosixPath}}}) at /home/yakir/.julia/packages/CSV/MKemC/src/file.jl:557
 [7] CSV.File(::CSV.Header{false,Parsers.Options{false,false,true,false,Missing,UInt8,Nothing},Array{UInt8,1}}; startingbyteposition::Nothing, endingbyteposition::Nothing, limit::Nothing, threaded::Nothing, typemap::Dict{Type,Type}, tasks::Int64, debug::Bool) at /home/yakir/.julia/packages/CSV/MKemC/src/file.jl:265
 [8] CSV.File(::String; header::Int64, normalizenames::Bool, datarow::Int64, skipto::Nothing, footerskip::Int64, transpose::Bool, comment::Nothing, use_mmap::Nothing, ignoreemptylines::Bool, select::Nothing, drop::Nothing, missingstrings::Array{String,1}, missingstring::String, delim::Nothing, ignorerepeated::Bool, quotechar::Char, openquotechar::Nothing, closequotechar::Nothing, escapechar::Char, dateformat::Nothing, dateformats::Nothing, decimal::UInt8, truestrings::Array{String,1}, falsestrings::Array{String,1}, type::Nothing, types::Dict{Symbol,DataType}, typemap::Dict{Type,Type}, categorical::Nothing, pool::Float64, lazystrings::Bool, strict::Bool, silencewarnings::Bool, debug::Bool, parsingdebug::Bool, kw::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/yakir/.julia/packages/CSV/MKemC/src/file.jl:217
 [9] loadcsv(::String) at /home/yakir/MAT2db.jl/src/MAT2db.jl:31
 [10] process_csv(::String) at /home/yakir/MAT2db.jl/src/MAT2db.jl:43
 [11] top-level scope at ./timing.jl:174 [inlined]
 [12] top-level scope at ./REPL[4]:0
 [13] run_repl(::REPL.AbstractREPL, ::Any) at /build/julia/src/julia-1.5.0/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:288

FilePaths.jl was written to merge concepts from Python's pathlib and Julia's existing filesystem API. It would probably help to review the new PyFilesystem ecosystem to inspire future development. In particular, they appear to support many of the filesystem types we'd want to implement or cleanup (e.g., FTP, S3, Memory, Tar).

It would be useful to define a more general copy that just uses read and write on AbstractPaths to make extensions easier.

Some existing funct ions like relative assume expanduser is defined for a path.
As a rule expanduser only makes sense for some local filepaths,
and for say S3Paths it should do nothing

Other things you can have if path is not a string are specialized getindex and iterate. It’s kind of cute if you can do path[end] to mean basename(path). Things like path[end-3:end] for constructing relative path is useful sometimes.

https://discourse.julialang.org/t/better-handling-of-pathnames/36792/19?u=rory-finnegan

Minimum Required:

• constructors
• ispathtype
• parts
• root
• read
• write
• readdir
• chown/chmod?

Derived Defaults:

• open defaults to reading and writing an IOBuffer (maybe define a FileBuffer that should be flushed/closed to write)?
• copy and move using read + write (permissions?)
• download using copy with different path types
• walk and glob using readdir
• sync using walk and copy (is last modified a consistent assumption?)
• abstract path manipulations (e.g., basename, extension, norm, rel, real, abs) with some assumptions?
• tmp related functions based on root
• mkdir default to a no-op?

Using a Path with DataDeps.jl I hit

ERROR: MethodError: no method matching splitext(::PosixPath)
Closest candidates are:
  splitext(::String) at path.jl:195
  splitext(::AbstractString) at path.jl:481

Is there any reason not to have Base.splitext(p::AbstractPath) = filename(p), extension(p)?

It is kind of annoying that I can't do
readdir.(readdir(somepath)) to exact the 2 level hierachy.

One solution to this would be for readdir to always return a path type.
Wether this is a fully qualified path of the same type as its input,
or wether it is some kind of PathPart{<:AbstractPath} <:AbstractPath type that
both Strings into just the last component,
but readdirs as a fully path using a remembered parent,
or some other solution
I am not sure.

We currently call ispathtype for each registered path type and then call it's constructor with a string. I think the better way to do this would be to simply call tryparse(T, str) until our result isn't nothing. This would likely simplify the code and tryparse is probably something the path types should support anyways.

The following questions should probably be addressed in the docs, with links to relevant blog posts, discussions and issues.

1. Why would you want a path type? Aren't strings good enough?
2. Why can't you just do AbstractPath <: AbstractString to support better interop?
3. How do I write generic code that works with paths and strings?
4. What do I do if package X doesn't support file paths?
5. Why do you still have functions like joinpath?
6. Why do I need to do using FilePathsBase: / to get the join operator?

I think documenting these decisions might help get a new potential contributor up-to-speed sooner (vs rehashing the same old things).

How can we convert the Windows path to the Posix path?

https://blogs.msdn.microsoft.com/commandline/2018/02/28/per-directory-case-sensitivity-and-wsl/

So now it is completely unclear how one should compare two WindowsPaths for equality...

Currently, parse(T, str) basically defaults to T(str) which doesn't seem ideal. Especially with #72, we may want to use parse more extensively and reserve constructors for explicit type construction.

Base on the julia docs this seems like what we want:

• Constructors are explicit, but may manipulate values as necessary.
• convert should be lossless (e.g., convert(String, convert(::PathType, str))) == str)
• parse/tryparse can be lossy, but are explicitly for parsing string representations of paths)

In theory, if you don't expect anyone to explicitly construct your type and folks are just relying on internal default behaviour of join, Path, p"" then a new type could probably just use the default constructor and only implement parse(T, str).