devlooped / dotnet-file Goto Github PK

Currently the tool only targets dotnet 3.1. We should also target 2.1 (LTS) and net5.0 to allow running the tool on environments with just those versions installed.

Discussed in devlooped/oss#1

In addition to allowing a trailing . (dot) in the target path for a file URL, it would be very convenient to also allow it for a directory URL (/tree/ in GH) and cause the entire folder to be downloaded to the target location with a flattened structure.

Since this is not an official Microsoft package, don't use that as the prefix, it can confuse users.

Currently we just persist the Etag for updated files, which is useful to check for updates, but opaque to determine what changed in a file.

If available when downloading a raw file, we should attempt to retrieve the commit/sha for the file being downloaded and store it as an additional metadata point. It could be used (for example) to automate generation of a changelog when updating entries, for example.

Even without further automation, having the commit can inform the user who can diff himself using the previous and current sha.

Describe the Bug

> dotnet file add https://github.com/devlooped/catbag/blob/main/System/Xml/XmlWrappingReader.cs src/common/
src/common/ x <- https://github.com/devlooped/catbag/blob/main/System/Xml/XmlWrappingReader.cs
                 Access to the path is denied.

Expected Behavior

> dotnet file add https://github.com/devlooped/catbag/blob/main/System/Xml/XmlWrappingReader.cs src/common/
src/common/System/Xml/XmlWrappingReader.cs √ <- https://github.com/devlooped/catbag/blob/main/System/Xml/XmlWrappingReader.cs

NOTE: the suggested base dir of src/common is prepended to the default file path given the URI.

The downlevel version of File.Move does not have the overwrite: true argument, so we should check & delete the target file before moving it from the temporary downloaded file.

Currently, whenever you run dotnet file add [uri] (without additionally passing a path), we download directly to the current directory.

This is not very useful when downloading entire folders from GH, or when downloading a single file that should go to a subdirectory to match the source structure.

The idea is to be more explicit as to where you want a file downloaded. If you want it in the current folder, you can use . as the path, like dotnet file add [uri] .. This would match the existing behavior.

When the URI is from github.com, we should apply a default heuristics as follows:

• [org]/[repo]/tree/[branch]/[relative_path]: these are directory URLs. We use the relative_path as the target path.
• [org]/[repo]/[blob|raw]/[branch]/[relative_path]: these are file URLs. We use the relative_path as the target path.

See https://docs.microsoft.com/en-us/dotnet/core/versions/selection#framework-dependent-apps-roll-forward

We could just build the lowest supported (ATM) .NETCore 3.1 and let the roll-forward behavior run the too under .net5/6

This could be file per file, or as an optimization, if a glob URL (i.e. GH repo/dir URL) is present, use that to pre-delete all files without checking them one by one.

Ran:

dotnet tool update -g dotnet-file

Got version 0.2.16:

You can invoke the tool using the following command: dotnet-file
Tool 'dotnet-file' (version '0.2.16') was successfully installed.

Then ran dotnet file and expected to see the help, but instead got:

Unhandled exception. System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')
   at Microsoft.DotNet.Program.Main(String[] args) in D:\a\1\s\src\File\Program.cs:line 25
   at Microsoft.DotNet.Program.<Main>(String[] args)

Showing an error seems brutal. Given no args and default behaviour, it seems reasonable to show help.

If the same upstream URL is used to target more than one file,
we currently just process it once and don't properly update the
other copies in other target folders.

We should instead index by file path and URI when skipping
duplicate processing.

NOTE: this would be a bit suboptimal since we'd still fetch the URL
multiple times, but we'd be correct. And it's a bit of a corner case
for now, not worth additional optimization for that for now, pending
user feedback.

Currently, we offer two choices for customizing the destination location of an added file:

• .: download to the current directory, don't preserve source URI directory structure
• some/directory/: use the given directory as the base directory, but preserve the source URI directory structure.

We're missing the scenario where it might be useful to download the file to a flat directory structure always, such as:

• src/External/.: download to the given directory, without recreating source URI directory structure

For cases like https://github.com/kzu/oss (applying a template) or others
where you want to initialize the current directory from files already listed
in a remote .netconfig, it would be quite convenient to provide this support
built-in, rather than having to first download the file to later do a dotnet update.

The remote .netconfig should be downloaded leveraging our existing support
for GH raw files and authenticated private repositories via the GH CLI.

It seems that dotnet-file only reads from "X-Etag" header, while some servers return the "Etag" header. It would be great if dotnet-file supported both headers.

Describe the Bug

Maybe I've not understood correctly the manual, but I thought that I could update a single file and not all referenced files at the same time using dotnet file update [file] as it is written in the README:

dotnet file update [file]       // updates a specific file, based on its dotnet-config configuration

Steps to Reproduce

.netconfig content:

[file "path/to/file1"]
    url = http://server/file1
[file "path/to/file2"]
    url = http://server/file2

Call:

dotnet file update "path/to/file1"

Expected Behavior

Only file1 is updated.

Actual Behavior

file1 and file2 are updated

Version Info

dotnet-file 1.4.1

A simple char-based progress when we know the length, as we copy to the output file stream

dotnet file add https://github.com/devlooped/oss/blob/main/.editorconfig

ends up adding an entry with

[file "editorconfig"]

(note the missing .)

Version: 1.1.1

When fetching GH repo/folder URLs, it's useful to be able to skip particular entries from subsequent updating (i.e. the repo readme.md).

For example:

[file]
	url = https://github.com/kzu/oss
[file "readme.md"]
	url = https://github.com/kzu/oss/blob/main/readme.md
	skip

To validate downloaded file contents

Previously, running an update was fairly quick, with the optimizations around using HEAD and ETag to conditionally and quickly determine if a full GET was needed.

Since the introduction of the SHA retrieval, however, things are much slower now, since we do a GH CLI call for (almost) all requests.

We should instead optimize and not retrieve the SHA if:

1. We have a matching ETag from request/response
2. We already have a SHA persisted

Given that we now persist the commit associated with a file, we could also detect updated files and build a unified changelog grouped by repo of all commits that changed all the updated files.

Some automation could use this feature to automatically create PRs when changes are found in the upstream repository at some scheduled check interval, say.