Tried running dagshub repo --help and this is the full help text that gets printed:

Usage: dagshub repo [OPTIONS] COMMAND [ARGS]...

  Operations on repo: currently only 'create'

Options:
  --help  Show this message and exit.

Commands:
  create  create a repo and:

The scroll feature doesn't seem to work. I'm trying to view the setup instructions in the "experiments" section. The card seems to have extra instructions at the bottom but i can't read them.

Steps to reproduce:
On a windows machine

ds = repo.directory("data_new/models")
ds.add(...)

Will result in data_new\models.dvc file at the root of the repo instead of a models.dvc file in the data_new directory.

Hi,
Is it possible to run DagsHub on premis?

Attempting to override the default settings for versioning when uploading a folder returns an error:

TypeError: dagshub.upload.wrapper.DataSet.commit() got multiple values for keyword argument 'versioning'

This is because we are passing kwargs which contains the versioning argument as well as our own versioning argument. We need to check for this first.

Right now you need to specify the columns you want to get. Would be nice to be able to download all of them at once

qr = ds.all()
qr.get_blob_field() # Should download all

The bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.

Currently repo.upload method in dagshub/upload/wrapper.py, just uploads the file and logs the error codes and its content. It would be helpful, if we can get the url of file uploaded.

It's pretty hard to understand how to use Repo, DataSet and related code in dagshub/upload/wrapper.py without more documentation.

Query
An error occurs while attempting to make an HTTPS request to https://dagshub.com/kalema3502/mlflow_test.mlflow/api/2.0/mlflow/runs/log-model, and the server responds with many 500 error responses.

Screenshots

To fully support all cases, it would make more sense to rewrite our patching logic in install_hooks to patch the more low-level os.open function, which maps directly to syscalls and is actually used by both builtins.open and pathlib.

The critical dagshub.init function currently has no docstring at all

Make it so it can upload to any arbitrary branch. Right now it can only create a new one via the new_branch argument

Logging levels should be standardized, and generally more logs added to the most verbose level.
After doing that, a --verbose flag can greatly help in debugging issues

The following twitter logo in footer needs to be updated to X.
I would like to work on this .

In non-colab Jupyter, we save the entire execution history, i.e. if you run a cell twice, the save notebook will include 2 cell copies, each with the specific things run in each version.

There should probably be a way to save the current state, which is what we want.

Currently, Python 3.11 isn't supported by popular ML frameworks such as Pytorch, so it wasn't a high priority to support it ourselves.
@kbolashev Might have more information on what's missing to add this support.

dagshub upload noa/Bears-Recognition <local_file_path> <path_in_remote>

I accept from the target path to behave like in github, start with the github server and be relative to that, rather than the location relatively to my local machine

Need to implement proper paginated fetching there

Similar to https://www.tensorflow.org/api_docs/python/tf/summary/image or https://mlflow.org/docs/latest/python_api/mlflow.tracking.html#mlflow.tracking.MlflowClient.log_image

In this code snippet: https://github.com/DagsHub/client/blob/master/dagshub/upload/wrapper.py#L177

The doc string is not up to date.

Also, remote_path defaults to None, which then fails with a "mysterious" error when run without a remote path provided. We should make it a required parameter or provide better defaults.

Sometimes the user wants to override our logic for whether to push a file to Git or DVC, they can not do this via the CLI currently, as dagshub upload doesn't have a versioning argument, and due to #293 uploading via API doesn't properly work either

Currently, you can use DagsHub to create a repo in 2 ways:

1. CLI
2. Python API

Each one doesn't have all the options, which creates a strange UX when they are necessary.

The CLI flow, doesn't have the option to create a private repo, but does enable cloning the repo locally after creating it. The Python API does have the visibility option, but not the cloning option.

It would be good to have either the CLI or the API have all options. This can be achieved either by adding a clone flag to the Python function, and moving the logic for cloning from the CLI to the API, and/or we can add a --private flag to the CLI command that uses the argument in the API to create a private repo

Save notebook has an extension issue:
In colab, we have the logic to add .ipynb extension backward. We add it when it exists, and don't add it when it doesn't.

My dataset - ~640GB - is failing on dagshub upload.
This is apparently due to some extremely large files (~70-80 MB) being grouped together to a single upload step of ~7GB.

The Dagshub CLI tool allows for the ability to upload datasets.

dagshub upload <repo> data/ data/

However, when we want to remove an item from the dataset, there is no CLI support.
Removing the items locally then running the upload cli tool will persist the items in the dataset.

The workaround is to use the DVC CLI tool to update the folder. However, this requires us to update the DVC Configurations if it has not already been done.

From the user perspective, there might be two ways to handle this:

1. New subcommand: dagshub sync <repo> data/ data
2. New flag for upload: dagshub upload <repo> data/ data/ --sync

Additional Thoughts
If there is a datasource that already points to the dataset folder, the rows will persist even after the items are deleted. To remove these rows, we would need to recreate the data source. Another solution would be to add a new method to the datasources that would remove the rows.

ds = datasources.get('<repo>, 'images')
ds.wait_until_ready()
ds.remove('/path/to/deleted')

When uploading a new directory with the CLI

dagshub upload <repo> <local-dir-path> <remote-dir--path>

The directory is uploaded using DVC.
When uploading a single file using the same command, the file is always uploaded with git.
It would be nice to have a size threshold (i.e 5MB) that would automatically decide to upload the file using DVC.

The interesting question is how do you prevent the repo from growing into a list of many single dvc tracked files, and make sure the user makes use of dvc directories to store big files in a manner that makes sense:

.
├── data  <-- dvc
│   ├── preprocessed
│   │   └──  003.png <-- single file
│   └── raw
├── models <-- dvc
└── src <-- git

Relevant for the various autologgers (keras etc.)

The scenario is that when using a large dataset, there might be many millions of training steps.
The logged metrics become huge.
When using the vanilla logger, the user can easily wrap the logger with an if clause to only log once every e.g. 100 steps.
However, when using an autologger, this isn't something they can easily modify, so it should be an optional configuration on the logger.

Im trying to use the data streaming capabilities from dagshub but when I run the install_hooks function its throws the attribute error shown in the attached file. Running on Windows 11, Dagshub version 0.2.17_1

error.txt

This way you would be able to do

dp = ds.head()[0]
dp["metadata_field"] = "metadata_value"
# Maybe will have to fire this, depending on how much performance tuning we want in this
dp.save_metadata()

I can upload data to DagsHub through python API. Can we use python API to remove files and search files (i.e. glob package) as well?

If I subquery an already queried Datasource, the subquery is not properly composited with the original query.

For instance:

The first query filters the Datasource down to 539 items, but the 2nd level queries ignore this (there are a total of 1008 items in the unfiltered Datasource)

The actually query description look like this:

Workaround:

If you use this notation, it works:

labeled = ds[ds['has_annotation'] == True]

train = labeled[labeled['split'] == 'train']

I was trying to list files in my data directory, simple enough.
I tried running:

fs = DagsHubFilesystem(repo_url=url)
fs.listdir("data")

and got the following error:

ValueError: Could not find a git repo. Either run the function inside of a git repo, specify project_root with the path to a cloned DagsHub repository, or specify repo_url (url of repo on DagsHub) and project_root (path to the folder where to mount the filesystem) arguments

The solution was to add project_root="." to DagsHubFilesystem(), but it was hard to understand and required me to look in the code.
I propose it will be the default when using DagsHubFilesystem with a repo_url.

Hi,
it seems that I cannot login from a notebook in mac M1. In this version I'm using Python 3.9.12.
I have tried both using a token and without it.

This is the code that I'm using, I have tried this same code in a kaggle notebook (python 3.7.12) and it works fine.

dagshub_token = os.getenv("DAGSHUB_TOKEN")
dagshub.auth.add_app_token(token=dagshub_token)
dagshub.init(kaggle_competition, dagshub_username, mlflow=True)

and here's the traceback:

---------------------------------------------------------------------------
UnboundLocalError                         Traceback (most recent call last)
Cell In[13], line 3
      1 dagshub_token = os.getenv("DAGSHUB_TOKEN")
      2 dagshub.auth.add_app_token(token="mysupersecrettoken")
----> 3 dagshub.init(kaggle_competition, dagshub_username, mlflow=True)

File ~/.conda/envs/machine-learning/lib/python3.10/site-packages/dagshub/common/init.py:57, in init(repo_name, repo_owner, url, root, host, mlflow, dvc)
     53 res = http_request("GET", urllib.parse.urljoin(host, config.REPO_INFO_URL.format(
     54     owner=repo_owner,
     55     reponame=repo_name)), auth=bearer)
     56 if res.status_code == 404:
---> 57     create_repo(repo_name)
     59 # Configure MLFlow
     60 if mlflow:

File ~/.conda/envs/machine-learning/lib/python3.10/site-packages/dagshub/upload/wrapper.py:108, in create_repo(repo_name, org_name, description, private, auto_init, gitignores, license, readme, template, host)
    105     if token is not None:
    106         auth = HTTPBearerAuth(token)
--> 108 if auth is None:
    109     raise RuntimeError("You can't create a repository without being authenticated.")
    111 if (license != "" or readme != "" or gitignores != "") and template == "none":

UnboundLocalError: local variable 'auth' referenced before assignment

Any ideas?

Discovered by Jinen

We need to make sure it's usable by users without having them figure out what the list actually has to be, which is a tuple of (path, binary_data).
Right now the workaround is to basically map(lambda p: DataSet.get_file(...), paths). Maybe that will be enough to handle most cases.

Right now there's no "neat" way to get a datapoint if you know its path, even though we can do query filtering by path.

Suggested syntax:

dp = ds.get_datapoint("file.jpg")

should prevent from oversaturating the network when making a mistake and doing something like:

for dp in ds.all():
    with ds.metadata_context() as ctx:
        ctx.upload_metadata(....)

Instead of having the context outside

Right now the DagsHubFilesystem offers a listdir method that returns a list. What if I am trying to access a very large bucket directory, I cannot expect that list to be infinitely big.
Example snippet that will time out:

from dagshub.streaming import DagsHubFilesystem
fs = DagsHubFilesystem(".", repo_url="https://dagshub.com/DagsHub-Datasets/radiant-mlhub-dataset")
fs.listdir("s3://radiant-mlhub/bigearthnet")

I propose that the client implements a fs.Walk that returns a generator with potentially infinite content.

Right now the cache doesn't check that the tokens in it are valid.
This way you can add a permanent token, then revoke it on the website, but it won't be "revoked" in the client

The user currently has to manually list all the files in a directory (probably by using something like os.Walk) and add them one by one to the DataSet. This should be a common use case that we can easily make much easier by rolling it into DataSet.add function.

RepoAPI should have a download() function that utilizes the common.download.download_files() function to download the whole repo or a directory in a repo in a parallelized fashion.

repo = RepoAPI("user/repo")
repo.download(outdir="/home/user/my-repo")
# OR
repo.download("data/images", "/home/user/images")

Probably should be recursive by default with the ability to turn it off

They have a tendency to get interrupted, and an optional retry mechanism would really improve the experience.
This will require care, since if the upload has any opened files, they either have to be re-opened or else the retry aborted.

Of course, it should also automatically use OAuth