Comments (28)
is there a workaround for me to achieve someting similar using DVC+CML (+Studio)?
If you have pushed your model file to the DVC remote, you can download the trained model from any git revision with https://dvc.org/doc/command-reference/get:
$ dvc get ${GIT_REPO} ${MODEL_PATH} --rev ${GIT_REV}
For example:
# 10-bigrams-experiments is a git tag present in the repo
$ dvc get https://github.com/iterative/example-get-started model.pkl --rev 10-bigrams-experiment
When you register a new model version through Studio UI, it actually creates a git tag in the repo, so you can use it as --rev
argument.
For example, in the public demo project https://github.com/iterative/demo-bank-customer-churn which has some model versions registered:
$ dvc get https://github.com/iterative/demo-bank-customer-churn .mlem/model/lightgbm-model --rev [email protected]
In the command above, [email protected]
is the tag created by Studio.
As a quick workaround, Studio should be able to construct and display this dvc get
command with appropriate arguments, given that it has all the info needed.
from studio-support.
Thanks for reaching out @haimat - I'll start discussing if we can support a something like this. I've reached out to some colleagues to see if we can help you out right now, we'll get back to you.
from studio-support.
@shcheklein Yes in the dvc.lock
file that has been checked out into the new directory (as you requested before), there is the "-1280.pt" version correctly:
- path: models/ski-defects-small-labels/yolov8-ski-defects-small-labels-1280.pt
md5: 40ab7fd5fa07b74f81a78bcef65685c6
size: 87709448
That is also the file MD5 that I can see on the DVC remote file system, as mentioned earlier (40ab7fd5fa07b74f81a78bcef65685c6).
The 320 version was part of some (local) experiments with DVC. Quite possible that at some point I also DVC- and & Git-pushed that file, but as said above, the checked-out project version from Git tag [email protected]#1
contains the 1280 version in the dvc.lock
file, as I would expect. But for some reason DVC thinks it needs the 320 version there ...
from studio-support.
@shcheklein No, at the moment - and and also when this model tag has been created on Github - we have "1280" in the params file, not "320". Now I am a bit surprised, I was under the impression that these parameters are matter to change on purpose and that is why we have this params.yaml
file in the first place, right?
But let me explain my use case, maybe I am using DVC not in the way it is supposed to be used. Say we have a dataset "ski-defects" and we need to train an object detection model for it. So our workflow is to first create a smaller sample dataset out of it, called "ski-defects-sample", then use that to locally experiment with various training parameters. Often we also train with smaller image sizes locally first, e.g. 320 pixels, in order to train faster. Then later when we are happy with the local training results, we want to train the "real" model with the full dataset and a larger image size, e.g. 1280, on our server.
Note that the results of the training, i.e. the trained model, should always end up in the /models
folder of our project, named based on some of the training parameters used, in particular the dataset name and image size, e.g yolo-model-<dataset>-<img-size>.pt
. Ideally I want to later find this or that model file basd on various training parameters, e.g. "give me the model from that repo with highest metric xyz." (that would be my main use case for Studio btw.)
So I thought now with DVC+CML I could do exactly that - first I would define all required training parameters (like dataset name and image size) in the params.yaml
file, then run local experiments via dvc repo
until I am happy with the results, then update the parameters for the real model and push them to Git, in order to let CML train the full model on the Github CI runner = our training server. And, using variables/templates in the dvc.yaml
file, I was under the impression that DVC would thus automatically track the resulting model files, with all the necessary training params in their file names.
If not the way I have set up my DVC project, how else am I supposed to achieve what I have described above with DVC and CML?
from studio-support.
@daavoo Thanks, that could be a workaround for now indeed. I tried it right away but got a Python FileNotFoundError
from dvc get
. So I wanted to see whether the requested file is tracked by DVC, however, I am not sure whether I am doing something wrong.
When I am in the DVC project's root and run dvc list .
or dvc list models
then in both cases I simply get a list of all the files and folders in the project root. But nothing from within the /models
folder. So how can I check whether the requested file is tracked by DVC?
Note: In my dvc.yaml
file I have a training step with the following outs
section:
outs:
- training/${dataset}
- models/${dataset}/yolov8-${dataset}-${training.image_size}.pt
I was under the impression, that with this DVC automatically tracks the
models/${dataset}/yolov8-${dataset}-${training.image_size}.pt
file and it would be pushed to the DVC remote from the Github runner via dvc push
after training. Is that correct?
from studio-support.
I was under the impression, that with this DVC automatically tracks the
models/${dataset}/yolov8-${dataset}-${training.image_size}.pt
file and it would be pushed to the DVC remote from the Github runner via dvc push after training. Is that correct?
Yes, it should be pushed.
Can you try dvc list . --recursive --dvc-only
?
from studio-support.
@daavoo Thanks, that helped. Now I get this:
$ dvc list . --recursive --dvc-only | grep models
models/ski-defects-small-labels/yolov8-ski-defects-small-labels-320.pt
This file is from my local DVC experiment, I dvc push
ed afterwards, so the model file seems to be tracked correctly. However, apparently that has not worked from the CI run, because that model file is missing in the DVC listing. Any ideas how I can see/debug why this file has not been pushed after the CI run?
from studio-support.
@haimat could you please share how your CI looks like?
because that model file is missing in the DVC listing
do you run it locally (after a pass on CI)?
Any ideas how I can see/debug why this file has not been pushed after the CI run?
I would check the dvc.lock
file - it should have an md5
for this file. Then I would try to find it in the remote storage, to see if file is present there. Here is how cache / remote storage are organized, so you should be able to find a file if you know its md5
.
from studio-support.
@daavoo @shcheklein So it seems there may be a bug in DVC, or I am missing something. First I checked in the Github commit related to the model I added via Studio:
- path: models/ski-defects-small-labels/yolov8-ski-defects-small-labels-1280.pt
md5: 40ab7fd5fa07b74f81a78bcef65685c6
size: 87709448
This is the file I am looking for, so it has been part of this commit in dvc.lock
. Then I checked directly on our server, which acts as DVC remote:
$ tree 40
40
├── 1b37696cbe1766a18296d520870b28
├── 41331c2ea39b415839c8af4d7e5624
├── 498e0aa229e6280cde397b34d79cc7
├── 7c42ee007c3db6a6547fec9af975c9
├── 82ada49323a148f50ba61be9f047ab
├── ab7fd5fa07b74f81a78bcef65685c6 <-- here it is
├── b100f4f999d7390851b9c13b620651
├── b5687a07b2f783d7c14bf1f0796835
├── c4f58f48c45904762f0ab8169b3f08
├── c8e0ea5a384b60fbb40b25ccaf3940
├── d15429e27f54d32be07b649468ac61
├── ddcbeb2970d186ba1adce6f0a3d1d1
└── e8fee04ba173d176c5fb24e03d3c92
0 directories, 13 files
Howver, from my local machine I still geht this error:
$ dvc get https://github.com/<company>/<repo.git> models/ski-defects-small-labels/yolov8-ski-defects-small-labels-1280.pt --rev [email protected]#1
ERROR: unexpected error - : ('models', 'ski-defects-small-labels', 'yolov8-ski-defects-small-labels-1280.pt')
$ dvc get https://github.com/<company>/<repo.git> models/ski-defects-small-labels/yolov8-ski-defects-small-labels-1280.pt --rev [email protected]#1 --show-url
ERROR: failed to show URL - The path 'models/ski-defects-small-labels/yolov8-ski-defects-small-labels-1280.pt' does not exist in the target repository 'https://github.com/<company>/<repo.git>' neither as a DVC output nor as a Git-tracked file.: [Errno 2] No such file or directory: '/models/ski-defects-small-labels/yolov8-ski-defects-small-labels-1280.pt'
Is this a bug or a feature?
from studio-support.
@haimat it looks like a bug, but let's try to double check a few things:
- Commit hash for the
commit related to the model I added via Studio
and for the[email protected]#1
- is it the same commit? Could you double check that please? - Could you run
dvc get -v ...
to see if there is something suspicious there. - is there something specific to the repository - is it a monorepo with multiple projects, do you have any
.dvcignore
files, etc?
Also, what version of DVC do you use? (share dvc version
please).
I've tried to reproduce it with my demo repo, all of these:
dvc get https://github.com/shcheklein/hackathon model/best_model.data-00000-of-00001 --rev [email protected]#1
dvc get https://github.com/shcheklein/hackathon model --rev [email protected]#1
dvc get https://github.com/iterative/demo-bank-customer-churn.git models/perm_imp.joblib --rev [email protected]#4
worked fine for me.
Could you please try to run them (the last one might require S3 access but if it fails on trying to get to S3 I think it's fine).
from studio-support.
@shcheklein Sure, so first I have only one version for this model registered:
That commit eedfb33
you see there is one from CML, it includes the aforementioned MD5 hash:
And that is also the one with this tag [email protected]#1
:
Then here is the verbose output of dvc get
:
$ dvc get -v https://github.com/<company>/<repo.git> models/ski-defects-small-labels/yolov8-ski-defects-small-labels-1280.pt --rev [email protected]#1
2023-03-01 11:15:35,155 DEBUG: v2.43.2 (pip), CPython 3.10.6 on Linux-5.15.0-60-generic-x86_64-with-glibc2.35
2023-03-01 11:15:35,155 DEBUG: command: /home/mfb/.local/bin/dvc get -v https://github.com/<company>/<repo.git> models/ski-defects-small-labels/yolov8-ski-defects-small-labels-1280.pt --rev [email protected]#1
2023-03-01 11:15:35,266 DEBUG: Creating external repo https://github.com/<company>/<repo.git>@[email protected]#1
2023-03-01 11:15:35,266 DEBUG: erepo: git clone 'https://github.com/<company>/<repo.git>' to a temporary dir
2023-03-01 11:15:36,411 DEBUG: erepo: using shallow clone for branch '[email protected]#1'
2023-03-01 11:15:36,565 DEBUG: Removing '/data/head/dev/.aRF7iimTs2Fg8f6iY4ksnH'
2023-03-01 11:15:36,565 ERROR: unexpected error - : ('models', 'ski-defects-small-labels', 'yolov8-ski-defects-small-labels-1280.pt')
------------------------------------------------------------
Traceback (most recent call last):
File "/home/mfb/.local/lib/python3.10/site-packages/dvc_data/index/index.py", line 219, in info
entry = self[key]
File "/home/mfb/.local/lib/python3.10/site-packages/dvc_data/index/index.py", line 274, in __getitem__
return self._trie[key]
File "/home/mfb/.local/lib/python3.10/site-packages/sqltrie/pygtrie.py", line 41, in __getitem__
return self._trie[key]
File "/home/mfb/.local/lib/python3.10/site-packages/pygtrie.py", line 937, in __getitem__
node, _ = self._get_node(key_or_slice)
File "/home/mfb/.local/lib/python3.10/site-packages/pygtrie.py", line 630, in _get_node
raise KeyError(key)
KeyError: ('models', 'ski-defects-small-labels', 'yolov8-ski-defects-small-labels-1280.pt')
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/mfb/.local/lib/python3.10/site-packages/dvc/cli/__init__.py", line 207, in main
ret = cmd.do_run()
File "/home/mfb/.local/lib/python3.10/site-packages/dvc/cli/command.py", line 40, in do_run
return self.run()
File "/home/mfb/.local/lib/python3.10/site-packages/dvc/commands/get.py", line 33, in run
return self._get_file_from_repo()
File "/home/mfb/.local/lib/python3.10/site-packages/dvc/commands/get.py", line 39, in _get_file_from_repo
Repo.get(
File "/home/mfb/.local/lib/python3.10/site-packages/dvc/repo/get.py", line 73, in get
fs.get(
File "/home/mfb/.local/lib/python3.10/site-packages/dvc_objects/fs/base.py", line 626, in get
return get_file(from_info, to_info)
File "/home/mfb/.local/lib/python3.10/site-packages/dvc_objects/fs/callbacks.py", line 69, in func
return wrapped(path1, path2, **kw)
File "/home/mfb/.local/lib/python3.10/site-packages/dvc_objects/fs/callbacks.py", line 41, in wrapped
res = fn(*args, **kwargs)
File "/home/mfb/.local/lib/python3.10/site-packages/dvc_objects/fs/base.py", line 613, in get_file
self.fs.get_file(rpath, lpath, **kwargs)
File "/home/mfb/.local/lib/python3.10/site-packages/fsspec/spec.py", line 859, in get_file
with self.open(rpath, "rb", **kwargs) as f1:
File "/home/mfb/.local/lib/python3.10/site-packages/fsspec/spec.py", line 1135, in open
f = self._open(
File "/home/mfb/.local/lib/python3.10/site-packages/dvc/fs/dvc.py", line 274, in _open
return dvc_fs.open(dvc_path, mode=mode)
File "/home/mfb/.local/lib/python3.10/site-packages/dvc_objects/fs/base.py", line 193, in open
return self.fs.open(path, mode=mode, **kwargs)
File "/home/mfb/.local/lib/python3.10/site-packages/dvc_data/fs.py", line 72, in open
fs, fspath = self._get_fs_path(path, **kwargs)
File "/home/mfb/.local/lib/python3.10/site-packages/dvc_data/fs.py", line 46, in _get_fs_path
info = self.info(path)
File "/home/mfb/.local/lib/python3.10/site-packages/dvc_data/fs.py", line 111, in info
info = self.index.info(key)
File "/home/mfb/.local/lib/python3.10/site-packages/dvc_data/index/index.py", line 223, in info
raise FileNotFoundError from exc
FileNotFoundError
------------------------------------------------------------
2023-03-01 11:15:36,617 DEBUG: link type reflink is not available ([Errno 95] no more link types left to try out)
2023-03-01 11:15:36,617 DEBUG: Removing '/data/head/.2UE8UkWtnULiS2CWf8HTnq.tmp'
2023-03-01 11:15:36,617 DEBUG: Removing '/data/head/.2UE8UkWtnULiS2CWf8HTnq.tmp'
2023-03-01 11:15:36,617 DEBUG: Removing '/data/head/.2UE8UkWtnULiS2CWf8HTnq.tmp'
2023-03-01 11:15:36,617 DEBUG: Removing '/data/head/dev/.dvc/cache/.J5X4StrSDfEHCjwHMDcXvM.tmp'
2023-03-01 11:15:36,643 DEBUG: Version info for developers:
DVC version: 2.43.2 (pip)
-------------------------
Platform: Python 3.10.6 on Linux-5.15.0-60-generic-x86_64-with-glibc2.35
Subprojects:
dvc_data = 0.37.3
dvc_objects = 0.19.1
dvc_render = 0.1.0
dvc_task = 0.1.11
dvclive = 1.4.0
scmrepo = 0.1.7
Supports:
http (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
https (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
ssh (sshfs = 2023.1.0)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/nvme0n1p2
Caches: local
Remotes: ssh, ssh
Workspace directory: ext4 on /dev/nvme0n1p2
Repo: dvc, git
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2023-03-01 11:15:36,645 DEBUG: Analytics is enabled.
2023-03-01 11:15:36,687 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmpw7hnridq']'
2023-03-01 11:15:36,689 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmpw7hnridq']'
To your third question, no it is not a monorepo, its just this single project there. My root .dvcignoe
file looks like this:
# Add patterns of files dvc should ignore, which could improve
# the performance. Learn more at
# https://dvc.org/doc/user-guide/dvcignore
/data/workspace
/training
No other .dvcignore
files:
$ find . -name ".dvcignore"
./.dvcignore
DVC version info is listed above.
Also the (first two) sample dvc get
commands worked fine for me.
from studio-support.
@haimat thanks a lot! I'll try to reproduce it. One quick question - is there something specific about the DVC config in this repo https://github.com/<company>/<repo.git>
, or any other repo, any changes to the link types, some other config options? Or may be something in the global config on your machine?
from studio-support.
@haimat thanks a lot! I'll try to reproduce it. One quick question - is there something specific about the DVC config in this repo
https://github.com/<company>/<repo.git>
, or any other repo, any changes to the link types, some other config options? Or may be something in the global config on your machine?
No nothing, the only config is our default remote. It's our first DVC project, we don't have anything fancy here - not that I am aware of at least.
from studio-support.
@shcheklein btw. I'm happy to send you our DVC configs via email if that helps.
from studio-support.
@haimat thanks. yes, that would be helpful - my email is ivan at iterative.ai (or shcheklein at gmail).
One line looks suspicious to me DEBUG: link type reflink is not available ([Errno 95] no more link types left to try out)
. Just to double check - did you change in this repo, globally, or a remote repo / branch link types?
from studio-support.
Also, could you try to run:
dvc get https://github.com/shcheklein/hackathon models/ski-defects-small-labels/yolov8-ski-defects-small-labels-1280.pt --rev [email protected]#1
I've tried to reproduce the same layout that you have (still can't reproduce the bug though ⌛ )
from studio-support.
@shcheklein I tried your last dvc get
command:
$ dvc get https://github.com/shcheklein/hackathon models/ski-defects-small-labels/yolov8-ski-defects-small-labels-1280.pt --rev [email protected]#1
ERROR: unexpected error - Forbidden: An error occurred (403) when calling the HeadObject operation: Forbidden
As for the link types, no I don't think that I have ever changed the link type in the repo. I checked the config
file in both my user home dir and in the DVC project, but nothing regarding link type there. Or do I have to look elsewhere?
from studio-support.
One line looks suspicious to me DEBUG: link type reflink is not available ([Errno 95] no more link types left to try out)
I think, in this case, is just a misleading debug message by DVC
from studio-support.
I think, in this case, is just a misleading debug message by DVC
@daavoo Yes, I'm trying to understand why it is happening though. I don't see it on my machine with a file that exists and with file that doesn't exist.
anyways, @haimat thanks.
There is something specific about this repo structure, or remote storage since everything else work fine.
Two things to try:
- Could you do a clean
git clone
+git checkout [email protected]#1
+dvc pull
? To check that we can get this file in a "regular" way? - Can you try to run
dvc get
outside the repo? - Can you try to
dvc get
theevaluation/results.json
for example?
Thanks for staying with me on this.
from studio-support.
@shcheklein Sure, here are the results:
ad 1) This works, however, with one errror:
$ dvc pull
WARNING: Output 'models/ski-defects-small-labels/yolov8-ski-defects-small-labels-320.pt'(stage: 'training') is missing version info. Cache for it will not be collected. Use `dvc repro` to get your pipeline up to date.
WARNING: No file hash info found for '/tmp/dvc/models/ski-defects-small-labels/yolov8-ski-defects-small-labels-320.pt'. It won't be created.
A data/ski-defects/
A data/ski-defects-small-labels/
A data/_yolo_config.yaml
A data/ski-defects-sample/
A training/ski-defects-small-labels/
A evaluation/results.json
6 files added and 1 file failed
ERROR: failed to pull data from the cloud - Checkout failed for following targets:
/tmp/dvc/models/ski-defects-small-labels/yolov8-ski-defects-small-labels-320.pt
Is your cache up to date?
Interestingly it is referring to the "-320.pt" version here, not the "-1280.pt" version of the model. Seems maybe something happened with the models/${dataset}/yolov8-${dataset}-${training.image_size}.pt
definition in the outs
section of my dvc.yaml
file. No /models
folder is being created that way at all.
ad 2) Does not work either.
ad 3) Yes, that does work, both with or without the --rev
switch.
from studio-support.
Interestingly it is referring to the "-320.pt" version here
hmm, interesting. I'll try to do templating. It might be the reason. Good thing that we simplified it to dvc pull
. We are getting to the bottom of it. Can you confirm please that you see -1280.pt
in the dvc.lock
? do you see -320.pt
in any DVC files, or params, etc.?
from studio-support.
Okay, I was able to reproduce this. I think the reason might be that in the params file you have 320. Could you please confirm that?
It's a bit of a question what is the right behavior here (and we'll discuss this), but from the DVC perspective there is a pipeline that is not up to date if params file is different from the dvc.lock
values, so no matter even if can make it work or not, I would recommend making sure that params is committed anyway with the same value as model was produced with. Otherwise even in the VS Code, Studio, dvc exp show
we'll have misleading information about the experiment.
Could you please confirm this?
from studio-support.
@haimat what you are describing is totally fine, and this is exactly how DVC, CML, other tools can be used. It's expected and absolutely fine to change params
. In this case I have a suspicious that when at least you did dvc pull
and got DVC complaining about the 320
, not 1280
it was because there was a discrepancy between params.yaml
and dvc.lock
. I'm just trying to confirm that. At the moment it is expected that you commit all DVC files together after you did dvc repro
- including params.yaml
.
Could do git clone
of the tag again and share with me dvc.yaml
, params.yaml
, dvc.lock
one more time please?
If that is not the case, it means I'll try to reproduce again and find another possible reason.
from studio-support.
@shcheklein The more I think of all this, the more I guess this all might be related to the various tryouts I did when I was playing around with DVC (this is our first project where we use DVC). I am willing to start from scratch though, so maybe that solves all these issues.
What would be your recommended way to get 100% rid of DVC in the Github repo, and start over from scratch with all the DVC yaml files I have now?
from studio-support.
remove .dvc
, remove dvc.lock
and (optionally) .dvc
files. Nothing else is required in this case I believe.
Btw, as @daavoo pointed out to me in our private conversation, it's not very clear if you actually need / want to template the model file name. It can stay the same across multiple commits, tags, branches. params
will be enough to distinguish them in Studio, and other tools.
"give me the model from that repo with highest metric xyz."
This is will work better also if you have the same name (it's the same model after all, just different params). We are incorporating now metrics and plots into the Studio Model Registry page, so it will be possible to find the best model right there, or it's now possible via the project page (table with commits, metrics, etc).
from studio-support.
@shcheklein I have followed your instructions and created a new Github and DVC project. Unfortunately I cannot test that now in regard of this issue here, since I hit another issue now with our SSH remote for DVC: iterative/dvc-ssh#33
from studio-support.
@shcheklein Heureka! After everything discussed in the other issues here and setting up the project from scratch again, I can confirm that I was able to dvc get
that model file from the DVC remote now, using the Github tag created during model creation in Studio. I compared that local model's md5 sum with the one from the Github commit, and they match perfectly.
So this issue is solved from my perspective, thanks for your support 👍
I have a few other issues and thoughts regarding the model registry and its usage in Studio in general. Should I post them here or do you want me to create another issue for that?
from studio-support.
@haimat hey, it's great to hear that!
Let's create an issue, it's fine even it's just a feedback / thoughts. We would appreciate that a lot!
We can close this one! I'm looking into some edge cases, and we'll try to improve the error message handling.
from studio-support.
Related Issues (20)
- Can't see any live updates HOT 17
- Filter commits by commit message HOT 4
- After re-adding a project to Studio I still see old commits HOT 1
- No permission to fetch remote data HOT 12
- New model never shows up in Studio HOT 14
- How is the model registry in Studio supposed to be used? HOT 7
- Show missing data for pushed experiments HOT 15
- Add support for shap interactive plots HOT 3
- Do we need to track DVCLive files for Studio? HOT 3
- How can a dev pushing to the repo provide his Studio token to the Github action for CML? HOT 3
- DVCLive metrics don't show up in Studio's column selection HOT 26
- New model never leaves "queued" state HOT 7
- 5000 column limit HOT 6
- Support dark mode in Studio
- Azure Blob connection in Studio HOT 2
- Chat window is partially covered by project table headers
- DVC tracked images don't appear in Studio HOT 9
- Missing parent commit (bnraches) for DVC experiments HOT 2
- `dvclive` stops sending data to studio during training HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from studio-support.