vivekratnavel / omniboard Goto Github PK

I think it's interesting to have an omniboard served running in the cloud so I can access it from any computer, just connecting to the ip, without having to install it.

The problem is that there is not any security feature to prevent unauthorized access. I think an easy feature to implement would be a new launch option from the command line, for example:
-p password

With this option omniboard will ask for a password everytime someone tries to access the url.

When you unselect the Status "Failed" from the top left menu you get the following error:

Error: h.has is not a function

Steps to reproduce

• Unselect the "Failed" status:

Environment

• Omniboard 0.02 (both npm and current docker versions)
• Ubuntu 16.04

Notes

• I've disabled browser extensions and tested in latest Chrome / Firefox / Safari with identical results (ie. This isn't the issue)
• The error appears to be thrown on this line

Hey, looks like a cool project!

I tried to start it in a container, linking it to my existing mongo container for which SCRAM-SHA authentification is enabled like so

docker run -it --rm -p 9000:9000 --name omniboard --network sacred_network vivekratnavel/omniboard "--mu mongodb://user:pwd@mongo:27017/?authMechanism=SCRAM-SHA-1 db"

I used --network instead of --link as I could not link the container running not in the default network.

However, after startup I get the following error

Database connection error:  { MongoError: failed to connect to server [localhost:27017] on first connect [MongoError: connect ECONNREFUSED 127.0.0.1:27017]
    at Pool.<anonymous> (/usr/omniboard/node_modules/mongoose/node_modules/mongodb-core/lib/topologies/server.js:336:35)
    at Pool.emit (events.js:182:13)
    at Connection.<anonymous> (/usr/omniboard/node_modules/mongoose/node_modules/mongodb-core/lib/connection/pool.js:280:12)
    at Object.onceWrapper (events.js:273:13)
    at Connection.emit (events.js:182:13)
    at Socket.<anonymous> (/usr/omniboard/node_modules/mongoose/node_modules/mongodb-core/lib/connection/connection.js:189:49)
    at Object.onceWrapper (events.js:273:13)
    at Socket.emit (events.js:182:13)
    at emitErrorNT (internal/streams/destroy.js:82:8)
    at emitErrorAndCloseNT (internal/streams/destroy.js:50:3)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  name: 'MongoError',
  message:
   'failed to connect to server [localhost:27017] on first connect [MongoError: connect ECONNREFUSED 127.0.0.1:27017]' }

So I believe it should try to connect to mongo:27017 instead of 127.0.0.1:27017. Am I missing something or might this be a bug?

Thank you for your support!

Hi,
thanks for you work. Looks pretty cool so far.
When looking at my experiments, I cannot see my configuration at all which I think is quite odd because I would assume that this is a basic feature Omniboard probably supports.

And how do I get additional columns to show up like seen in your screenshot ('Optimizer name', 'BatchNorm' etc). I'm kind of guessing that this is related to my config not showing up.

Similar to the feature many online shops have, it might be useful to have a button to compare multiple selected runs (using a checkbox) . This could be shown in an overlay.

Changed values could be highlighted with a specific color.

I was run in kubernetes. first I use local mongdb, it looks normal. but when I use cosmosdb
I get

I find error api is /api/v1/Runs?distinct=omniboard.tags

Please let me know if you need any additional information.

Hi Vivek,

Thank you so much for your efforts in this amazing tool. I am just wondering if in the future there will be links for downloading artifacts.

Best,
Yao

Currently, neither "steps" nor "timestamps" are set by default and this results in X's being consecutive numbers starting from 0, instead of steps.

Hello and thanks for the great work,

I am currently checking whether I should go for Sacredboard or Omniboard. The thing is, I have a Sacredboard and a Mongo database already running within a Kubernetes cluster. With your current implementation, passing any other mongo uri won't point to the other mongo database (other than localhost:27017).

I edited server/database.js and added process.env.MONGO_URI which could be passed as an environment variable in a docker container instead of passing it via command line.

If this is a change you'd like i'll submit the PR tomorrow. If not, then just ignore this :)

Take it easy

It would be nice if one could use the dashboard for removing some experiments.

The database name could be displayed somewhere. Useful when I have multiple omniboards opened in different tabs.

Is there a possibility for disabling caching in Omniboard?

If I add a new parameter to the configuration I have to manually delete the cache for the parameter column appear in the webpage.

Hi,

I'm having issues with binary artifacts, e.g. PNG plots or pickled data. After adding artifacts with sacred by ex.add_artifact, Omniboard provides an option to download those artifacts. Currently this works fine only with text data, downloading binary data results in corrupted files.

I've tried to manually export binary artifact from MongoDB and it works fine – resulting file can be opened without problems. I've tested both PNG files (plots) and pickled DataFrames.

Steps to reproduce:

1. Prepare an experiment which saves binary artifacts
2. Execute the experiment
3. Download the artifacts from Omniboard
4. Open downloaded artifacts

When installing through npm install -g omniboard, I only get a blank screen for the latest version.
I checked @0.0.3 that still worked. From source also works.

Any changes for a configuring the size of the metric plots, which are currently very small?

Currently, Omniboard always uses port 9000, so I cannot run two instances of it. The desired behaviour: try other ports until found an available one.

Upgrade bootstrap and react-bootstrap to support Bootstrap 4.

Sacred supports following the output of the tf.summary.FileWriter, as the docs show.

I'm using this setup in my experiment, but I don't see any of the TF outputs, is this a bug or not yet supported?

I tried setting up Omniboard as described in the docs as follows

docker run --name sacred-db -p 27017:27017 -v /path/to/database/:/data/db -d
docker run -it --rm -p 9000:9000 --name omniboard --link sacred-db:mongo vivekratnavel/omniboard -m mongo:27017:sacre

and everything is working as expected.

However, reading from the docker documentation, I see that --link is deprecated

Warning: The --link flag is a legacy feature of Docker. It may eventually be removed. Unless you absolutely need to continue using it, we recommend that you use user-defined networks to facilitate communication between two containers instead of using --link

What is the preferred way to run through docker without using the --link option?

I tried

docker network create sacred-network
docker run --name sacred-db -p 27017:27017 -v /path/to/database/:/data/db -d --net=sacred-network mongo
docker run -it --rm -p 9000:9000 --name omniboard vivekratnavel/omniboard --net=sacred-network -m mongo:27017:sacred

But it I only get a failed to connect to server error:

> [email protected] prod /usr/omniboard
> cross-env NODE_ENV=production node dist/index.js "--net=sacred-network" "-m" "mongo:27017:sacred"

Omniboard is listening on port 9000!
2018-11-13T11:41:53.551Z 'MongoNetworkError: failed to connect to server [mongo:27017] on first connect [MongoNetworkError: getaddrinfo ENOTFOUND mongo mongo:27017]'
Retrying in 20 seconds...
Retrying first connect...
(node:29) DeprecationWarning: current URL string parser is deprecated, and will be removed in a future version. To use the new parser, pass option { useNewUrlParser: true } to MongoClient.connect.
2018-11-13T11:42:13.556Z 'MongoNetworkError: failed to connect to server [mongo:27017] on first connect [MongoNetworkError: getaddrinfo ENOTFOUND mongo mongo:27017]'
Retrying in 20 seconds...
Retrying first connect...
2018-11-13T11:42:33.554Z 'MongoNetworkError: failed to connect to server [mongo:27017] on first connect [MongoNetworkError: getaddrinfo ENOTFOUND mongo mongo:27017]'
Failed to establish connection to mongodb://mongo:27017/sacred after 2 retries. Exiting now...
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] prod: `cross-env NODE_ENV=production node dist/index.js "--net=sacred-network" "-m" "mongo:27017:sacred"`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] prod script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2018-11-13T11_42_33_587Z-debug.log

It would be quite convenient to be able to quickly browse the code to fully understand the experiment.
Currently, you have to first download the file before you can open it. It would be more convenient if it was additionally possible to just open the file in a new tab without having it to download it first.

It would be useful to allow the users to add tag and notes while the experiment is running.
At the moment I can add tag and notes but as soon as I refresh the omniboard page the tags and notes disappear.

I love the new "delete experiment" feature. Thx! But I don't like the fact that it reloads the table after each delete. This can take a lot of time, e.g.:

Omniboard is listening on port 9000!
Connection to mongodb://localhost:27017/db established successfully!
::1 - - [08/Jan/2019:09:45:11 +0000] "GET / HTTP/1.1" 200 509 8.116 ms
::1 - - [08/Jan/2019:09:45:11 +0000] "GET /static/css/main.696de9ab.css HTTP/1.1" 200 169175 2.876 ms
::1 - - [08/Jan/2019:09:45:11 +0000] "GET /static/js/main.94b5db79.js HTTP/1.1" 200 4269552 2.789 ms
::1 - - [08/Jan/2019:09:45:12 +0000] "GET /static/media/glyphicons-halflings-regular.448c34a5.woff2 HTTP/1.1" 200 18028 0.952 ms
::1 - - [08/Jan/2019:09:45:12 +0000] "GET /api/v1/Runs?distinct=omniboard.tags HTTP/1.1" 200 2 85.670 ms
::1 - - [08/Jan/2019:09:45:12 +0000] "GET /api/v1/Omniboard.Columns HTTP/1.1" 200 2 194.087 ms
::1 - - [08/Jan/2019:09:45:12 +0000] "GET /api/v1/Omniboard.Config.Columns HTTP/1.1" 200 2 193.392 ms
::1 - - [08/Jan/2019:09:45:29 +0000] "GET /api/v1/Runs?select=_id,heartbeat,experiment,command,artifacts,host,stop_time,config,result,start_time,resources,format,status,omniboard,metrics,meta&sort=-_id&query=%7B%7D&populate=metrics HTTP/1.1" 200 211646157 16874.124 ms
(node:31693) DeprecationWarning: collection.findAndModify is deprecated. Use findOneAndUpdate, findOneAndReplace or findOneAndDelete instead.
::1 - - [08/Jan/2019:09:45:58 +0000] "DELETE /api/v1/Runs/38 HTTP/1.1" 204 - 51.159 ms
::1 - - [08/Jan/2019:09:46:00 +0000] "GET /api/v1/Omniboard.Columns HTTP/1.1" 304 - 28.909 ms
::1 - - [08/Jan/2019:09:46:00 +0000] "GET /api/v1/Omniboard.Config.Columns HTTP/1.1" 304 - 53.584 ms
::1 - - [08/Jan/2019:09:46:00 +0000] "GET /api/v1/Runs?distinct=omniboard.tags HTTP/1.1" 304 - 65.375 ms
::1 - - [08/Jan/2019:09:46:16 +0000] "GET /api/v1/Runs?select=_id,heartbeat,experiment,command,artifacts,host,stop_time,config,result,start_time,resources,format,status,omniboard,metrics,meta&sort=-_id&query=%7B%7D&populate=metrics HTTP/1.1" 200 211664898 16373.722 ms

Btw. do I see correctly that all artifacts are fetched right away? Artifacts can be large...

This does not have a high priority to me, but nevertheless would make the overall experience nicer.

I keep reloading omniboard to see

• new experiments
• new results
• current progress

Therefore, it would be very convenient, if running experiments would update continuously while they are running until they finish.

This is similar to #2 but that it is not just for Captured Out but for the whole table view for running experiments.

Additionally, it would be cool if also new experiments were displayed automatically.

As I said, I don't see this as very urgent, as a refresh isn't the world, but it would still nice to see it at some point.

If I have time, I am also going to look into it myself. But it seems to me that it requires more work.

I guess a simple solution would be to just automatically reload omniboard every few seconds, but this would then be awful while looking through the different runs.
It would be nicer if just the values in the columns of running experiments are being refreshed. This would for example allow for a nice display of progress or the current validation score etc.

When I deselect some status like 'Failed' and completely reload the page, they are selected again.
Could this also be saved?

Actually, I always refresh the page to see newly added experiments. The more convenient solution would obviously be, if omniboard would refresh itself automatically when a new experiment is added or finished.

Thanks for the great work!

It seems the server is bound to 127.0.0.1 but I cannot find where in the code this is specified, I am trying to change this to 0.0.0.0 to accept external connections? I thought on the docker side just setting -p 9000:9000 would route external connections to 9000 but that seems to be not happening?

Anyway when trying to access from machine on same network via omniboards machine ip I always just get a browser timeout.

Any help much appreciated!

The duration is displayed quite imprecisely (if experiment takes hours, the minutes are cut off).

I saw that this has to do with the ms package. There exists an alternative called pretty-ms. Could this be used instead?
https://www.npmjs.com/package/pretty-ms

Best regards

Hi,
I installed the omniboard using npm. When I open http://localhost:9000/ in the browser I get below error: Error: h.has is not a function.

Attaching the screenshot here. Please advise.

The screen could fit much more rows/experiments.

When viewing the details of an experiment the "Metrics Plot" shows an empty plot by default. A nice quality of life improvement would be if Omniboard remembered the selection across experiments so I do not have to click my desired metrics on every run.

It would be really convenient if omniboard could also work with the FileStorage backend of sacred as I have a bunch of older runs which use the FileStorageObserver. I think the file-based storage stores the same information as the MongoDB observer so it should theoretically be possible.

My hyperparameters are stored in a dictionary. Currently omniboard does not support displaying nested values in a seperate column. So far example:

"train" : {"batch_size":32,"epochs":100,"lr":0.01}

It would be very nice to be able to display "batch_size " in it's own seperate column, opposed to the "train" column I have now.

The sacredboard had a config tab (all config options listed). Omniboard has columns, which are nice but are of no use if one has 50 config options. A sacredboard-style config tab would be nice to have.

🚨 You need to enable Continuous Integration on all branches of this repository. 🚨

To enable Greenkeeper, you need to make sure that a commit status is reported on all branches. This is required by Greenkeeper because it uses your CI build statuses to figure out when to notify you about breaking changes.

Since we didn’t receive a CI status on the greenkeeper/initial branch, it’s possible that you don’t have CI set up yet. We recommend using Travis CI, but Greenkeeper will work with every other CI service as well.

If you have already set up a CI for this repository, you might need to check how it’s configured. Make sure it is set to run on all new branches. If you don’t want it to run on absolutely every branch, you can whitelist branches starting with greenkeeper/.

Once you have installed and configured CI on this repository correctly, you’ll need to re-trigger Greenkeeper’s initial pull request. To do this, please delete the greenkeeper/initial branch in this repository, and then remove and re-add this repository to the Greenkeeper App’s white list on Github. You'll find this list on your repo or organization’s settings page, under Installed GitHub Apps.

I think it would be useful to be able add columns using the "+/- Config Columns" from the run_info or even from any other key from the mongodb document(without the prefix run.config).
For example: I can log or calculate some values from the run like the last loss.

Also I think it would be useful to have other maxima from the metric column like "last value"

I really appreciate the work you have done on ominiboard, and really like the notes feature.

I frequently save matplotlib plots to track the progress of my models as they are trained, and I am wondering if it would be possible to add a image viewer for these artifacts.

Thanks!

It's not sorting according to the actual duration.

Cheers.

I think it would useful to see the complete config the same way run_info are shown.

I use the columns to see/sort/filter the most important config values, but then if i want to see the detailed config of an experiment I'll have to show/hide columns for all the table which is very slow when you have a lot of experiments or complex config types (lists) or paths.

When I run this command (from the Quick Start):

omniboard --mu mongodb://user:pwd@host/admin?authMechanism=SCRAM-SHA-1 sacred

Omniboard is connected to the admin database instead of the sacred database.

With different operators (=, >=, <=, !=, etc.)

When the app is started while mongo is not yet ready, it crashes. It would be nice to have it try to reconnect. This is especially handy when using docker-compose, because most of the times my omniboard container is up before mongo has fully started up (even with container dependencies set up).

If I have many experiments (so I need to scroll to get to the most bottom ones), I am unable to inspect the ones at the bottom. When I open them, the plots open but they show up outside the screen and the main scroll on the right is not updated to let me scroll more.

For larger databases I get Error: No response was received from the server. (I suspect a timeout - sacredboard works fine). Can we remove the timeout?

What about live reload support?

Filter the RUNNING status task in new filter functionality. It displays "PROBABLY_DEAD" tasks.

If you add a tag to a running experiment.
it seems that the tag is getting removed after a while (maybe after the next heartbeat, after refreshing the webpage)

Currently, when an experiment run is deleted, only the run detail is deleted from the database. It should also delete all the associated artifacts and metric data

It would be useful to be able to generate plots based on metric and config columns. For example, if I have several runs with different learning rates where I am measuring F1 score, I would like to be able to generate a plot of "F1 vs. Learning Rate" for those runs. So it would involve:

• Ability to select a column to use as the X axis and a column to use as the Y axis
• Ability to filter which runs are present in the plot
• Generate a plot based on this criteria

🚨 You need to enable Continuous Integration on all branches of this repository. 🚨

To enable Greenkeeper, you need to make sure that a commit status is reported on all branches. This is required by Greenkeeper because it uses your CI build statuses to figure out when to notify you about breaking changes.

Since we didn’t receive a CI status on the greenkeeper/initial branch, it’s possible that you don’t have CI set up yet. We recommend using Travis CI, but Greenkeeper will work with every other CI service as well.

If you have already set up a CI for this repository, you might need to check how it’s configured. Make sure it is set to run on all new branches. If you don’t want it to run on absolutely every branch, you can whitelist branches starting with greenkeeper/.

Once you have installed and configured CI on this repository correctly, you’ll need to re-trigger Greenkeeper’s initial pull request. To do this, please delete the greenkeeper/initial branch in this repository, and then remove and re-add this repository to the Greenkeeper App’s white list on Github. You'll find this list on your repo or organization’s settings page, under Installed GitHub Apps.