Hi,
I have " Failed to create temporary user " as #81 .

Thank you

Hey all - we've had need several times now for some simplified Binder + JupyterHub diagrams. I took a shot at riffing off of @yuvipanda's more complex kubernetes diagram. You can find a draft here:

https://docs.google.com/presentation/d/1t5W4Rnez6xBRz4YxCxWYAx8t4KRfUosbCjS4Z1or7rM/edit#slide=id.g25dbc82125_0_53

This has a ways to go most likely (and may be incorrect in some places). It'd replace the diagram here and could be re-used in workshops / demos / etc.

LMKWYT and please do add edits / comments etc! cc @willingc @yuvipanda @minrk

Screenshots for https://hackmd.io/CYVgDAZgRhAcBMBaKsLEQFgJzCVkAzAGyKwDGFApvCBAIzADswQA?edit

Times in the screenshot are Zurich (CET) timezone

Not asking if we should create one, just saying I did:

https://github.com/Carreau/open-with-binder-firefox-extension

Ok, it's 15 lines and it took me 5 days (3lines/day is not too bad !) but mostly learning how to do it.

Happy to transfer it to the JupyterHub org, and I think we should try to figure how to publish it on the Addon Store. AFAICT it uses API that should also be compatible with Chrome.

Typo in the twitter bio: "We're here to give you informations about and help you with ..."

https://avatars3.githubusercontent.com/u/30417857?s=800&v=4 is a higher quality image

I've had a few people ask me for links to Binders that serve a specific use-case. It would be great if we had a place to point people to that contained prototypical, high-quality repository examples. Off the top of my head, I can think of these categories:

• Textbooks
• Course materials
• Bootcamps
• Package examples/tutorials
• Supplementary material to papers
• Blog posts / general interactive media
• Apps / visualizations
• Data exploration sandbox

It would be great if we could use this issue to farm examples that'd fall into each category, and then I can weave them into some kind of page that could eventually turn into a "gallery" of binder repos.

We should mention resource limits and guarantees in the docs, pointing to z2jh docs on the matter.

mybinder.org currently uses 1G guarantee and 4G limit:

jupyterhub:
  singleuser:
    memory:
      guarantee: 1G
      limit: 4G

Hi! I was hoping we could work out how to support IHaskell as a first-class citizen on mybinder.org. The Dockerfile in that repo works, as well as this one, and I think that's a good start. What would the next step be?

Hi,

I'm getting a following error when launching binder:
Failed to create temporary user for gcr.io/binder-prod/r2d-acae7f9-miykael-2dnipype-5ftutorial-ec3723:36c138e27c34ad742e9e82dc422aa8fd54dc26cb

This is the repo I'm trying to build

There are particular considerations someone should take in using Binder for reproducible computation. For example, if using a Dockerfile make sure that you don't source from latest. For requirements.txt etc then you must specify versions of everything, etc. What do folks think about this?

I saw we don't have any checks on this repo. Is this intentional? Should we at least build the docs? @choldgraf @willingc @betatim

That way we can figure out if people are actually reading the docs :-)

@yuvipanda I believe you own the account for GA on the MyBinder account, yes? Could you add me as an owner so I can create a new analytics property to point here? (and for binderhub etc)

We should put together documentation to help users figure out the service, and to point people to other tools in the ecosystem if they're relevant.

I took a first stab at this and it's already merged. See

http://mybinder.readthedocs.io/en/latest

or

docs.mybinder.org (currently points to the binderhub documentation but this should change soon)

what else should we add? I want this page to be more user-facing (e.g. for people who want to create Binder repos, or who want to use other people's Binder repos, not for people who want to create their own Binder deployments).

Picking up @Carreau's idea of binder blog posts here as I think it is more general than mybinder.org.

I like the idea of having a steady stream of posts. To help with getting them started I think having a central place to collect ideas would be good.

Post ideas:

• grant announcement
• usage statistics from the last 30 days (ending just after the eLife post?)
• how to contribute, maybe highlight a first-contrib with some narration of the timeline of the contribution
• things to do with binder:
  • appmode
  • RISE
• binder, your advent of code platform
• binder-ready repo of the week (recurring post?), just a cool repo, or does something clever on the binder front, or just a nice idea
  • pytudes
  • pandas cookbook
  • molecules
• Binder browser extensions.

If you have the rights, feel free to edit this list directly, otherwise post below and I will update it.

My repo needs access to the PATH solver which is available via an ftp server. Here is the Dockerfile that I am using to build the container.

Issue I am running into is that the ftp connection keeps timing out. Here is a snippet of the build log.

Retrying.

--2017-12-04 06:31:15--  ftp://ftp.cs.wisc.edu/math-prog/solvers/path/ampl/lnx/pathampl
  (try: 4) => ‘pathampl’
Connecting to ftp.cs.wisc.edu (ftp.cs.wisc.edu)|128.105.2.31|:21... failed: Connection timed out.
Retrying.

--2017-12-04 06:33:26--  ftp://ftp.cs.wisc.edu/math-prog/solvers/path/ampl/lnx/pathampl
  (try: 5) => ‘pathampl’
Connecting to ftp.cs.wisc.edu (ftp.cs.wisc.edu)|128.105.2.31|:21... failed: Connection timed out.
Retrying.

I have successfully downloaded this file using wget and curl from outside of a Docker container using

wget -q ftp://ftp.cs.wisc.edu/math-prog/solvers/path/ampl/lnx/pathampl

or

curl ftp://ftp.cs.wisc.edu/math-prog/solvers/path/ampl/lnx/pathampl -o pathampl

which makes me think that the issue has to do with Dockerfile. Any help would be greatly appreciated!

Thanks so much for this great service!

I am trying to set up a binder for PyMC3 (pymc-devs/pymc#2758), which almost works like a dream, but we removed matplotlib from the requirements, so the notebooks are not actually executable. Is there some way to specify installing from requirements.txt and requirements-dev.txt? Or specifying a few extra requirements that the notebooks need, since requirements-dev.txt includes a bunch of testing and documentation packages as well.

Tangentially related to #85, I tried to accomplish the same goal with a different packaging ecosystem here which requires a custom CMD.

The command from my previous attempt was just jupyter notebook --ip 0.0.0.0 which matches the default command in repo2docker and that Dockerfile works, but my new one with a custom command result/bin/ihaskell-notebook --ip 0.0.0.0 does not.

I see that dockerspawner has logic to handle this but I'm wondering if that's being overridden somewhere else?

It would also be really useful to see the console output of the Launching server... stage so I can troubleshoot without opening issues each time a Dockerfile builds but does not launch 😄.

I just came across this:

https://elifesciences.org/events/c40798c3/save-the-date-for-the-elife-innovation-sprint

Seems up the alley of the Binder world, and maybe a good way to connect with some other projects/people/etc. @betatim or @minrk maybe you'd be particularly interested since you're a bit closer already? Or @jzf2101 since you seem to be traveling around the world once per month anyway :-)

My guess is that people thinks this is the repo one should go to for binder - based on my observations from our similar meta-package https://github.com/jupyter/jupyter. Should we increase the explanation of this package in the README and link to binderhub?

Should we add a section to the documentation where people can add themselves if they have a Binderhub deployment?

Having some idea on who is using it, where they are, and what they are doing would be cool. It also let's us point at it for measuring impact.

Via a discussion on kubeflow I found https://github.com/kubernetes-incubator/spartakus which collects anonymous data about a deployment. We could add this to binderhub to collect data automatically. Despite being anonymous it makes me feel a bit weird as a user, and I'd opt-out straight away :-/

Hi all, I have a new binder here that is picking up the latest JupyterLab that has been deployed (0.27):

https://github.com/sympy/quantum_notebooks

However, another binder that I had built early isn't picking up the new version of JupyterLab, even though I keep pushing new commits:

https://github.com/altair-viz/altair_notebooks

@minrk thought there might be a bug...

Hi! 4144d03 at jupyterhub/binderhub@4144d03#diff-1a1c3dd142f76a5fad803a0c52839881 seems to be telling me that I'm in the right issue tracker tor report a repository isn't building properly.

I let this repo run for over an hour, but it seems stuck on the "Pushing" stage (the "Waiting" and "Building" Stages seemed fairly slow as well, taking several minutes) https://github.com/IQSS/workshops

Here's a screenshot after a hour and twenty minutes but it looked like this after ten minutes as well:

To give you a little more context, I hadn't heard of BinderHub under @aculich mentioned it at http://irclog.iq.harvard.edu/dataverse/2018-03-02#i_64028 and he and I chatted more about a potential integration of Dataverse and BinderHub on Friday. I took some notes at https://docs.google.com/document/d/12njDDHHfoNdos0By3lKlxYbRWLsAp8Eo-AGpjuBF_rY/edit?usp=sharing

This morning I asked @izahn if he had heard of BinderHub and he hadn't but he's well aware of tmpnb because he uses it in a somewhat similar service that he runs using code he wrote at https://github.com/IQSS/iqss-jupyter-notebook . Ista is also the author of the "workshops" repo above that's failing. He and I did note the notice at the top of https://github.com/jupyter/tmpnb that says, "DEPRECATED - See BinderHub project".

Hi,

I am totally new to this tool.

I have got the Spec is not of the form "user/repo/ref", provided: "/master". message with some repos and also with https://github.com/binder-examples/jupyter-rise/blob/master/index.ipynb

Thanks for clarification.

I was reviewing if anything special is needed for an "environment.yml", and I note that it currently says said, "Important, You must leave the environment.yml’s name field empty for this to work out of the box." Is that outdated and should be removed?
I question it because I have had ones with a name in name: field work. I copied one that works from https://github.com/quantopian/qgrid-notebooks that works and it has "name: qgrid-environment" for name:.

The section on using conda mentions sklearn/numpy as packages that are hard to install via numpy. They aren't that hard to install really. Change that example to use geopandas or similar.

via #36 (comment)

Right now our buildpack logic can sometimes break. Here's what's up:

What we'd like

If a binder/ folder exists in a repo then it will be used in the buildpack process. Currently this is checked by using the binder_path method (link) within each detect call.

Where this goes wrong

If a repository has a binder/ folder with config files in it, but there is a higher-priority config file in the repository root, then that file will take precedence and short-circuit all subsequent config file searching (e.g., having a myproject/binder/requirements.txt file, but also a myproject/Dockerfile).

How we could fix this

Pass a prefix to the "detect" method that'd define a specific folder in which to look for config files. Then just before here we could check whether we need to be searching in a binder/ folder, and if so then pass that to the detect method as a prefix for all subsequent buildpacks.

cc @yuvipanda - if you think that's a reasonable path forward then I can make a PR

related to #82

There are some times when people have useful information about a binder deployment / jupyterhub setup / hacky solution etc. Oftentimes this doesn't fit within our "official" documentation, but it'd be great if there were a place that people could put common patterns/hacks/lessons-learned/etc that would be more community-driven and have a lower bar to contribution than a PR.

I was thinking we could use GitHub wikis for this, though I still haven't found a great process for people to naturally discover and/or contribute to them. Has anybody on the @jupyterhub/binder-team or @jupyterhub/jupyterhubteam found use in these? Maybe it's worth trying out for a project to see if it catches on.

cc @yuvipanda who I think is interested in this in general as well

Hi,
I would like to post an idea (already mentioned in the chat today):

Enabling the notebook to "autorun" in a browser (i.e. not having a user not having to go through "click on cell run or CTRL + SHIFT" etc)?

It would be a great feature because any user (not only Jupyter savvy persons) could dwelve right into the functionality of the notebook! Imagine clicking on a link and getting to a page with only input fields , interactive sliders etc. ready to go straight out of the box!

In any case I just started using jupyterhub and the binder and find them absolutely amazing. Fantastic achievement from you all and many thanks.

Cheers,
Mats

A few users of the xtensor binder have reported that they were confused by our "Try it Now" button, which "was redirecting to some binder page". They eventually closed their browser tab.

So it was not obvious to them that something was going to be launched eventually.

Maybe more obvious visual indication that something is loading would help.

(as reported on the gitter chat)

When opening issues, there are two references to use https://github.com/jupyterhub/binder/new . It looks like that should probably be "https://github.com/jupyterhub/binder/issues/new" (or just "https://github.com/jupyterhub/binder/issues", to encourage me to search for my problem?)

OpenRefine is an insanely useful Web tool for cleaning up spreadsheets in nice reproducible ways. I hear that it is one of the most popular Data carpentry lessons once delivered :). Since it's a "simple" web app, should be possible to binderize it to make install easier.

I'm putting together a series of binder executed repos that are composed of particular components with their own dependencies.

At one level, this might be a quite simple dependency, such as a Python package installed by pip that requires a particular Linux package to be installed via apt.txt first.

Or it might be more elaborate - for example a pseudo collection of packages to "implement" a particular sort of functionality, such as a range of tools too handle map display.

To try to ease the pain of remembering what Linux package is a pre-requisite of which python package, it might be useful if we could bundle the requirements as pseudo/self-named collections.

For example, I have three "topics" in the following build (the build may not be sensible!): some map handling stuff (maps), some astronomical stuff (astro-stuff) and some notebook presentation stuff (presentation).

maps:
  - apt
    - libproj-dev
    - libgeos-dev
  - conda
    - channels
      - conda-forge
    - dependencies
      - python
      - matplotlib
      - basemap
    - pip
      - folium

astro-stuff:
  - conda
    - channels  
      - astropy
    - dependencies
      - python
      - astropy

presentation:
  - pip:
    - jupyter_contrib_nbextensions
    - RISE
    - jupyter-wysiwyg
  - postBuild
    - jupyter contrib nbextension install --user
    - jupyter nbextension enable widgetsnbextension  --py --user
    - jupyter nbextension enable python-markdown/main  --py --user
    - jupyter-nbextension install rise --py --user
    - jupyter-nbextension enable rise  --py --user 
    - jupyter-nbextension install jupyter_wysiwyg --py --user
    - jupyter nbextension enable jupyter_wysiwyg  --py --user 

What I imagine happening is that binder.yaml parser could take such a file and generate separate apt.txt, environment.yml or requirements.txt, and postBuild files representing the set intersection of the various build components. (I guess there could be version conflicts in there.)

For the user, it would mean they have just a single config file to worry about, plus they can reuse components that represent complex builds (the Linux package you always forget is a dependency, the post-build enabling of an extension, etc).

Looking at the above file, I could also imagine further levels of nesting, eg in terms of jupyter-nbextension install or jupyter-nbextension enable, though adding too many hard-wired paths to particular sorts of operation may reduce the overall utility of a binder.yaml parser?

A place to discuss the Binder 2.0 blog post, currently planned for publishing on the eLife blog. eLife has slots open for a blog post on November 9th, as well as a few dates at the end of November.

Links

• Binder 2.0 announcement (eLife blog)
• Binder 2.0 tech post (Jupyter blog)

Binder blog post hackmd in case we want to iterate there. As the Jupyter blog post was in docs, I figure this will be an easier platform for people to leave comments, edits, etc. We can transfer to hackmd if that's what people prefer.

Other notes

thoughts from naomi on the eLife topic:

From my perspective, it would be great to tell the story of how the team picked up Binder, why it needed further work, what you have done, and how readers can act on this info - e.g. try it out, contribute. The aim is to spread the word about Binder to life scientists, so this blogpost could act as a news piece, with links to your own blogs for further detail. However, technical details are also very welcome.

The dockerfile section have literally guide to creating Binder-ready repositories <LINK>.

Over a long period of time, there are going to be base upgrades that will break Binders. The easiest to see is upgrading our base image in such a way that breaks apt.txt. We are probably going to need to allow users to specify the base distro, as Travis has done with image: trusty, etc., and make transitions in our base image extremely infrequent.

Even if we eventually drop support for an old base image, allowing users to specify it will at least have a reproducibility pointer, indicating what they were building on.

See this commit https://github.com/alexgoodell/uganda-model/tree/b5fb35bad63d9edff6c6a976dede83150506deaa

and this binder attempt https://mybinder.org/v2/gh/alexgoodell/uganda-model/b5fb35bad63d9edff6c6a976dede83150506deaa

We should have a nomenclature discussion about the binder ecosystem.

We have a start for this here:

https://hackmd.io/CwQwjATAZgbMCcBaAzAU3iRwwGMCsiAHFMjogOwgBGeyVcCMAJkA

Please add comments / edits / etc!

We'd like to use BinderHub kernels to allow for widgets to display in the Data 8 and Prob 140 textbooks. Although this is still in the works, I wanted to give you a heads up so we can work out potential issues sooner rather than later.

Rough timeline (exact dates TBD):

• Start of Dec: Discussion with Data 8 + Prob 140 staff about using widgets in textbook
• Middle of Dec: Preliminary rollout of "Interact" button in Data 8 + Prob 140 textbook that starts BinderHub kernel. We expect ~10 pages from each textbook to use kernels.
• Middle of Jan: Classes start, so traffic will be higher.

Back-of-the-envelope calculations on usage:

Data 8's textbook gets around 8k views on a peak day. If we assume that the views are evenly distributed through an 8-hour workday, that's 1k views per hour, or 17 views per minute.

Suppose each user's kernel lasts 10 minutes on average, and that each view creates a kernel. This means that there will be an extra 170 kernels running on average during peak hours.

(Prob 140's textbook gets significantly less traffic, so I believe Data 8's usage will dominate.)

How I'm going to help manage load:

1. Only start a kernel when a user specifically clicks an interact button. This means that not every page view will create a kernel. In addition, only some pages will have the interact button. If 10% of views create a new kernel instead of 100%, this gives an average of 17 extra kernels during peak hours instead of 170.
2. Reuse the same kernel for the same user moving between pages instead of starting a new one. This should lower the kernel per view ratio since there are 1k unique visitors on a peak day (versus 8k page views).

One idea for managing load on the Binder team side: Set a quota for Data 8 textbook servers and deny requests when the quota is filled.

Please let me know if you foresee any issues with this! Happy to work around any constraints that you have.

cc @yuvipanda @CalebS97 @ryanlovett

I think there would be value in creating a subset of the docs that covered "intro to Binder". Maybe something like:

1. Conceptual overview of Binder + the tech that it uses
2. The concept of config files for Binder
3. A super-simple guide to creating a binder-ready repo (e.g. one notebook + one reqs.txt file)
4. An explanation of Binder links and how to generate them
5. A few common use-cases for other config files to show off functionality.
6. Pointers to some particularly cool repositories for next steps.
7. What happens under-the-hood when somebody clicks a Binder link

Both @ctb @betatim had thoughts on this in the gitter, I believe

I am trying to build the following repo:

https://mybinder.org/v2/gh/tpoisot/ScientificComputingForTheRestOfUs/master

It was working up until last week-ish.

When I try now, the build fails, and the only message I get is:

Error during build: Build failing on repo with Julia 0.6, NoneType object has no attribute get

The notebooks run on multiple machines locally.

You should mention the use of an apt.txt file in the documentation! It came handy for me to install gfortran alongside a standard Python environment defined with a requirements.txt file.

I would also add a note or mention the (advanced) possibility of using a Dockerfile in preparing.rst.

It'd be great if we still had instructions for how people could deploy R with Binder using a Rocker docker image. Would this be a complex path for people? I don't think it needs any tech development, but perhaps creating a new repo in binder-examples that showed the "old" way of using an R kernel with Binder.

closes binder-examples/r#1

Recently @tjd2002 posted some comments about their "first time user" experience clicking a Binder link in the eLife article. I'll post their thoughts in their entirety below:

• Hi all, I encountered my first Binder link in the wild this week, and wanted to capture and share that 'new user experience'. Before I go submitting issues, I thought I'd share thoughts here
• I clicked the big blue 'Launch Binder' button from the elife blog post here (https://elifesciences.org/labs/8653a61d/introducing-binder-2-0-share-your-interactive-research-environment)
• Turns out the upstream data source used in that example had changed format, so the import was broken. Being a good citizen (and wanting a learning experience) I figured I'd try to fix it and submit a pull request
• Since I did not manually create the binder (but instead just clicked a button), I struggled at first to figure out where to find the source.
• I did see the interstitial mybinder.org page, but then I was automatically forwarded to a bare notebook page: https://hub.mybinder.org/user/wildtreetech-explore-open-data-7u6uw3bw/notebooks/bikes-per-week.ipynb
• Where I'm going with all this is... I think that the unbranded Notebook page leaves a naive user (like myself) a little bit adrift
• I ended up finding the source repo by using the file browser to find a README.md
• But then it took me a while longer to figure out that the Big Blue Button (tm) was pointing to a particular tag
  Ideally, there would have been a prominent link back to the Repo (and Ref) that was used to build the binder

Some challenges to overcome

This is me spit-balling a few of the challenges here:

1. The eLife post had a binder link that pointed to a data source that changed, thus rendering the binder in an error state when run by users
   • We should be careful to make our binders air-tight when we make big public posts that'll get a lot of traffic.
   • We should fix up @betatim 's tag so that it points to an updated repo that will now work, since that eLife post will continue to see traffic
2. It was difficult to determine which repository the binder pointed to once you were in the live session.
   • We should give a visual cue so users can figure out which repository they're running. We could do this with a UI change we inject into the page (e.g. I think @minrk 's appendix PR in repo2docker could help with this).

cc @willingc

Hey all - I was thinking of writing a kind of "yearly recap" blog post that would touch on some of the big changes in the Binder world (of which there were many!). I'll try to keep a list here of some major things to mention. @Carreau think this would be appropriate for the jupyter blog?

Here's a hackmd with the latest draft:

https://hackmd.io/JwYwDApgjArBwFoBmEBsAOBAWJIsPQGYQoEoAmJAEwCMcBDCQnIA?both

I'm trying to get IHaskell running on mybinder.org, and I've made what I think is a simple Dockerfile that builds correctly but fails to launch. I've reproduced it below:

FROM fpco/stack-build:lts-11.2

# Install all necessary Ubuntu packages
RUN apt-get update && apt-get install -y python3-pip libgmp-dev libmagic-dev libtinfo-dev libzmq3-dev libcairo2-dev libpango1.0-dev libblas-dev liblapack-dev gcc g++ && \
    rm -rf /var/lib/apt/lists/*

# Install Jupyter notebook
RUN pip3 install -U jupyter

ENV LANG en_US.UTF-8
ENV NB_USER jovyan
ENV NB_UID 1000
ENV HOME /home/${NB_USER}

RUN adduser --disabled-password \
    --gecos "Default user" \
    --uid ${NB_UID} \
    ${NB_USER}

COPY . ${HOME}
USER root
RUN chown -R ${NB_UID} ${HOME}
USER ${NB_USER}

ENV PATH $(stack path --local-install-root)/bin:$(stack path --snapshot-install-root)/bin:$(stack path --compiler-bin):${HOME}/.local/bin:${PATH}
ENV GHC_PACKAGE_PATH $(stack path --ghc-package-path)
RUN stack config set system-ghc --global true
RUN stack install ihaskell
RUN ihaskell install --stack

I've looked at the documentation on using Dockerfiles and I think I'm doing the right thing with installing Jupyter and running under the user jovyan with ID 1000. When I do a docker build locally and run it myself it seems to work correctly:

$ docker build -t ihaskell-mybinder:v1 .
$ docker run -it -p8888:8888 ihaskell-mybinder:v1 jupyter notebook --ip='0.0.0.0'

Is there something obvious I'm missing? I'm relatively inexperienced with Docker so any help would be appreciated.

Hey all - currently Binder doesn't exist in any kind of state that it could be cited. What do people think about making a quick submission to JOSS so that we can get a DOI and tell people to cite that?

@minrk @yuvipanda @mpacer @jzf2101 @Carreau @fperez @ellisonbg @willingc @betatim (and any others I may have missed)

We've had a few people try and install both Python and R at the same time (I think this is relatively common), and they've almost all run into problems. I think we should create an example repository for how to do this with a postBuild file. This would be a stop-gap until a proper "Python+R" buildpack is created.

@betatim have you gotten both of these things installed in the same repo2docker build before?

We will have an office hour on Thursday February 22nd at 6pm Zurich/9am Berkely time.

The idea is to offer a time and place for people who are using mybinder.org and all its related tools and who have questions about them to meet. There will be several members of the binder team there too. The main meeting point will be the gitter channel (https://gitter.im/jupyterhub/binder) and maybe also a video chat. If you have questions about your Binderhub deployment, want to get involved with the projects or chat about ways to use it come and stop by.

This is an experiment for us as well, so what exactly will happen is up in the air.

How about the above as a starting point to publicise the office hours?

I think we should have a link to binder-examples somewhere early on in the documentation:

https://github.com/binder-examples

This is a collection of example repositories that are all short and to-the-point to show off functionality. I don't think this would be a proper "gallery", it's more about showing how to compose various files in the repository.

Thoughts?