Like shown here: http://www.pymvpa.org/examples/searchlight.html

As mentioned in #2. Gallery images show the header permanent link in red background.
This depends entirely on the sphinx version used. It does not happens in the read the docs default sphinx version or my conda environment with sphinx 1.2.3, but if build with sphinx 1.3 it happens.

Put an extra CSS instruction to avoid this from being displayed?

(Unless I'm mistaken:) Inserting plots from examples requires knowing the path that sphinx-gallery assigns the image. This is a poor abstraction, and it would be nicer if there were an sphinx-gallery-image directive which determines the path from other key information. Another useful directive would place a selected example gallery at any point in the documentation.

Is it possible to have subsubfolders or specify more than one examples directory? (our project differentiates between examples and tutorials, which themselves are structured into topics)

The get_data() function in docs_resolv.py tries to open a cache file at _build/searchindex. If a project uses a non-standard output location the _build directory may not present. As a result the build fails with:

DBNoSuchFileError: (2, 'No such file or directory')

caused by shelve.open(...)

At present, there is no way to tweak this default location from the outside. As a workaround it is sufficient to create a dummy _build directory.

Just wondering whether it should be called sphinx-gallery (with a dash in the middle) rather than sphinxgallery on PyPI. This would be consistent with the github name.

Just to show the feature of grouping examples in diferent subsections.

The simplest is to create a few subfolders in examples/ and move the current example scripts there.

Paths to directories where examples are located and to where they are to be written is deeply coded into the gen_rst.py file.
I was considering to write a dictionary into the conf.py that sphinx uses to generate the documentation to configure the gallery paths.

At mne-python, we have the issue that the entire examples directory takes more than 45 minutes to build (last I built it -- which was admittedly a long time ago). Sometimes, it is useful to link examples in the pages without having to build all the examples. See for instance here. If I use make html_dev-noplot, the links to the examples are missing / broken.

I came up with a quick hack which allows me to build only specific directories. Apart from the fact that it's a hack, it's probably not the best solution in any case. I think the cleanest solution would be something that scans the *.rst files and only builds examples which are linked there. What do you guys think?

cc @agramfort

Because sphinx never deletes empties images in this directory. Sphinx in order to avoid overwriting images in this directory saves new images with a different name. As such html/_images grows larger every time, gen_rst has a hack to empty images in html/_images on every execution.
The problem arises if within the documentation one uses the plot directive from matplotlib .. plot:: file.py
in this case the code is executed once again and a new image is generated with a new filename. But once gen_rst comes to execution it deletes the gallery images(which are regenerated) and the images from the documentation directive. As such it happens that the end documentation does not display images from the plot directive.
Of course a simple make clean walks around the problem.

Now that you can specify multiple examples folders, examples_dir is a bit confusing ...

We use print(__doc__) everywhere in examples and this generates now :

Built-in functions, exceptions, and other objects.

Noteworthy: None is the `nil' object; Ellipsis represents `...' in slices.

See for example in:

http://martinos.org/mne/dev/auto_examples/plot_extract_events_from_raw.html#sphx-glr-auto-examples-plot-extract-events-from-raw-py

Can we avoid this? The reason for the print(__doc__) is that when running the script from the terminal it prints the doc.

Based on this use case to avoid seaborn conflicting with examples and matplotlib defaults.
GaelVaroquaux/stats_in_python_tutorial@b53e0c7

Is there any particular reason they are in setup.cfg?

debug=nose.loader
pdb=1
pdb-failures=1

For example in a CI environment, I think that test would hang on the first error. The nose.loader debug is probably best enabled on the command line in the rare cases when you actually care.

I came to notice that if the examples docstring have an unicode character, sphinx-gallery fails in python2. It is not the case if the unicode characters are outside the main docstring, because the encoding header of the file makes the scripts work. The reason the files having my foreign accented name still work.

this PR scikit-learn/scikit-learn#3777 didn't help with my issue.

More of a reminder based on nilearn/nilearn#592 (comment)

Incorporate touch_example_backreferences as part of the Gallery build process, It touches back references files that have no examples to save some warnings from Sphinx.

The traceback

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/sphinx-gallery/envs/latest/local/lib/python2.7/site-packages/sphinx/cmdline.py", line 245, in main
    app.build(opts.force_all, filenames)
  File "/home/docs/checkouts/readthedocs.org/user_builds/sphinx-gallery/envs/latest/local/lib/python2.7/site-packages/sphinx/application.py", line 282, in build
    self.emit('build-finished', None)
  File "/home/docs/checkouts/readthedocs.org/user_builds/sphinx-gallery/envs/latest/local/lib/python2.7/site-packages/sphinx/application.py", line 497, in emit
    results.append(callback(self, *args))
  File "/home/docs/checkouts/readthedocs.org/user_builds/sphinx-gallery/envs/latest/local/lib/python2.7/site-packages/sphinx_gallery-0.0.11-py2.7.egg/sphinxgallery/docs_resolv.py", line 432, in embed_code_links
    _embed_code_links(app, gallery_conf, gallery_dir)
  File "/home/docs/checkouts/readthedocs.org/user_builds/sphinx-gallery/envs/latest/local/lib/python2.7/site-packages/sphinx_gallery-0.0.11-py2.7.egg/sphinxgallery/docs_resolv.py", line 328, in _embed_code_links
    relative=True)
  File "/home/docs/checkouts/readthedocs.org/user_builds/sphinx-gallery/envs/latest/local/lib/python2.7/site-packages/sphinx_gallery-0.0.11-py2.7.egg/sphinxgallery/docs_resolv.py", line 214, in __init__
    sindex = get_data(searchindex_url, gallery_dir)
  File "/home/docs/checkouts/readthedocs.org/user_builds/sphinx-gallery/envs/latest/local/lib/python2.7/site-packages/sphinx_gallery-0.0.11-py2.7.egg/sphinxgallery/docs_resolv.py", line 66, in get_data
    data = _get_data(url)
  File "/home/docs/checkouts/readthedocs.org/user_builds/sphinx-gallery/envs/latest/local/lib/python2.7/site-packages/sphinx_gallery-0.0.11-py2.7.egg/sphinxgallery/docs_resolv.py", line 48, in _get_data
    with open(url, 'r') as fid:
IOError: [Errno 2] No such file or directory: '/home/docs/checkouts/readthedocs.org/user_builds/sphinx-gallery/checkouts/latest/doc/_build/json/searchindex.js'

Exception occurred:
  File "/home/docs/checkouts/readthedocs.org/user_builds/sphinx-gallery/envs/latest/local/lib/python2.7/site-packages/sphinx_gallery-0.0.11-py2.7.egg/sphinxgallery/docs_resolv.py", line 48, in _get_data
    with open(url, 'r') as fid:
IOError: [Errno 2] No such file or directory: '/home/docs/checkouts/readthedocs.org/user_builds/sphinx-gallery/checkouts/latest/doc/_build/json/searchindex.js'

In the link embedding step it searches for searchindex.js in the json folder and not in the html folder. It might be a change in sphinx from pip?

I was wondering if one could generate the examples in parallel processes?
I now python has a feature for sending processes to individual cpu. I can not be directly used right now because currently we scan recursively the example folder instead of generating a list of example files to treat first. It will also need to keep track of the back references.

The CSS styles should be as such

element {
   prop1: value;
   prop2: value;
}

element2 {
   prop: value;
}

The sphinx extension autosummary was incorporated to automatically generate the in-documentation galleries that display which examples use a specific function.

But this causes that the hyperlinks in code stop working. Aparently in the current inplementation of the autosummary no reference nodes are being created for each documented function. Thus sphinx-gallery can't find the documentation of the functions.

We are trying to build docs (and run examples) in CI setting, but it's hard to figure out if the docs run or failed since sphinx-gallery is ignoring errors. See https://circleci.com/gh/chrisfilo/nilearn/17

Since documenting strings are possible since PR #36. A possible feature would be an optional "Download as IPython Notebook" button for example scripts, when an IPython installation is found.

They still the focus which is very annoying. The solution was mentionned here.

The is a version of the docs on:
http://pythonhosted.org/sphinx-gallery/

I have the feeling that it is not maintained. If this is the case, we should take it down.

A nice to have is the ability to specify an ordering for example directories and examples within them. While this is trivial to do by prefixing with numbers, this breaks any existing URLs. Perhaps we can conceive of a way to specify order.

Is it possible to allow absolute paths to be set for examples_dir and gallery_dir? Or in other words: Is it necessary to transform the input variables to paths relative to app.build.srcdir? If not, this would enable a bit more flexibility when using cmake.

There are features included in this gallery script aimed at scikit-learn.
Such as:

• a hack to treat some images for the carousel. Remove it?
• Should the carousel be included as a feature of the gallery? Certainly it will be an extra module to declutter all the scripts embedded within the main gen_rst. Take scikit-learn implementation or nilearn's?
• An index is generated of all scikit-learn modules used in the examples while gen_rst is executed. This index can be accessed later by sphinx when creating the modules autosummary, to include gallery examples where a specific module is used. I'll try to get this into a separate module as is to embedded is the gen_rst code.

For some builders, it may be better to produce plot files in formats other than PNG. It should be possible to request that the system outputs both png and pdf files, for instance, so that text and lines are not rasterised in LaTeX builds etc. By replacing the file extension in the image directive with .*, Sphinx will choose the format preferred for inclusion by each builder.

However, I've found that many of scikit-learn's plots naively exported as PDF by the matplotlib Agg backend are very very large (e.g. svm/plot_rbf_parameters is >100MB). Perhaps as a heuristic one could mark portions of a figure containing more than threshold vertices to be rasterized:

def set_rasterized_by_n_vertices(obj, threshold=10000):
    if hasattr(obj, 'get_paths'):
        total = sum(len(path.veritces) for path in obj.get_paths() or [])
    else:
        total = 0
    if not hasattr(obj, 'get_children'):
        return 0
    for child in obj.get_children():
        n_child = set_rasterized_by_n_vertices(child, threshold)
        total += n_child
    if total > threshold:
        obj.set_rasterized(True)
        total = 0
    return total

to list main changes in new releases.

From #66:

Do a CHANGES.txt file and 'include' it somewhere in the docs. You can see how joblib does it for instance.

Maybe we want to do something similar as scikit-learn/nilearn and have a doc/whats_new.rst + include the rst in the doc somewhere.

Hi,

I love that examples built with "plot_" prefix will autogenerate a plot. I'm wondering what's the recommended approach to add a custom image to an example that doesn't render a plot. For example, we have a code that returns HTML tables, or a code that launches an application, and I want to put a screenshot of this at the top of the example in the same place where the plot appears in the examples named "plot_". Do you have any suggestions?

Great! Thanks for making this a dedicated package. The only thing I miss in comparison to http://tonysyu.github.io/mpltools/auto_examples/sphinx/plot_plot2rst.html is the ability to add documenting strings that hold rst and the option to place figures within the text/code examples.

This would allow for more notebook-like examples that walk the user through each step.

When #36 is merged, it would be good to find a better way to separate stdout from the input.

Not sure about the best way how to do that at the moment, but just putting "Script output:" doesn't seem to cut it for me ...

I have a python script, which creates the rst files for my module-reference. Now I would like to modify it, such that it automatically includes

.. include:: modules/generated/***
.. raw:: html

    <div style='clear:both'></div>

where *** should be replaced with the example list generated by sphinxgallery. Yet when I call my script prior to make html these files do not exist yet, and if I call it afterwards the module references are not included in the html files. Do you see a clever way to this without building the documentation twice?

Thanks for your help!

I feel there is a section missing in http://sphinx-gallery.readthedocs.org/en/latest/getting_started.html in order to explain how to build the documentation locally and where to find the generated html, in particular the location of the gallery. That would probably go before "Setting up your project".

To showcase the project in the docs.

Namely structlearn and nilearn for now. Maybe others are using it. We can find by searching github for sphinx gallery.

A very minor annoyance, but I thought I would vent it out and ask for opinions, it seems to me that sphinx_gallery for the package name is more in line with sphinx-gallery as the project name.

Basically if people agree it would be wise to fix that sooner rather than later, to reduce the impact on potential users.

I have asked something similar in the closed issue #40, but I still think there is something wrong with the paths. Shouldn't they be relative to the documentation source (i.e. the location of conf.py) rather than relative to the location where sphinx-build is called?

cd ~/tmp
git clone https://github.com/sphinx-gallery/sphinx-gallery
sphinx-build -q sphinx-gallery/doc html_output # does not work
cd sphinx-gallery/doc && sphinx-build -q . html_output # works fine

The output of the first sphinx-build call is:

WARNING: 'default' html theme has been renamed to 'classic'. Please change your html_theme setting either to the new 'alabaster' default theme, or to 'classic' to keep using the old default.

Exception occurred:
  File "/usr/lib/python3.4/os.py", line 237, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '../../../examples'

In a large majority of cases, an error when executing one cell, is going to cause further cells to have errors. I think it just adds unnecessary noise:

This is the root cause of the problem:

  File "/home/ubuntu/nilearn/doc/sphinxext/sphinx_gallery/gen_rst.py", line 427, in execute_script
    exec(code_block, example_globals)
  File "<string>", line 3, in <module>
  File "/home/ubuntu/nilearn/nilearn/datasets/func.py", line 481, in fetch_adhd
    resume=resume, verbose=verbose)
  File "/home/ubuntu/nilearn/nilearn/datasets/utils.py", line 728, in _fetch_files
    raise IOError('Fetching aborted: ' + abort)
IOError: Fetching aborted: CRC check failed 0x55f58d0f != 0xd6aafce5L

And this is all the additional stuff you get from trying to execute cells after the first failure:

Traceback (most recent call last):
  File "/home/ubuntu/nilearn/doc/sphinxext/sphinx_gallery/gen_rst.py", line 427, in execute_script
    exec(code_block, example_globals)
  File "<string>", line 6, in <module>
NameError: name 'func_filenames' is not defined
Traceback (most recent call last):
  File "/home/ubuntu/nilearn/doc/sphinxext/sphinx_gallery/gen_rst.py", line 427, in execute_script
    exec(code_block, example_globals)
  File "<string>", line 4, in <module>
NameError: name 'components_img' is not defined
Traceback (most recent call last):
  File "/home/ubuntu/nilearn/doc/sphinxext/sphinx_gallery/gen_rst.py", line 427, in execute_script
    exec(code_block, example_globals)
  File "<string>", line 4, in <module>
NameError: name 'components_img' is not defined
Traceback (most recent call last):
  File "/home/ubuntu/nilearn/doc/sphinxext/sphinx_gallery/gen_rst.py", line 427, in execute_script
    exec(code_block, example_globals)
  File "<string>", line 3, in <module>
  File "/home/ubuntu/nilearn/nilearn/datasets/func.py", line 481, in fetch_adhd
    resume=resume, verbose=verbose)
  File "/home/ubuntu/nilearn/nilearn/datasets/utils.py", line 695, in _fetch_files
    overwrite=overwrite)
  File "/home/ubuntu/nilearn/nilearn/datasets/utils.py", line 523, in _fetch_file
    data = url_opener.open(request)
  File "/usr/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1222, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1184, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 111] Connection refused| Error while fetching file adhd40_0010064.tgz; dataset fetching aborted.>
Traceback (most recent call last):
  File "/home/ubuntu/nilearn/doc/sphinxext/sphinx_gallery/gen_rst.py", line 427, in execute_script
    exec(code_block, example_globals)
  File "<string>", line 7, in <module>
NameError: name 'func_filenames' is not defined

We have a few umlauts in our project (i.e. in two of the authors lastnames). make html-noplot works fine, but sphinxgallery runs into problems:

Traceback (most recent call last):
  File "/usr/lib64/python3.4/site-packages/sphinx/cmdline.py", line 254, in main
    app.build(force_all, filenames)
  File "/usr/lib64/python3.4/site-packages/sphinx/application.py", line 221, in build
    self.emit('build-finished', None)
  File "/usr/lib64/python3.4/site-packages/sphinx/application.py", line 400, in emit
    results.append(callback(self, *args))
  File "/usr/lib64/python3.4/site-packages/sphinxgallery/docs_resolv.py", line 432, in embed_code_links
    _embed_code_links(app, gallery_conf, gallery_dir)
  File "/usr/lib64/python3.4/site-packages/sphinxgallery/docs_resolv.py", line 328, in _embed_code_links
    relative=True)
  File "/usr/lib64/python3.4/site-packages/sphinxgallery/docs_resolv.py", line 212, in __init__
    sindex = get_data(searchindex_url, gallery_dir)
  File "/usr/lib64/python3.4/site-packages/sphinxgallery/docs_resolv.py", line 66, in get_data
    data = _get_data(url)
  File "/usr/lib64/python3.4/site-packages/sphinxgallery/docs_resolv.py", line 49, in _get_data
    data = fid.read()
  File "/usr/lib64/python3.4/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 10117: ordinal not in range(128)

They appear cause problems when searchindex.js is parsed. Can this be improved?

Minor detail: doc seems to be more widely used than docs. I checked numpy, scipy, scikit-learn, nilearn.

I was wondering if it is necessary to put the stdout of the examples in the build as is done. Would it be better to hide it and instead only output those which failed.
Currently I want to debug why some nilearn examples are not building nilearn/nilearn#592 and I notice that in stdout there is no traceback, and in the rendered html example I only get the message that the script didn't compile.

mfpgt@lab-01:~/Documents/PHD_research/open_source_contributions/nilearn/doc$
make
sphinx-build -D plot_gallery=False -b html -d _build/doctrees . _build/html
Running Sphinx v1.3.1
WARNING: invalid number u'False' for config value 'plot_gallery', ignoring
loading translations [en]... done
loading pickled environment... not yet created
plotting plot_nifti_simple.py
Downloading data from http://www.nitrc.org/frs/download.php/1071/
NYU_TRT_session1a.tar.gz ...

Sphinx experimentally supports parallel building. But plotting can take as much time as the rest of the documentation generation when starting from scratch. Adopting sphinx's parallelisation for all plotting may be a nice option to have.

• joblib must be replaced by shelve
• _path_static should be renamed path_static, as it is used in the conf.py
• the css names should be harmonized. In particular, they should be all given a prefix, such as sphx-glr-...
• functions should not be defined in closures: _select_block, _parse_dict_structure must be moved out of the function where they are defined
• no @author in docstrings, but author as a comment in the beginning of a file, with also the full name of the author, the original authors if any, and "license: BSD"
• the code isn't completely pep8: there should be two empty lines between first level functions

It doesn't seem to be used in the config, right?

Can you please put the example gallery online so that I can check it out without having to download and build it locally?
E.g. on readthedocs?

I'm trying to purge the code of this gallery to mostly autonomous. But it carries into it some extra dependencies that don't seem necesary to me like scipy and scikit-learn.
From scikit-learn it uses it to call joblib, which I still don't know what it does.
But scikit-learn calls for scipy as a dependency

in order to show the functionality that the stdout is captured and shown on the example page.

Also while I am at it seems that nilearn gen_rst.py has a small tweak to restrict the max-height of the output div (I am guessing around there). For example this example has a lot of output.

Hello sphinx-gallery people,

this issue might be not related to sphinx-gallery (but to mayavi instead), so feel free to close it if you think it's not relevant for the project.

I'm using sphinx-gallery master (24a1131), and I tried to build the doc running make html inside the docs folder.

The build works fine until the mayavi example, that crashes the build

plotting code blocks in ../mayavi_examples/plot_3d.py
The program 'sphinx-build' received an X Window System error.
This probably reflects a bug in the program.
The error was 'BadValue (integer parameter out of range for operation)'.
  (Details: serial 55 error_code 2 request_code 154 minor_code 3)
  (Note to programmers: normally, X errors are reported asynchronously;
   that is, you will receive the error a while after causing it.
   To debug your program, run it with the --sync command line
   option to change this behavior. You can then get a meaningful
   backtrace from your debugger if you break on the gdk_x_error() function.)
Makefile:62: recipe for target 'html' failed
make: *** [html] Error 1

It might be a problem with my installation of Mayavi (but I tried both with mayavi 4.3 packaged by Ubuntu and with mayavi master 4.4.4dev) , since closing a Mayavi figure inside of IPython results in a segmentation fault. Adding

from mayavi.core.engine import Engine
from mayavi import mlab

e = Engine()
e.start()
mlab.set_engine(e)

to the script solves the problem for me, since the build is not crashing any more.

Also, it seems that Travis is not testing the build of mayavi gallery examples (I might be wrong, but a quick search through Travis logs convinced me that it is not tested). This example might be skipped on purpose, but since .travis.yml says "# This environment tests is for mayavi" (https://github.com/sphinx-gallery/sphinx-gallery/blob/master/.travis.yml#L18), I was a bit puzzled.