Code Monkey home page Code Monkey logo

hdmf-docutils's Introduction

HDMF Documentation Utilities

This project is under active development. Its content, API and behavior may change at any time. We mean it.

PyPI - License PyPI Build Status

Overview

This project is a collection of CLIs, scripts and modules useful to generate the HDMF documentation.

Using hdmf-docutils to generate documentation for an extension: http://pynwb.readthedocs.io/en/latest/extensions.html#documenting-extensions

To cite this tool use: (HDMF Documentation Utilities, RRID:SCR_021341)

Installation

pip install hdmf-docutils

Available Tools

  • hdmf_generate_format_docs: Generate figures and RST documents from the HDMF YAML specification for the format specification documentation. Previously called "nwb_generate_format_docs".
  • hdmf_init_sphinx_extension_doc: Create format specification SPHINX documentation for an HDMF extension. Previously called "nwb_init_sphinx_extension_doc".
  • hdmf_gallery_prototype: Tool for prototyping sphinx gallery examples. Previously called "nwb_gallery_prototype".

Available Modules

  • hdmf_docutils/doctools/*: This package contains modules used to generate figures of the hierarchies of HDMF files and specifications as well as to help with the programmatic generation of reStructuredText (RST) documents.

Available Notebooks

History

nwb-docutils was renamed to hdmf-docutils and generalized to be (mostly) independent of NWB in January, 2020.

nwb-docutils was initially a sub-directory of the nwb-schema project. Corresponding history was extracted during the 4th NWB Hackathon into a dedicated pip-installable project to facilitate its use by both core NWB documentation projects and various NWB extensions.

Usage

pip install hdmf-docutils

For the purpose of this example, we assume that our current directory has the following structure.

- my_extension/
  - my_extension_source/
      - mylab.namespace.yaml
      - mylab.specs.yaml
      - ...
      - docs/  (Optional)
          - mylab_description.rst
          - mylab_release_notes.rst

In addition to Python 3.x, you will also need sphinx (including the sphinx-quickstart tool) installed. Sphinx is available here http://www.sphinx-doc.org/en/stable/install.html .

We can now create the sources of our documentation as follows:

   python3 hdmf_init_sphinx_extension_doc  \
                --project my-extension \
                --author "Dr. Master Expert" \
                --version "1.2.3" \
                --release alpha \
                --output my_extension_docs \
                --spec_dir my_extension_source \
                --namespace_filename mylab.namespace.yaml \
                --default_namespace mylab
                --external_description my_extension_source/docs/mylab_description.rst \  (Optional)
                --external_release_notes my_extension_source/docs/mylab_release_notes.rst \  (Optional)

.. tip::

   Additional instructions for how to use and customize the extension documentations are also available
   in the ``Readme.md`` file that  ``init_sphinx_extension_doc.py`` automatically adds to the docs.

Tip

See make help for a list of available options for building the documentation in many different output formats (e.g., PDF, ePub, LaTeX, etc.).

Tip

See python3 init_sphinx_extension_doc.py --help for a complete list of option to customize the documentation directly during initialization.

Tip

The above example included additional description and release note docs as part of the specification. These are included in the docs via .. include commands so that changes in those files are automatically picked up when rebuilding to docs. Alternatively, we can also add custom documentation directly to the docs. In this case the options --custom_description format_description.rst and --custom_release_notes format_release_notes.rst of the init_sphinx_extension_doc.py script are useful to automatically generate the basic setup for those files so that one can easily start to add content directly without having to worry about the additional setup.

hdmf-docutils's People

Contributors

bendichter avatar jcfr avatar mavaylon1 avatar nicain avatar nilegraddis avatar oruebel avatar rly avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

hdmf-docutils's Issues

make apidoc error

following https://pynwb.readthedocs.io/en/latest/extensions.html#documenting-extensions

tried in https://github.com/bendichter/nwbext_simulation_output/tree/follow_guidelines

$ make apidoc
PYTHONPATH=/Users/bendichter/dev/nwbext_simulation_output/docs/source: nwb_generate_format_docs
Output directory already exists: /Users/bendichter/dev/nwbext_simulation_output/docs/source/_format_auto_docs

SORTING TYPES INTO SECTIONS
---------------------------
Traceback (most recent call last):
  File "/Users/bendichter/anaconda3/bin/nwb_generate_format_docs", line 11, in <module>
    load_entry_point('nwb-docutils', 'console_scripts', 'nwb_generate_format_docs')()
  File "/Users/bendichter/dev/nwb-docutils/nwb_docutils/generate_format_docs.py", line 473, in main
    type_sections = DataTypeSection.sort_types_to_sections(default_namespace)
  File "/Users/bendichter/dev/nwb-docutils/nwb_docutils/doctools/renderrst.py", line 78, in sort_types_to_sections
    sections[spec_filename]['data_types'].append(nt)
KeyError: 'nwb.base.yaml'
make: *** [apidoc] Error 1

Links listed twice

Links are listed in both table of "Datasets, Links, and Attributes" as well as the list of "Groups". They should be listed only in one.

Decide whether to show included spec fields

make apidoc gives the following error:

Bens-MacBook-Pro:docs bendichter$ make apidoc
PYTHONPATH=/Users/bendichter/dev/nwb-extensions/simulation_output/docs/nwb_docutils:/Users/bendichter/dev/nwb-extensions/simulation_output/docs/source:/usr/local/hdf5: python /Users/bendichter/dev/nwb-extensions/simulation_output/docs/nwb_docutils/generate_format_docs.py
/Users/bendichter/dev/nwb-extensions/simulation_output/docs/nwb_docutils/generate_format_docs.py:60: UserWarning: DISABLING RENDERING OF SPEC GRAPHS DUE TO IMPORT ERROR
  warnings.warn('DISABLING RENDERING OF SPEC GRAPHS DUE TO IMPORT ERROR')
Generating output directory: /Users/bendichter/dev/nwb-extensions/simulation_output/docs/source/_format_auto_docs
Traceback (most recent call last):
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/spec/namespace.py", line 379, in load_namespaces
    inc_ns = self.get_namespace(s['namespace'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/utils.py", line 348, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/spec/namespace.py", line 240, in get_namespace
    raise KeyError("'%s' not a namespace" % name)
KeyError: "'core' not a namespace"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/bendichter/dev/nwb-extensions/simulation_output/docs/nwb_docutils/generate_format_docs.py", line 1356, in <module>
    main()
  File "/Users/bendichter/dev/nwb-extensions/simulation_output/docs/nwb_docutils/generate_format_docs.py", line 1263, in main
    resolve=spec_resolve_type_inc)
  File "/Users/bendichter/dev/nwb-extensions/simulation_output/docs/nwb_docutils/generate_format_docs.py", line 88, in load_nwb_namespace
    namespace.load_namespaces(namespace_file, resolve=resolve)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/utils.py", line 348, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/spec/namespace.py", line 381, in load_namespaces
    raise ValueError("Could not load namespace '%s'" % s['namespace'])
ValueError: Could not load namespace 'core'
make: *** [apidoc] Error 1

Update nwb-docutils to reflect new output requirements captured in nwb-extensions/ndx-template

the differences reported below were gathered by comparing the content of the docs folder generated using https://github.com/nwb-extensions/ndx-template and the one generated using nwb_init_sphinx_extension_doc:

  • docs/conf.py
    • update list of extensions in conf.py
    • htmlhelp_basename set to simulation_output_doc instead of simulation_outputdoc
    • add missing intersphinx_mapping var

These differences are not directly related to nwb-docutils but are documented here for future reference. They corresponds to differences between the cookiecutter and example like this one

  • ReadMe.md -> README.md
  • license.md -> LICENSE.md
  • pynwb-src -> src/pynwb and src/spec

sg_prototype test fails on remote CI

The sg_prototype test fails on Azure and GitHub CI but succeeds locally. It might be an issue with environments. The failure seems to have to do with Sphinx modules not being loaded correctly.

@nicain @NileGraddis Is there still a need for sg_prototype.py?

It looks like it was developed during the April 2018 hackathon to allow rapid prototyping of Sphinx Gallery examples without rebuilding the full documentation, but it has not been used or updated anytime after that.
NeurodataWithoutBorders/pynwb#414
NeurodataWithoutBorders/pynwb#429
#1

Can it be removed? Since it is not in use, as far as I can tell, let's comment out the test.

Heading underline to short in index.rst

In the init_sphinx_extension_doc.py script get_index_rst the name of the extension is added to heading but the underline for the heading is not adjusted to match the length of the title. This results in a heading underline that is too short and results in a subsequent warning from Sphynx when rendering the docs.

sg_prototype input parsing fails

env: Python 3.6, Ubuntu 17.10

Running:

nwb_gallery_prototype pynwb/docs/gallery/domain/ecephys.py -o <output_path>

fails with:

FileNotFoundError: [Errno 2] No such file or directory: 'p'

The culprit is optional input argument parsing which slices off the first element of the supplied input paths.

Error generating docs for LinkSpec without name

A group spec in an extension has:

        links=[
            NWBLinkSpec(
                doc="The Subject object in the NWB file, if this Skeleton corresponds to the Subject.",
                target_type="Subject",
                quantity="?",
            ),
        ],

Running sphinx-build -b html -v -d build/doctrees source build/html results in:

Traceback (most recent call last):
  File "/Users/rly/mambaforge/envs/test/lib/python3.11/site-packages/sphinx/events.py", line 97, in emit
    results.append(listener.handler(self.app, *args))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rly/Documents/NWB/ndx-pose/docs/source/conf.py", line 73, in run_doc_autogen
    generate_docs()
  File "/Users/rly/mambaforge/envs/test/lib/python3.11/site-packages/hdmf_docutils/generate_format_docs.py", line 535, in main
    render_data_type_section(section=sec,
  File "/Users/rly/mambaforge/envs/test/lib/python3.11/site-packages/hdmf_docutils/generate_format_docs.py", line 334, in render_data_type_section
    rt_spec_data_table = SpecToRST.render_spec_table(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rly/mambaforge/envs/test/lib/python3.11/site-packages/hdmf_docutils/doctools/renderrst.py", line 697, in render_spec_table
    SpecToRST.render_spec_table(spec=link,
  File "/Users/rly/mambaforge/envs/test/lib/python3.11/site-packages/hdmf_docutils/doctools/renderrst.py", line 644, in render_spec_table
    elif spec.data_type_def is not None:
         ^^^^^^^^^^^^^^^^^^
AttributeError: 'LinkSpec' object has no attribute 'data_type_def'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/rly/mambaforge/envs/test/lib/python3.11/site-packages/sphinx/cmd/build.py", line 293, in build_main
    app = Sphinx(args.sourcedir, args.confdir, args.outputdir,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rly/mambaforge/envs/test/lib/python3.11/site-packages/sphinx/application.py", line 272, in __init__
    self._init_builder()
  File "/Users/rly/mambaforge/envs/test/lib/python3.11/site-packages/sphinx/application.py", line 343, in _init_builder
    self.events.emit('builder-inited')
  File "/Users/rly/mambaforge/envs/test/lib/python3.11/site-packages/sphinx/events.py", line 108, in emit
    raise ExtensionError(__("Handler %r for event %r threw an exception") %
sphinx.errors.ExtensionError: Handler <function run_doc_autogen at 0x1032058a0> for event 'builder-inited' threw an exception (exception: 'LinkSpec' object has no attribute 'data_type_def')

TypeError: draw_networkx_nodes() got an unexpected keyword argument 'font_family'

When generating HDMF Common documentation, I get the following error:

  File "c:\users\ryan\miniconda3\envs\dev\lib\site-packages\hdmf_docutils\doctools\render.py", line 682, in draw_g
raph                                                                                                              
    nx.draw_networkx_nodes(graph, pos,                                                                            
TypeError: draw_networkx_nodes() got an unexpected keyword argument 'font_family'     

Networkx docs for draw_network_nodes() says that 'font_family' is not a keyword argument.
https://networkx.org/documentation/stable/reference/generated/networkx.drawing.nx_pylab.draw_networkx_nodes.html

New in networkx 2.5, released on 22 August 2020, is that extra keywords are no longer accepted in draw_network_nodes().

As far as I can tell from the networkx 2.4 source code, the extra keywords were never used, so removing 'font_family' from the call to draw_network_nodes() should result in no change in behavior.

Full output:

(dev) λ make html                                                                                                 
Running Sphinx v3.5.4                                                                                             
making output directory... done                                                                                   
Removed old sources at: C:\Users\Ryan\Documents\NWB\hdmf-common-schema\docs\source\_format_auto_docs              
Generating output directory: C:\Users\Ryan\Documents\NWB\hdmf-common-schema\docs\source\_format_auto_docs         
                                                                                                                  
SORTING TYPES INTO SECTIONS                                                                                       
---------------------------                                                                                       
Base data types                                                                                                   
    base data types                                                                                               
    ['Data', 'Container', 'SimpleMultiContainer']                                                                 
Table data types                                                                                                  
    data types for a column-based table                                                                           
    ['VectorData', 'VectorIndex', 'ElementIdentifiers', 'DynamicTableRegion', 'DynamicTable', 'AlignedDynamicTable
']                                                                                                                
Sparse data types                                                                                                 
    data types for different types of sparse matrices                                                             
    ['CSRMatrix']                                                                                                 
RENDERING TYPE HIERARCHY                                                                                          
------------------------                                                                                          
Container                                                                                                         
   CSRMatrix                                                                                                      
   DynamicTable                                                                                                   
      AlignedDynamicTable                                                                                         
   SimpleMultiContainer                                                                                           
Data                                                                                                              
   ElementIdentifiers                                                                                             
   VectorData                                                                                                     
      DynamicTableRegion                                                                                          
      VectorIndex                                                                                                 
RENDERING NAMESPACE SPECIFICATION                                                                                 
---------------------------------                                                                                 
    hdmf-common-- WRITE NAMESPACE DESCRIPTION DOC OK.                                                             
    hdmf-common-- WRITE NAMESPACE SOURCE DOC OK.                                                                  
RENDERING TYPE SPECIFICATIONS                                                                                     
------------------------------                                                                                    
BUILDING Data                                                                                                     
    Data-- SKIPPED RENDER HIERARCHY. TWO OR FEWER NODES.                                                          
    Data-- WRITE DESCRIPTION DOC OK.                                                                              
    Data-- WRITE SOURCE DOC OK.                                                                                   
BUILDING Container                                                                                                
    Container-- SKIPPED RENDER HIERARCHY. TWO OR FEWER NODES.                                                     
    Container-- WRITE DESCRIPTION DOC OK.                                                                         
    Container-- WRITE SOURCE DOC OK.                                                                              
BUILDING SimpleMultiContainer                                                                                     
SimpleMultiContainer-- RENDER HIERARCHY FAILED: Traceback (most recent call last):                                
  File "c:\users\ryan\miniconda3\envs\dev\lib\site-packages\hdmf_docutils\generate_format_docs.py", line 268, in r
ender_data_type_section                                                                                           
    fig = temp_graph.draw(show_plot=False,     # noqa F841                                                        
  File "c:\users\ryan\miniconda3\envs\dev\lib\site-packages\hdmf_docutils\doctools\render.py", line 343, in draw  
    return self.draw_graph(graph=self.graph, pos=self.pos, data=self.data, **kwargs)                              
  File "c:\users\ryan\miniconda3\envs\dev\lib\site-packages\hdmf_docutils\doctools\render.py", line 682, in draw_g
raph                                                                                                              
    nx.draw_networkx_nodes(graph, pos,                                                                            
TypeError: draw_networkx_nodes() got an unexpected keyword argument 'font_family'                                 
                                                                                                                  
    SimpleMultiContainer-- WRITE DESCRIPTION DOC OK.                                                              
    SimpleMultiContainer-- WRITE SOURCE DOC OK.                                                                   
BUILDING VectorData                                                                                               
    VectorData-- SKIPPED RENDER HIERARCHY. TWO OR FEWER NODES.                                                    
    VectorData-- WRITE DESCRIPTION DOC OK.                                                                        
    VectorData-- WRITE SOURCE DOC OK.                                                                             
BUILDING VectorIndex                                                                                              
    VectorIndex-- SKIPPED RENDER HIERARCHY. TWO OR FEWER NODES.                                                   
    VectorIndex-- WRITE DESCRIPTION DOC OK.                                                                       
    VectorIndex-- WRITE SOURCE DOC OK.                                                                            
BUILDING ElementIdentifiers                                                                                       
    ElementIdentifiers-- SKIPPED RENDER HIERARCHY. TWO OR FEWER NODES.                                            
    ElementIdentifiers-- WRITE DESCRIPTION DOC OK.                                                                
    ElementIdentifiers-- WRITE SOURCE DOC OK.                                                                     
BUILDING DynamicTableRegion                                                                                       
DynamicTableRegion-- RENDER HIERARCHY FAILED: Traceback (most recent call last):                                  
  File "c:\users\ryan\miniconda3\envs\dev\lib\site-packages\hdmf_docutils\generate_format_docs.py", line 268, in r
ender_data_type_section                                                                                           
    fig = temp_graph.draw(show_plot=False,     # noqa F841                                                        
  File "c:\users\ryan\miniconda3\envs\dev\lib\site-packages\hdmf_docutils\doctools\render.py", line 343, in draw  
    return self.draw_graph(graph=self.graph, pos=self.pos, data=self.data, **kwargs)                              
  File "c:\users\ryan\miniconda3\envs\dev\lib\site-packages\hdmf_docutils\doctools\render.py", line 682, in draw_g
raph                                                                                                              
    nx.draw_networkx_nodes(graph, pos,                                                                            
TypeError: draw_networkx_nodes() got an unexpected keyword argument 'font_family'                                 
                                                                                                                  
    DynamicTableRegion-- WRITE DESCRIPTION DOC OK.                                                                
    DynamicTableRegion-- WRITE SOURCE DOC OK.                                                                     
BUILDING DynamicTable                                                                                             
DynamicTable-- RENDER HIERARCHY FAILED: Traceback (most recent call last):                                        
  File "c:\users\ryan\miniconda3\envs\dev\lib\site-packages\hdmf_docutils\generate_format_docs.py", line 268, in r
ender_data_type_section                                                                                           
    fig = temp_graph.draw(show_plot=False,     # noqa F841                                                        
  File "c:\users\ryan\miniconda3\envs\dev\lib\site-packages\hdmf_docutils\doctools\render.py", line 343, in draw  
    return self.draw_graph(graph=self.graph, pos=self.pos, data=self.data, **kwargs)                              
  File "c:\users\ryan\miniconda3\envs\dev\lib\site-packages\hdmf_docutils\doctools\render.py", line 682, in draw_g
raph                                                                                                              
    nx.draw_networkx_nodes(graph, pos,                                                                            
TypeError: draw_networkx_nodes() got an unexpected keyword argument 'font_family'                                 
                                                                                                                  
    DynamicTable-- WRITE DESCRIPTION DOC OK.                                                                      
    DynamicTable-- WRITE SOURCE DOC OK.                                                                           
BUILDING AlignedDynamicTable                                                                                      
AlignedDynamicTable-- RENDER HIERARCHY FAILED: Traceback (most recent call last):                                 
  File "c:\users\ryan\miniconda3\envs\dev\lib\site-packages\hdmf_docutils\generate_format_docs.py", line 268, in r
ender_data_type_section                                                                                           
    fig = temp_graph.draw(show_plot=False,     # noqa F841                                                        
  File "c:\users\ryan\miniconda3\envs\dev\lib\site-packages\hdmf_docutils\doctools\render.py", line 343, in draw  
    return self.draw_graph(graph=self.graph, pos=self.pos, data=self.data, **kwargs)                              
  File "c:\users\ryan\miniconda3\envs\dev\lib\site-packages\hdmf_docutils\doctools\render.py", line 682, in draw_g
raph                                                                                                              
    nx.draw_networkx_nodes(graph, pos,                                                                            
TypeError: draw_networkx_nodes() got an unexpected keyword argument 'font_family'                                 
                                                                                                                  
    AlignedDynamicTable-- WRITE DESCRIPTION DOC OK.                                                               
    AlignedDynamicTable-- WRITE SOURCE DOC OK.                                                                    
BUILDING CSRMatrix                                                                                                
CSRMatrix-- RENDER HIERARCHY FAILED: Traceback (most recent call last):                                           
  File "c:\users\ryan\miniconda3\envs\dev\lib\site-packages\hdmf_docutils\generate_format_docs.py", line 268, in r
ender_data_type_section                                                                                           
    fig = temp_graph.draw(show_plot=False,     # noqa F841                                                        
  File "c:\users\ryan\miniconda3\envs\dev\lib\site-packages\hdmf_docutils\doctools\render.py", line 343, in draw  
    return self.draw_graph(graph=self.graph, pos=self.pos, data=self.data, **kwargs)                              
  File "c:\users\ryan\miniconda3\envs\dev\lib\site-packages\hdmf_docutils\doctools\render.py", line 682, in draw_g
raph                                                                                                              
    nx.draw_networkx_nodes(graph, pos,                                                                            
TypeError: draw_networkx_nodes() got an unexpected keyword argument 'font_family'                                 
                                                                                                                  
    CSRMatrix-- WRITE DESCRIPTION DOC OK.                                                                         
    CSRMatrix-- WRITE SOURCE DOC OK.                                                                              
Write C:\Users\Ryan\Documents\NWB\hdmf-common-schema\docs\source\_format_auto_docs\format_spec_doc.inc            
Write C:\Users\Ryan\Documents\NWB\hdmf-common-schema\docs\source\_format_auto_docs\format_spec_sources.inc        
Write C:\Users\Ryan\Documents\NWB\hdmf-common-schema\docs\source\_format_auto_docs\format_spec_main.inc           
building [mo]: targets for 0 po files that are out of date                                                        
building [html]: targets for 6 source files that are out of date                                                  
updating environment: [new config] 6 added, 0 changed, 0 removed                                                  
reading sources... [100%] software_process                                                                        
looking for now-outdated files... none found                                                                      
pickling environment... done                                                                                      
checking consistency... done                                                                                      
preparing documents... done                                                                                       
writing output... [100%] software_process                                                                         
generating indices... genindex done                                                                               
writing additional pages... search done                                                                           
copying images... [100%] figures/ragged-array.png                                                                 
copying static files... done                                                                                      
copying extra files... done                                                                                       
dumping search index in English (code: en)... done                                                                
dumping object inventory... done                                                                                  
build succeeded.                                                                                                  
                                                                                                                  
The HTML pages are in _build\html.                                                                                
                                                                                                                  
Build finished. The HTML pages are in _build/html.                                                                

Generated conf.py is broken

The generated conf.py includes the following:

# -- Options for intersphinx extension ---------------------------------------

intersphinx_mapping = {
############################################################################
#  CUSTOM CONFIGURATIONS ADDED BY THE NWB TOOL FOR GENERATING FORMAT DOCS
###########################################################################

import sphinx_rtd_theme  # noqa: E402
import textwrap  # noqa: E402

# -- Options for intersphinx  ---------------------------------------------
intersphinx_mapping = {'core': ('https://nwb-schema.readthedocs.io/en/latest/', None)}

Note the syntax error at intersphinx_mapping = {. This is due to the (hacky) file cleaning in init_sphinx_extension_doc.py. It looks like much of this cleaning code can be removed and/or replaced.

This whole script should probably be cleaned up, but that is a separate, larger issue.

Build for core schema broken

#5 broke the generation of the format docs for the core schema because the namespace is already loaded from the spec that ships with PyNWB

nwb_init_sphinx_extension_doc: Improve settings of default values

Get metadata from namespace file, and better default values

If specified in namespace file, obtain author, version, project name, ... from it. Additionally reasonable default could be set for few other parameters.

Doing so would simplify the invocation of nwb_init_sphinx_extension_doc from:

nwb_init_sphinx_extension_doc  \
             --project simulation_output \
             --author "Ben Dichter" \
             --version "0.2.0" \
             --release alpha \
             --output docs2 \
             --spec_dir /tmp/nwbext_simulation_output/spec \
             --namespace_filename simulation_output.namespace.yaml \
             --default_namespace simulation_output

to

nwb_init_sphinx_extension_doc  \
             --output docs \
             --spec_dir /tmp/nwbext_simulation_output/spec \
             --namespace_filename simulation_output.namespace.yaml

where:

  • author is the first name specified in the namespace file
  • project and default_namespace are the namespace name
  • release is initialized to alpha by default

The namespace file could also be automatically found.

automatic updates of namespace file

On the other hand, if information like author, version are passed to nwb_init_sphinx_extension_doc, the namespace file could be updated if it doesn't already contain the info.

Last, if the info are already in the namespace file and the one passed in the command line contradict ... an exception could be raised.

how do I add images to the sphinx gallery?

I can add a link to an image with

# .. image:: multicompartment_schema_1.png

but this creates a dead link because the image isn't copied over to the _sources directory. Is there a way to do this automatically?

NWBNamespace.get_source_files missing

https://github.com/NeurodataWithoutBorders/nwb-docutils/blob/1c1ac1761450b11311bbe29147fdb1525747bfcf/nwb_docutils/doctools/renderrst.py#L66

NWBNamespace has no method get_source_files and neither does hdmf.spec.SpecNamespace.

I think this bug can be resolved by instead using namespace['schema'], but then I find that get_source_description is also missing. This bug comes up when running make apidoc and make html and causes the following traceback:

(base) Bens-MacBook-Pro-2:docs bendichter$ make apidoc
PYTHONPATH=/Users/bendichter/dev/nwbext_simulation_output/docs/source: nwb_generate_format_docs
Output directory already exists: /Users/bendichter/dev/nwbext_simulation_output/docs/source/_format_auto_docs

SORTING TYPES INTO SECTIONS
---------------------------
Traceback (most recent call last):
  File "/Users/bendichter/anaconda3/bin/nwb_generate_format_docs", line 11, in <module>
    load_entry_point('nwb-docutils', 'console_scripts', 'nwb_generate_format_docs')()
  File "/Users/bendichter/dev/nwb-docutils/nwb_docutils/generate_format_docs.py", line 473, in main
    type_sections = DataTypeSection.sort_types_to_sections(default_namespace)
  File "/Users/bendichter/dev/nwb-docutils/nwb_docutils/doctools/renderrst.py", line 67, in sort_types_to_sections
    spec_descr = namespace.get_source_description(spec_filename)
AttributeError: 'NWBNamespace' object has no attribute 'get_source_description'
make: *** [apidoc] Error 1
(base) Bens-MacBook-Pro-2:docs bendichter$ 

add clobber option to sg_prototype

When I use sg_prototype, I often want to make changes in the source, then immediately recompile and check the outputs. The check_tgt_dir function raises an error if the output directory already exists. As a minor QoL change, I would like to add an optional flag which clobbers existing outputs.

Add support for UML class diagrams

Generating extension docs yields encoding error

If I pip install nwb-docutils and then run nwb_init_sphinx_extension_doc on Windows 10, Python 3.7, I get the following error:

Creating file C:\Users\Ryan\Documents\NWB\nwbext_simulation_output\docs\source\conf.py.
Creating file C:\Users\Ryan\Documents\NWB\nwbext_simulation_output\docs\source\index.rst.
Creating file C:\Users\Ryan\Documents\NWB\nwbext_simulation_output\docs\make.bat.

Finished: An initial directory structure has been created.

You should now populate your master file C:\Users\Ryan\Documents\NWB\nwbext_simulation_output\docs\source\index.rst and create other documentation
source files. Use the Makefile to build the docs, like so:
   make builder
where "builder" is one of the supported builders, e.g. html, latex or linkcheck.

Traceback (most recent call last):
  File "c:\users\ryan\anaconda3\envs\nwb-ext-test\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\ryan\anaconda3\envs\nwb-ext-test\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\Ryan\Anaconda3\envs\nwb-ext-test\Scripts\nwb_init_sphinx_extension_doc.exe\__main__.py", line 9, in <module>
  File "c:\users\ryan\anaconda3\envs\nwb-ext-test\lib\site-packages\nwb_docutils\init_sphinx_extension_doc.py", line 918, in main
    write_custom_conf(**clargs)
  File "c:\users\ryan\anaconda3\envs\nwb-ext-test\lib\site-packages\nwb_docutils\init_sphinx_extension_doc.py", line 784, in write_custom_conf
    outfile.write(custom_doc_autogen_settings)
  File "c:\users\ryan\anaconda3\envs\nwb-ext-test\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u2192' in position 3077: character maps to <undefined>

\u2192 is the rightwards arrow →
which is in one of the comments generated by conf_doc_autogen.py.

Solution: Encode strings in Unicode before writing, e.g.
outfile.write(custom_doc_autogen_settings) -> outfile.write(custom_doc_autogen_settings.encode('utf8'))

Support figure width in %

hdmf_docutils.doctools.rst.RSTDocument.add_figure currently requires width and height to be specified in pixel. It would be useful to allow figure width to be specified in percent as well.

Add rendering for DynamicTable

We should add a way to render DynamicTables as an actual table to better illustrate the created structure in the docs.

YAML error: "dump()" has been removed

In ndx-hed, running the command:

sphinx-build -b linkcheck ./source ./test_build

results in:

Extension error:
Handler <function run_doc_autogen at 0x103403ba0> for event 'builder-inited' threw an exception (exception: 
"dump()" has been removed, use

  yaml = YAML(typ='unsafe', pure=True)
  yaml.dump(...)

instead of file "/Users/rly/mambaforge/envs/ndx-hed/lib/python3.11/site-packages/hdmf_docutils/doctools/rst.py", line 299

            return yaml.dump(clean_spec, default_flow_style=False)

)

The latest ruamel.yaml is 0.18.0 which removes the dump() function:

As announced, in 0.18.0, the old PyYAML functions have been deprecated. (scan, parse, compose, load, emit, serialize, dump and their variants (all, safe, round_trip_, etc)). If you only read this after your program has stopped working: I am sorry to hear that, but that also means you, or the person developing your program, has not tested with warnings on (which is the recommendation in PEP 565, and e.g. defaultin when using pytest).

Pip environment:

Package                       Version      Editable project location
----------------------------- ------------ --------------------------------
alabaster                     0.7.13
attrs                         23.1.0
Babel                         2.13.1
black                         23.9.1
cachetools                    5.3.2
certifi                       2023.7.22
cfgv                          3.4.0
chardet                       5.2.0
charset-normalizer            3.3.1
click                         8.1.7
codespell                     2.2.6
colorama                      0.4.6
contourpy                     1.1.1
coverage                      7.3.2
cycler                        0.12.1
distlib                       0.3.7
docutils                      0.18.1
filelock                      3.12.4
fonttools                     4.43.1
h5py                          3.10.0
hdmf                          3.10.0
hdmf-docutils                 0.4.5
identify                      2.5.30
idna                          3.4
imagesize                     1.4.1
iniconfig                     2.0.0
Jinja2                        3.1.2
jsonschema                    4.19.1
jsonschema-specifications     2023.7.1
kiwisolver                    1.4.5
MarkupSafe                    2.1.3
matplotlib                    3.8.0
mypy-extensions               1.0.0
ndx-hed                       0.1.0        /Users/rly/Documents/NWB/ndx-hed
networkx                      3.2
nodeenv                       1.8.0
numpy                         1.26.1
packaging                     23.2
pandas                        2.1.1
pathspec                      0.11.2
Pillow                        10.1.0
pip                           23.3.1
platformdirs                  3.11.0
pluggy                        1.3.0
pre-commit                    3.4.0
Pygments                      2.16.1
pynwb                         2.5.0
pyparsing                     3.1.1
pyproject-api                 1.6.1
pytest                        7.4.2
pytest-cov                    4.1.0
pytest-subtests               0.6.0
python-dateutil               2.8.2
pytz                          2023.3.post1
PyYAML                        6.0.1
referencing                   0.30.2
requests                      2.31.0
rpds-py                       0.10.6
ruamel.yaml                   0.18.2
ruamel.yaml.clib              0.2.8
ruff                          0.0.292
scipy                         1.11.3
setuptools                    68.2.2
six                           1.16.0
snowballstemmer               2.2.0
Sphinx                        7.2.6
sphinx-gallery                0.14.0
sphinx-rtd-theme              1.3.0
sphinxcontrib-applehelp       1.0.7
sphinxcontrib-devhelp         1.0.5
sphinxcontrib-htmlhelp        2.0.4
sphinxcontrib-jquery          4.1
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.6
sphinxcontrib-serializinghtml 1.1.9
tox                           4.11.3
tzdata                        2023.3
urllib3                       2.0.7
virtualenv                    20.24.6
wheel                         0.41.2

nwb_gallery_prototype error

I ran:

nwb_gallery_prototype docs/gallery/general/extensions.py

error:

Traceback (most recent call last):
  File "/Users/bendichter/anaconda3/bin/nwb_gallery_prototype", line 11, in <module>
    sys.exit(main())
  File "/Users/bendichter/anaconda3/lib/python3.6/site-packages/nwb_docutils/sg_prototype.py", line 124, in main
    return build(src_file, tgt_dir=tgt_dir, open_html=open_html)
  File "/Users/bendichter/anaconda3/lib/python3.6/site-packages/nwb_docutils/sg_prototype.py", line 54, in build
    shutil.copy(src_file, os.path.join(temp_dir, 'docs', 'gallery', os.path.basename(src_file)))
  File "/Users/bendichter/anaconda3/lib/python3.6/shutil.py", line 241, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/Users/bendichter/anaconda3/lib/python3.6/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: 'd'

pip install error

Obtaining file:///Users/bendichter/dev/nwb-docutils
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/Users/bendichter/dev/nwb-docutils/setup.py", line 7, in <module>
        with open('README.doctools', 'r') as fp:
    FileNotFoundError: [Errno 2] No such file or directory: 'README.doctools'

Use standard HDMF spec classes in render.py

render.HierarchyDescription should use the GroupSec, DatasetSpec etc. classes from HDMF rather than defining its on dicts for the hierarchy. Since this is only used in render it is not critical but should be addressed to make the functionality also more broadly useful and compliant with HDMF.

update sorting of types to sections

In the generate_format_docs script the sorting of types to sections for the core is implemented in the function sort_type_hierarchy_to_sections https://github.com/NeurodataWithoutBorders/nwb-docutils/blob/69de69741e6e46f7360d9c914fe7b989fe300339/nwb_docutils/generate_format_docs.py#L269

This function is brittle in that type names are hard-coded to sort the core types to sections. Because of this, this function tends to break whenever types are changed in the core. The strategy for sorting into sections should be updated to make it less brittle. We could either have a fully automatic strategy (e.g. by sorting types based on the YAML they are defined in) or we could have a configuration file alongside the YAML schema to control the sorting. In that way, at least everything is controlled from the schema rather than having to update to docutils code. This would also be useful to make the tool more generally useful for extensions.

Setup GitHub test pipelines

As part of PRs we should run the following test pipelines:

  1. Build the NWB schema docs to ensure changes are not affecting the docs
  2. Build hdmf-common-schema docs

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.