hdmf-dev / hdmf-docutils Goto Github PK

View Code? Open in Web Editor NEW

0.0 0.0 3.0 1.89 MB

Collection of CLIs, scripts and modules useful to generate HDMF/NWB schema documentation

License: Other

Python 11.99% Jupyter Notebook 88.01%

documentation-generator neurodata-without-borders neuroscience nwb python

hdmf-docutils's People

Contributors

Watchers

Forkers

nilegraddis jcfr starborn

hdmf-docutils's Issues

Replace of add_stylesheet function

Theadd_stylesheet function has been removed in Sphinx and has been replaced with add_css_file() (see also
sphinx-doc/sphinx#7747). The init_sphinx_extension_docs script should be updated accordingly.

Support figure width in %

hdmf_docutils.doctools.rst.RSTDocument.add_figure currently requires width and height to be specified in pixel. It would be useful to allow figure width to be specified in percent as well.

sg_prototype test fails on remote CI

The sg_prototype test fails on Azure and GitHub CI but succeeds locally. It might be an issue with environments. The failure seems to have to do with Sphinx modules not being loaded correctly.

@nicain @NileGraddis Is there still a need for sg_prototype.py?

It looks like it was developed during the April 2018 hackathon to allow rapid prototyping of Sphinx Gallery examples without rebuilding the full documentation, but it has not been used or updated anytime after that.
NeurodataWithoutBorders/pynwb#414
NeurodataWithoutBorders/pynwb#429
#1

Can it be removed? Since it is not in use, as far as I can tell, let's comment out the test.

Add rendering for DynamicTable

We should add a way to render DynamicTables as an actual table to better illustrate the created structure in the docs.

Sorting of types to sections for the core

TimeSeries should appear as a separate section in the schema docs but for some reason is now being missed as a section. This is likely either due to a bug in generate_format_docs.py or changes in the schema.

See: http://nwb-schema.readthedocs.io/en/latest/

Package requires sphinx==1.6.5

And this sphinx version is python2 only.

But if I look into https://github.com/NeurodataWithoutBorders/nwb-docutils/blob/02e94c831f91f03add180e4fbab6c396d85b3845/setup.py#L44 I would presume that it is python 3.5 or above.

Release new version on PyPI

Needed for bug fixes and use in ndx-template

make apidoc error

following https://pynwb.readthedocs.io/en/latest/extensions.html#documenting-extensions

tried in https://github.com/bendichter/nwbext_simulation_output/tree/follow_guidelines

$ make apidoc
PYTHONPATH=/Users/bendichter/dev/nwbext_simulation_output/docs/source: nwb_generate_format_docs
Output directory already exists: /Users/bendichter/dev/nwbext_simulation_output/docs/source/_format_auto_docs

SORTING TYPES INTO SECTIONS
---------------------------
Traceback (most recent call last):
  File "/Users/bendichter/anaconda3/bin/nwb_generate_format_docs", line 11, in <module>
    load_entry_point('nwb-docutils', 'console_scripts', 'nwb_generate_format_docs')()
  File "/Users/bendichter/dev/nwb-docutils/nwb_docutils/generate_format_docs.py", line 473, in main
    type_sections = DataTypeSection.sort_types_to_sections(default_namespace)
  File "/Users/bendichter/dev/nwb-docutils/nwb_docutils/doctools/renderrst.py", line 78, in sort_types_to_sections
    sections[spec_filename]['data_types'].append(nt)
KeyError: 'nwb.base.yaml'
make: *** [apidoc] Error 1

pip install error

Obtaining file:///Users/bendichter/dev/nwb-docutils
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/Users/bendichter/dev/nwb-docutils/setup.py", line 7, in <module>
        with open('README.doctools', 'r') as fp:
    FileNotFoundError: [Errno 2] No such file or directory: 'README.doctools'

Links listed twice

Links are listed in both table of "Datasets, Links, and Attributes" as well as the list of "Groups". They should be listed only in one.

how do I add images to the sphinx gallery?

I can add a link to an image with

# .. image:: multicompartment_schema_1.png

but this creates a dead link because the image isn't copied over to the _sources directory. Is there a way to do this automatically?

update sorting of types to sections

In the generate_format_docs script the sorting of types to sections for the core is implemented in the function sort_type_hierarchy_to_sections https://github.com/NeurodataWithoutBorders/nwb-docutils/blob/69de69741e6e46f7360d9c914fe7b989fe300339/nwb_docutils/generate_format_docs.py#L269

This function is brittle in that type names are hard-coded to sort the core types to sections. Because of this, this function tends to break whenever types are changed in the core. The strategy for sorting into sections should be updated to make it less brittle. We could either have a fully automatic strategy (e.g. by sorting types based on the YAML they are defined in) or we could have a configuration file alongside the YAML schema to control the sorting. In that way, at least everything is controlled from the schema rather than having to update to docutils code. This would also be useful to make the tool more generally useful for extensions.

Add support for UML class diagrams

It would be nice to be able to plot format specification as UML class diagrams. Some options for this are:

Mermaid:
- https://mermaid-js.github.io/mermaid/#/
- https://sphinxcontrib-mermaid-demo.readthedocs.io/
Graphviz:
- http://www.ffnn.nl/pages/articles/media/uml-diagrams-using-graphviz-dot.php
- https://www.sphinx-doc.org/en/master/usage/extensions/graphviz.html
Text UML tools:
- https://modeling-languages.com/text-uml-tools-complete-list/
D3 and other JS libs (web-only)

Update nwb-docutils to reflect new output requirements captured in nwb-extensions/ndx-template

the differences reported below were gathered by comparing the content of the docs folder generated using https://github.com/nwb-extensions/ndx-template and the one generated using nwb_init_sphinx_extension_doc:

docs/conf.py
- update list of extensions in conf.py
- htmlhelp_basename set to simulation_output_doc instead of simulation_outputdoc
- add missing intersphinx_mapping var

These differences are not directly related to nwb-docutils but are documented here for future reference. They corresponds to differences between the cookiecutter and example like this one

ReadMe.md -> README.md
license.md -> LICENSE.md
pynwb-src -> src/pynwb and src/spec

Graph rendering throws error for some data types

In HDMF, rendering fails specifically for Data, Container, and ElementIdentifiers.

Build and publish action fails

See https://github.com/hdmf-dev/hdmf-docutils/actions/runs/6659349295/job/18098227206

add clobber option to sg_prototype

When I use sg_prototype, I often want to make changes in the source, then immediately recompile and check the outputs. The check_tgt_dir function raises an error if the output directory already exists. As a minor QoL change, I would like to add an optional flag which clobbers existing outputs.

Explore using MERMAID for rending schema graphs

Mermaid graphs are a nice way to express UML and other diagrams in text. There appears to be also a sphinx extension for including MERMAID graphs in sphinx docs https://github.com/mgaitan/sphinxcontrib-mermaid

Generated conf.py is broken

The generated conf.py includes the following:

# -- Options for intersphinx extension ---------------------------------------

intersphinx_mapping = {
############################################################################
#  CUSTOM CONFIGURATIONS ADDED BY THE NWB TOOL FOR GENERATING FORMAT DOCS
###########################################################################

import sphinx_rtd_theme  # noqa: E402
import textwrap  # noqa: E402

# -- Options for intersphinx  ---------------------------------------------
intersphinx_mapping = {'core': ('https://nwb-schema.readthedocs.io/en/latest/', None)}

Note the syntax error at intersphinx_mapping = {. This is due to the (hacky) file cleaning in init_sphinx_extension_doc.py. It looks like much of this cleaning code can be removed and/or replaced.

This whole script should probably be cleaned up, but that is a separate, larger issue.

Heading underline to short in index.rst

In the init_sphinx_extension_doc.py script get_index_rst the name of the extension is added to heading but the underline for the heading is not adjusted to match the length of the title. This results in a heading underline that is too short and results in a subsequent warning from Sphynx when rendering the docs.

Migrate sphinx gallery prototyping module out of pynwb PR and into here

This issue is self-assigned to @nicain

Move sg_prototype.py into this repo
Add entrypoint to setup.py
Add functionality test

Use standard HDMF spec classes in render.py

render.HierarchyDescription should use the GroupSec, DatasetSpec etc. classes from HDMF rather than defining its on dicts for the hierarchy. Since this is only used in render it is not critical but should be addressed to make the functionality also more broadly useful and compliant with HDMF.

spec_catalog.get_subtypes missing

https://github.com/NeurodataWithoutBorders/nwb-docutils/blob/42bd1f4c6d390f004a1f0fd9b93182b384ad73bc/nwb_docutils/doctools/renderrst.py#L234

hdmf.spec.catalog.SpecCatalog has no method get_subtypes

Show hierarchy depth in spec tables much more clearly

See NeurodataWithoutBorders/nwb-schema#514

A quick fix is to change this line:

hdmf-docutils/hdmf_docutils/doctools/renderrst.py

Line 601 in 96d381e

depth_char=".",

to use depth_char="----" or depth_char="└───" or something

TypeError: draw_networkx_nodes() got an unexpected keyword argument 'font_family'

When generating HDMF Common documentation, I get the following error:

  File "c:\users\ryan\miniconda3\envs\dev\lib\site-packages\hdmf_docutils\doctools\render.py", line 682, in draw_g
raph                                                                                                              
    nx.draw_networkx_nodes(graph, pos,                                                                            
TypeError: draw_networkx_nodes() got an unexpected keyword argument 'font_family'

Networkx docs for draw_network_nodes() says that 'font_family' is not a keyword argument.
https://networkx.org/documentation/stable/reference/generated/networkx.drawing.nx_pylab.draw_networkx_nodes.html

New in networkx 2.5, released on 22 August 2020, is that extra keywords are no longer accepted in draw_network_nodes().

As far as I can tell from the networkx 2.4 source code, the extra keywords were never used, so removing 'font_family' from the call to draw_network_nodes() should result in no change in behavior.

Full output:

(dev) λ make html                                                                                                 
Running Sphinx v3.5.4                                                                                             
making output directory... done                                                                                   
Removed old sources at: C:\Users\Ryan\Documents\NWB\hdmf-common-schema\docs\source\_format_auto_docs              
Generating output directory: C:\Users\Ryan\Documents\NWB\hdmf-common-schema\docs\source\_format_auto_docs         
                                                                                                                  
SORTING TYPES INTO SECTIONS                                                                                       
---------------------------                                                                                       
Base data types                                                                                                   
    base data types                                                                                               
    ['Data', 'Container', 'SimpleMultiContainer']                                                                 
Table data types                                                                                                  
    data types for a column-based table                                                                           
    ['VectorData', 'VectorIndex', 'ElementIdentifiers', 'DynamicTableRegion', 'DynamicTable', 'AlignedDynamicTable
']                                                                                                                
Sparse data types                                                                                                 
    data types for different types of sparse matrices                                                             
    ['CSRMatrix']                                                                                                 
RENDERING TYPE HIERARCHY                                                                                          
------------------------                                                                                          
Container                                                                                                         
   CSRMatrix                                                                                                      
   DynamicTable                                                                                                   
      AlignedDynamicTable                                                                                         
   SimpleMultiContainer                                                                                           
Data                                                                                                              
   ElementIdentifiers                                                                                             
   VectorData                                                                                                     
      DynamicTableRegion                                                                                          
      VectorIndex                                                                                                 
RENDERING NAMESPACE SPECIFICATION                                                                                 
---------------------------------                                                                                 
    hdmf-common-- WRITE NAMESPACE DESCRIPTION DOC OK.                                                             
    hdmf-common-- WRITE NAMESPACE SOURCE DOC OK.                                                                  
RENDERING TYPE SPECIFICATIONS                                                                                     
------------------------------                                                                                    
BUILDING Data                                                                                                     
    Data-- SKIPPED RENDER HIERARCHY. TWO OR FEWER NODES.                                                          
    Data-- WRITE DESCRIPTION DOC OK.                                                                              
    Data-- WRITE SOURCE DOC OK.                                                                                   
BUILDING Container                                                                                                
    Container-- SKIPPED RENDER HIERARCHY. TWO OR FEWER NODES.                                                     
    Container-- WRITE DESCRIPTION DOC OK.                                                                         
    Container-- WRITE SOURCE DOC OK.                                                                              
BUILDING SimpleMultiContainer                                                                                     
SimpleMultiContainer-- RENDER HIERARCHY FAILED: Traceback (most recent call last):                                
  File "c:\users\ryan\miniconda3\envs\dev\lib\site-packages\hdmf_docutils\generate_format_docs.py", line 268, in r
ender_data_type_section                                                                                           
    fig = temp_graph.draw(show_plot=False,     # noqa F841                                                        
  File "c:\users\ryan\miniconda3\envs\dev\lib\site-packages\hdmf_docutils\doctools\render.py", line 343, in draw  
    return self.draw_graph(graph=self.graph, pos=self.pos, data=self.data, **kwargs)                              
  File "c:\users\ryan\miniconda3\envs\dev\lib\site-packages\hdmf_docutils\doctools\render.py", line 682, in draw_g
raph                                                                                                              
    nx.draw_networkx_nodes(graph, pos,                                                                            
TypeError: draw_networkx_nodes() got an unexpected keyword argument 'font_family'                                 
                                                                                                                  
    SimpleMultiContainer-- WRITE DESCRIPTION DOC OK.                                                              
    SimpleMultiContainer-- WRITE SOURCE DOC OK.                                                                   
BUILDING VectorData                                                                                               
    VectorData-- SKIPPED RENDER HIERARCHY. TWO OR FEWER NODES.                                                    
    VectorData-- WRITE DESCRIPTION DOC OK.                                                                        
    VectorData-- WRITE SOURCE DOC OK.                                                                             
BUILDING VectorIndex                                                                                              
    VectorIndex-- SKIPPED RENDER HIERARCHY. TWO OR FEWER NODES.                                                   
    VectorIndex-- WRITE DESCRIPTION DOC OK.                                                                       
    VectorIndex-- WRITE SOURCE DOC OK.                                                                            
BUILDING ElementIdentifiers                                                                                       
    ElementIdentifiers-- SKIPPED RENDER HIERARCHY. TWO OR FEWER NODES.                                            
    ElementIdentifiers-- WRITE DESCRIPTION DOC OK.                                                                
    ElementIdentifiers-- WRITE SOURCE DOC OK.                                                                     
BUILDING DynamicTableRegion                                                                                       
DynamicTableRegion-- RENDER HIERARCHY FAILED: Traceback (most recent call last):                                  
  File "c:\users\ryan\miniconda3\envs\dev\lib\site-packages\hdmf_docutils\generate_format_docs.py", line 268, in r
ender_data_type_section                                                                                           
    fig = temp_graph.draw(show_plot=False,     # noqa F841                                                        
  File "c:\users\ryan\miniconda3\envs\dev\lib\site-packages\hdmf_docutils\doctools\render.py", line 343, in draw  
    return self.draw_graph(graph=self.graph, pos=self.pos, data=self.data, **kwargs)                              
  File "c:\users\ryan\miniconda3\envs\dev\lib\site-packages\hdmf_docutils\doctools\render.py", line 682, in draw_g
raph                                                                                                              
    nx.draw_networkx_nodes(graph, pos,                                                                            
TypeError: draw_networkx_nodes() got an unexpected keyword argument 'font_family'                                 
                                                                                                                  
    DynamicTableRegion-- WRITE DESCRIPTION DOC OK.                                                                
    DynamicTableRegion-- WRITE SOURCE DOC OK.                                                                     
BUILDING DynamicTable                                                                                             
DynamicTable-- RENDER HIERARCHY FAILED: Traceback (most recent call last):                                        
  File "c:\users\ryan\miniconda3\envs\dev\lib\site-packages\hdmf_docutils\generate_format_docs.py", line 268, in r
ender_data_type_section                                                                                           
    fig = temp_graph.draw(show_plot=False,     # noqa F841                                                        
  File "c:\users\ryan\miniconda3\envs\dev\lib\site-packages\hdmf_docutils\doctools\render.py", line 343, in draw  
    return self.draw_graph(graph=self.graph, pos=self.pos, data=self.data, **kwargs)                              
  File "c:\users\ryan\miniconda3\envs\dev\lib\site-packages\hdmf_docutils\doctools\render.py", line 682, in draw_g
raph                                                                                                              
    nx.draw_networkx_nodes(graph, pos,                                                                            
TypeError: draw_networkx_nodes() got an unexpected keyword argument 'font_family'                                 
                                                                                                                  
    DynamicTable-- WRITE DESCRIPTION DOC OK.                                                                      
    DynamicTable-- WRITE SOURCE DOC OK.                                                                           
BUILDING AlignedDynamicTable                                                                                      
AlignedDynamicTable-- RENDER HIERARCHY FAILED: Traceback (most recent call last):                                 
  File "c:\users\ryan\miniconda3\envs\dev\lib\site-packages\hdmf_docutils\generate_format_docs.py", line 268, in r
ender_data_type_section                                                                                           
    fig = temp_graph.draw(show_plot=False,     # noqa F841                                                        
  File "c:\users\ryan\miniconda3\envs\dev\lib\site-packages\hdmf_docutils\doctools\render.py", line 343, in draw  
    return self.draw_graph(graph=self.graph, pos=self.pos, data=self.data, **kwargs)                              
  File "c:\users\ryan\miniconda3\envs\dev\lib\site-packages\hdmf_docutils\doctools\render.py", line 682, in draw_g
raph                                                                                                              
    nx.draw_networkx_nodes(graph, pos,                                                                            
TypeError: draw_networkx_nodes() got an unexpected keyword argument 'font_family'                                 
                                                                                                                  
    AlignedDynamicTable-- WRITE DESCRIPTION DOC OK.                                                               
    AlignedDynamicTable-- WRITE SOURCE DOC OK.                                                                    
BUILDING CSRMatrix                                                                                                
CSRMatrix-- RENDER HIERARCHY FAILED: Traceback (most recent call last):                                           
  File "c:\users\ryan\miniconda3\envs\dev\lib\site-packages\hdmf_docutils\generate_format_docs.py", line 268, in r
ender_data_type_section                                                                                           
    fig = temp_graph.draw(show_plot=False,     # noqa F841                                                        
  File "c:\users\ryan\miniconda3\envs\dev\lib\site-packages\hdmf_docutils\doctools\render.py", line 343, in draw  
    return self.draw_graph(graph=self.graph, pos=self.pos, data=self.data, **kwargs)                              
  File "c:\users\ryan\miniconda3\envs\dev\lib\site-packages\hdmf_docutils\doctools\render.py", line 682, in draw_g
raph                                                                                                              
    nx.draw_networkx_nodes(graph, pos,                                                                            
TypeError: draw_networkx_nodes() got an unexpected keyword argument 'font_family'                                 
                                                                                                                  
    CSRMatrix-- WRITE DESCRIPTION DOC OK.                                                                         
    CSRMatrix-- WRITE SOURCE DOC OK.                                                                              
Write C:\Users\Ryan\Documents\NWB\hdmf-common-schema\docs\source\_format_auto_docs\format_spec_doc.inc            
Write C:\Users\Ryan\Documents\NWB\hdmf-common-schema\docs\source\_format_auto_docs\format_spec_sources.inc        
Write C:\Users\Ryan\Documents\NWB\hdmf-common-schema\docs\source\_format_auto_docs\format_spec_main.inc           
building [mo]: targets for 0 po files that are out of date                                                        
building [html]: targets for 6 source files that are out of date                                                  
updating environment: [new config] 6 added, 0 changed, 0 removed                                                  
reading sources... [100%] software_process                                                                        
looking for now-outdated files... none found                                                                      
pickling environment... done                                                                                      
checking consistency... done                                                                                      
preparing documents... done                                                                                       
writing output... [100%] software_process                                                                         
generating indices... genindex done                                                                               
writing additional pages... search done                                                                           
copying images... [100%] figures/ragged-array.png                                                                 
copying static files... done                                                                                      
copying extra files... done                                                                                       
dumping search index in English (code: en)... done                                                                
dumping object inventory... done                                                                                  
build succeeded.                                                                                                  
                                                                                                                  
The HTML pages are in _build\html.                                                                                
                                                                                                                  
Build finished. The HTML pages are in _build/html.

Generating extension docs yields encoding error

If I pip install nwb-docutils and then run nwb_init_sphinx_extension_doc on Windows 10, Python 3.7, I get the following error:

Creating file C:\Users\Ryan\Documents\NWB\nwbext_simulation_output\docs\source\conf.py.
Creating file C:\Users\Ryan\Documents\NWB\nwbext_simulation_output\docs\source\index.rst.
Creating file C:\Users\Ryan\Documents\NWB\nwbext_simulation_output\docs\make.bat.

Finished: An initial directory structure has been created.

You should now populate your master file C:\Users\Ryan\Documents\NWB\nwbext_simulation_output\docs\source\index.rst and create other documentation
source files. Use the Makefile to build the docs, like so:
   make builder
where "builder" is one of the supported builders, e.g. html, latex or linkcheck.

Traceback (most recent call last):
  File "c:\users\ryan\anaconda3\envs\nwb-ext-test\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\ryan\anaconda3\envs\nwb-ext-test\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\Ryan\Anaconda3\envs\nwb-ext-test\Scripts\nwb_init_sphinx_extension_doc.exe\__main__.py", line 9, in <module>
  File "c:\users\ryan\anaconda3\envs\nwb-ext-test\lib\site-packages\nwb_docutils\init_sphinx_extension_doc.py", line 918, in main
    write_custom_conf(**clargs)
  File "c:\users\ryan\anaconda3\envs\nwb-ext-test\lib\site-packages\nwb_docutils\init_sphinx_extension_doc.py", line 784, in write_custom_conf
    outfile.write(custom_doc_autogen_settings)
  File "c:\users\ryan\anaconda3\envs\nwb-ext-test\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u2192' in position 3077: character maps to <undefined>

\u2192 is the rightwards arrow →
which is in one of the comments generated by conf_doc_autogen.py.

Solution: Encode strings in Unicode before writing, e.g.
outfile.write(custom_doc_autogen_settings) -> outfile.write(custom_doc_autogen_settings.encode('utf8'))

Rendered schema should use the schema in the YAML, not from the TypeMap

The docs for the schema, e.g. https://nwb-schema.readthedocs.io/en/stable/format.html, currently use the schema from the TypeMap, i.e., from HDMF or PyNWB, not from the YAML files in the schema directory. This is problematic when there is an inconsistency between the released schema and the schema used in PyNWB / HDMF.

Setup GitHub test pipelines

As part of PRs we should run the following test pipelines:

Build the NWB schema docs to ensure changes are not affecting the docs
Build hdmf-common-schema docs

nwb_init_sphinx_extension_doc: Improve settings of default values

Get metadata from namespace file, and better default values

If specified in namespace file, obtain author, version, project name, ... from it. Additionally reasonable default could be set for few other parameters.

Doing so would simplify the invocation of nwb_init_sphinx_extension_doc from:

nwb_init_sphinx_extension_doc  \
             --project simulation_output \
             --author "Ben Dichter" \
             --version "0.2.0" \
             --release alpha \
             --output docs2 \
             --spec_dir /tmp/nwbext_simulation_output/spec \
             --namespace_filename simulation_output.namespace.yaml \
             --default_namespace simulation_output

nwb_init_sphinx_extension_doc  \
             --output docs \
             --spec_dir /tmp/nwbext_simulation_output/spec \
             --namespace_filename simulation_output.namespace.yaml

where:

author is the first name specified in the namespace file
project and default_namespace are the namespace name
release is initialized to alpha by default

The namespace file could also be automatically found.

automatic updates of namespace file

On the other hand, if information like author, version are passed to nwb_init_sphinx_extension_doc, the namespace file could be updated if it doesn't already contain the info.

Last, if the info are already in the namespace file and the one passed in the command line contradict ... an exception could be raised.

NWBNamespace.get_source_files missing

https://github.com/NeurodataWithoutBorders/nwb-docutils/blob/1c1ac1761450b11311bbe29147fdb1525747bfcf/nwb_docutils/doctools/renderrst.py#L66

NWBNamespace has no method get_source_files and neither does hdmf.spec.SpecNamespace.

I think this bug can be resolved by instead using namespace['schema'], but then I find that get_source_description is also missing. This bug comes up when running make apidoc and make html and causes the following traceback:

(base) Bens-MacBook-Pro-2:docs bendichter$ make apidoc
PYTHONPATH=/Users/bendichter/dev/nwbext_simulation_output/docs/source: nwb_generate_format_docs
Output directory already exists: /Users/bendichter/dev/nwbext_simulation_output/docs/source/_format_auto_docs

SORTING TYPES INTO SECTIONS
---------------------------
Traceback (most recent call last):
  File "/Users/bendichter/anaconda3/bin/nwb_generate_format_docs", line 11, in <module>
    load_entry_point('nwb-docutils', 'console_scripts', 'nwb_generate_format_docs')()
  File "/Users/bendichter/dev/nwb-docutils/nwb_docutils/generate_format_docs.py", line 473, in main
    type_sections = DataTypeSection.sort_types_to_sections(default_namespace)
  File "/Users/bendichter/dev/nwb-docutils/nwb_docutils/doctools/renderrst.py", line 67, in sort_types_to_sections
    spec_descr = namespace.get_source_description(spec_filename)
AttributeError: 'NWBNamespace' object has no attribute 'get_source_description'
make: *** [apidoc] Error 1
(base) Bens-MacBook-Pro-2:docs bendichter$

Error generating docs for LinkSpec without name

A group spec in an extension has:

        links=[
            NWBLinkSpec(
                doc="The Subject object in the NWB file, if this Skeleton corresponds to the Subject.",
                target_type="Subject",
                quantity="?",
            ),
        ],

Running sphinx-build -b html -v -d build/doctrees source build/html results in:

Traceback (most recent call last):
  File "/Users/rly/mambaforge/envs/test/lib/python3.11/site-packages/sphinx/events.py", line 97, in emit
    results.append(listener.handler(self.app, *args))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rly/Documents/NWB/ndx-pose/docs/source/conf.py", line 73, in run_doc_autogen
    generate_docs()
  File "/Users/rly/mambaforge/envs/test/lib/python3.11/site-packages/hdmf_docutils/generate_format_docs.py", line 535, in main
    render_data_type_section(section=sec,
  File "/Users/rly/mambaforge/envs/test/lib/python3.11/site-packages/hdmf_docutils/generate_format_docs.py", line 334, in render_data_type_section
    rt_spec_data_table = SpecToRST.render_spec_table(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rly/mambaforge/envs/test/lib/python3.11/site-packages/hdmf_docutils/doctools/renderrst.py", line 697, in render_spec_table
    SpecToRST.render_spec_table(spec=link,
  File "/Users/rly/mambaforge/envs/test/lib/python3.11/site-packages/hdmf_docutils/doctools/renderrst.py", line 644, in render_spec_table
    elif spec.data_type_def is not None:
         ^^^^^^^^^^^^^^^^^^
AttributeError: 'LinkSpec' object has no attribute 'data_type_def'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/rly/mambaforge/envs/test/lib/python3.11/site-packages/sphinx/cmd/build.py", line 293, in build_main
    app = Sphinx(args.sourcedir, args.confdir, args.outputdir,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rly/mambaforge/envs/test/lib/python3.11/site-packages/sphinx/application.py", line 272, in __init__
    self._init_builder()
  File "/Users/rly/mambaforge/envs/test/lib/python3.11/site-packages/sphinx/application.py", line 343, in _init_builder
    self.events.emit('builder-inited')
  File "/Users/rly/mambaforge/envs/test/lib/python3.11/site-packages/sphinx/events.py", line 108, in emit
    raise ExtensionError(__("Handler %r for event %r threw an exception") %
sphinx.errors.ExtensionError: Handler <function run_doc_autogen at 0x1032058a0> for event 'builder-inited' threw an exception (exception: 'LinkSpec' object has no attribute 'data_type_def')

sg_prototype input parsing fails

env: Python 3.6, Ubuntu 17.10

Running:

nwb_gallery_prototype pynwb/docs/gallery/domain/ecephys.py -o <output_path>

fails with:

FileNotFoundError: [Errno 2] No such file or directory: 'p'

The culprit is optional input argument parsing which slices off the first element of the supplied input paths.

Decide whether to show included spec fields

make apidoc gives the following error:

Bens-MacBook-Pro:docs bendichter$ make apidoc
PYTHONPATH=/Users/bendichter/dev/nwb-extensions/simulation_output/docs/nwb_docutils:/Users/bendichter/dev/nwb-extensions/simulation_output/docs/source:/usr/local/hdf5: python /Users/bendichter/dev/nwb-extensions/simulation_output/docs/nwb_docutils/generate_format_docs.py
/Users/bendichter/dev/nwb-extensions/simulation_output/docs/nwb_docutils/generate_format_docs.py:60: UserWarning: DISABLING RENDERING OF SPEC GRAPHS DUE TO IMPORT ERROR
  warnings.warn('DISABLING RENDERING OF SPEC GRAPHS DUE TO IMPORT ERROR')
Generating output directory: /Users/bendichter/dev/nwb-extensions/simulation_output/docs/source/_format_auto_docs
Traceback (most recent call last):
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/spec/namespace.py", line 379, in load_namespaces
    inc_ns = self.get_namespace(s['namespace'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/utils.py", line 348, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/spec/namespace.py", line 240, in get_namespace
    raise KeyError("'%s' not a namespace" % name)
KeyError: "'core' not a namespace"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/bendichter/dev/nwb-extensions/simulation_output/docs/nwb_docutils/generate_format_docs.py", line 1356, in <module>
    main()
  File "/Users/bendichter/dev/nwb-extensions/simulation_output/docs/nwb_docutils/generate_format_docs.py", line 1263, in main
    resolve=spec_resolve_type_inc)
  File "/Users/bendichter/dev/nwb-extensions/simulation_output/docs/nwb_docutils/generate_format_docs.py", line 88, in load_nwb_namespace
    namespace.load_namespaces(namespace_file, resolve=resolve)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/utils.py", line 348, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/spec/namespace.py", line 381, in load_namespaces
    raise ValueError("Could not load namespace '%s'" % s['namespace'])
ValueError: Could not load namespace 'core'
make: *** [apidoc] Error 1

nwb_gallery_prototype error

I ran:

nwb_gallery_prototype docs/gallery/general/extensions.py

error:

Traceback (most recent call last):
  File "/Users/bendichter/anaconda3/bin/nwb_gallery_prototype", line 11, in <module>
    sys.exit(main())
  File "/Users/bendichter/anaconda3/lib/python3.6/site-packages/nwb_docutils/sg_prototype.py", line 124, in main
    return build(src_file, tgt_dir=tgt_dir, open_html=open_html)
  File "/Users/bendichter/anaconda3/lib/python3.6/site-packages/nwb_docutils/sg_prototype.py", line 54, in build
    shutil.copy(src_file, os.path.join(temp_dir, 'docs', 'gallery', os.path.basename(src_file)))
  File "/Users/bendichter/anaconda3/lib/python3.6/shutil.py", line 241, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/Users/bendichter/anaconda3/lib/python3.6/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: 'd'

YAML error: "dump()" has been removed

In ndx-hed, running the command:

sphinx-build -b linkcheck ./source ./test_build

results in:

Extension error:
Handler <function run_doc_autogen at 0x103403ba0> for event 'builder-inited' threw an exception (exception: 
"dump()" has been removed, use

  yaml = YAML(typ='unsafe', pure=True)
  yaml.dump(...)

instead of file "/Users/rly/mambaforge/envs/ndx-hed/lib/python3.11/site-packages/hdmf_docutils/doctools/rst.py", line 299

            return yaml.dump(clean_spec, default_flow_style=False)

)

The latest ruamel.yaml is 0.18.0 which removes the dump() function:

As announced, in 0.18.0, the old PyYAML functions have been deprecated. (scan, parse, compose, load, emit, serialize, dump and their variants (all, safe, round_trip_, etc)). If you only read this after your program has stopped working: I am sorry to hear that, but that also means you, or the person developing your program, has not tested with warnings on (which is the recommendation in PEP 565, and e.g. defaultin when using pytest).

https://yaml.readthedocs.io/en/latest/

Pip environment:

Package                       Version      Editable project location
----------------------------- ------------ --------------------------------
alabaster                     0.7.13
attrs                         23.1.0
Babel                         2.13.1
black                         23.9.1
cachetools                    5.3.2
certifi                       2023.7.22
cfgv                          3.4.0
chardet                       5.2.0
charset-normalizer            3.3.1
click                         8.1.7
codespell                     2.2.6
colorama                      0.4.6
contourpy                     1.1.1
coverage                      7.3.2
cycler                        0.12.1
distlib                       0.3.7
docutils                      0.18.1
filelock                      3.12.4
fonttools                     4.43.1
h5py                          3.10.0
hdmf                          3.10.0
hdmf-docutils                 0.4.5
identify                      2.5.30
idna                          3.4
imagesize                     1.4.1
iniconfig                     2.0.0
Jinja2                        3.1.2
jsonschema                    4.19.1
jsonschema-specifications     2023.7.1
kiwisolver                    1.4.5
MarkupSafe                    2.1.3
matplotlib                    3.8.0
mypy-extensions               1.0.0
ndx-hed                       0.1.0        /Users/rly/Documents/NWB/ndx-hed
networkx                      3.2
nodeenv                       1.8.0
numpy                         1.26.1
packaging                     23.2
pandas                        2.1.1
pathspec                      0.11.2
Pillow                        10.1.0
pip                           23.3.1
platformdirs                  3.11.0
pluggy                        1.3.0
pre-commit                    3.4.0
Pygments                      2.16.1
pynwb                         2.5.0
pyparsing                     3.1.1
pyproject-api                 1.6.1
pytest                        7.4.2
pytest-cov                    4.1.0
pytest-subtests               0.6.0
python-dateutil               2.8.2
pytz                          2023.3.post1
PyYAML                        6.0.1
referencing                   0.30.2
requests                      2.31.0
rpds-py                       0.10.6
ruamel.yaml                   0.18.2
ruamel.yaml.clib              0.2.8
ruff                          0.0.292
scipy                         1.11.3
setuptools                    68.2.2
six                           1.16.0
snowballstemmer               2.2.0
Sphinx                        7.2.6
sphinx-gallery                0.14.0
sphinx-rtd-theme              1.3.0
sphinxcontrib-applehelp       1.0.7
sphinxcontrib-devhelp         1.0.5
sphinxcontrib-htmlhelp        2.0.4
sphinxcontrib-jquery          4.1
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.6
sphinxcontrib-serializinghtml 1.1.9
tox                           4.11.3
tzdata                        2023.3
urllib3                       2.0.7
virtualenv                    20.24.6
wheel                         0.41.2

Build for core schema broken

#5 broke the generation of the format docs for the core schema because the namespace is already loaded from the spec that ships with PyNWB

Simplify sg_prototype script using webbrowser module

The following code could be simplified using https://docs.python.org/2.7/library/webbrowser.html

https://github.com/NeurodataWithoutBorders/nwb-docutils/blob/467dc07916e5af56392b15c1b23f9b362bb4a7d9/nwb_docutils/sg_prototype.py#L79-L94