dbbs-lab / bsb-core Goto Github PK

Describe the work done

List which issues this resolves:

(e.g. "closes #1, #2, #3")

Tasks

• Added tests
• Updated documentation

Currently the morphology_repository attribute is problematic: It looks for the specified file relative to the cwd and not to where the configuration was found or some other sane behavior, so when you compile from another folder it won't find the morphology repository anymore

Right now, hours of radio-silence pass when connecting GrCs to PCs. It would be useful to track the progress %.

For some connections, divergence values are not used for connectivity (e.g. for IO to PC connections), but it should be checked and a warning should be thrown.

• For IO to PC values see Helveg/cerebellum-scaffold#104.

• For PC-DCN: "Because 30–50 Purkinje cells converge onto each CbN cell (Person & Raman, 2012a), the number of inhibitory inputs was held constant at 40 in all experiments." [https://physoc.onlinelibrary.wiley.com/doi/pdf/10.1113/JP274321]

In VoxelIntersection it's contacts and in TouchDetection it's synapses

When we complete Helveg/cerebellum-scaffold#369 we rename the old nest to nest2 and the class to Nest2Adapter

When we #369 we rename the old nest to nest2 and the class to Nest2Adapter

Transform the Configuration and its current state back into a configuration tree (ignore breakpoints and multifiles for now)

Currently the global CLI arguments are position specific and there's no unified way access global scaffold settings. It would be better that there's an options module where the values of singleton settings can be read from and stored in from 3 sources:

• environment variables
• cli arguments
• set by script bsb.options.verbosity = 2?

class Options(types.ModuleType):
  verbosity = IntegerOption(cli=("verbosity", "v"), env="BSB_VERBOSITY", default=1)
  log = StringOption(cli=("log", "l"), env="BSB_LOG")

sys.modules[__name__] = Options()

Previous (less) or future (more)?

Time to split foolks.

File "/home/alberto/workspace/cerebellum-scaffold/scaffold/postprocessing.py", line 263, in after_connectivity
    compartment_matrix = self.scaffold.connection_compartments[tag]
KeyError: 'golgi_to_glomerulus'

Even if using "detailed" for golgi_to_glomerulus:

"golgi_to_glomerulus": {
      "class": "scaffold.connectivity.ConnectomeGolgiGlomerulus",
      "from_cell_types": [{"type": "golgi_cell", "compartments": ["axon"]}],
      "to_cell_types": [{"type": "glomerulus", "compartments": ["soma"]}],
      "divergence": 40,
      "detailed": true
    },

@AliceGem is this a correct explanation of num_neurons? (git blame says you added it 😋) Could you also look into #471 for this PR?

We should check that NEST-specific variables are properly set when running NEST simulations with the scaffold (e.g. local_num_threads or RNG variables).

Tests are being skipped to allow #494 to merge. We should:

A) Check if we can do weekly builds and tests of full scale networks that we can then cache on Travis, or upload as build artifacts to possibly Amazon AWS or our own cluster?

B) Re-enable push and branch testing by using the old mouse_cerebellum.json as basis for these smaller per-commit tests

File "/home/alberto/workspace/cerebellum-scaffold/scaffold/postprocessing.py", line 263, in after_connectivity
    compartment_matrix = self.scaffold.connection_compartments[tag]
KeyError: 'golgi_to_glomerulus'

Even if using "detailed" for golgi_to_glomerulus:

"golgi_to_glomerulus": {
      "class": "scaffold.connectivity.ConnectomeGolgiGlomerulus",
      "from_cell_types": [{"type": "golgi_cell", "compartments": ["axon"]}],
      "to_cell_types": [{"type": "glomerulus", "compartments": ["soma"]}],
      "divergence": 40,
      "detailed": true
    },

as_cache should return an empty MorphologyCache based on the MR. MorphologyCaches should produce singleton morphologies that are immutable by default.

Usecase:

The user wants to compile an hdf5 exploiting the morphologies, but it does not want to simulate it in NEURON and does not have NEURON installed.

Actual behaviour:

The user has to remove from the json the"NEURON" simulation part.

Desired behaviour:

The compilation should in principle work even if NEURON is not installed.

Tests are being skipped to allow #494 to merge. We should:

A) Check if we can do weekly builds and tests of full scale networks that we can then cache on Travis, or upload as build artifacts to possibly Amazon AWS or our own cluster?

B) Re-enable push and branch testing by using the old mouse_cerebellum.json as basis for these smaller per-commit tests

Describe the work done

• The setup now produces a wheel named bsb
• The console_script entry point is renamed to bsb (so the command is bsb compile instead of scaffold compile)
• The package source directory has been renamed to bsb
• Imports should import the bsb module instead of the scaffold module.
• Removed the obfuscation module. Since the package will go public we don't need to encrypt our public releases anymore.

Migration

Users (which we don't have) need to uninstall the dbbs-scaffold package and pip install bsb instead.

Developers need to run pip install -e . from the git repo root to register the new bsb command and change all from scaffold/ import scaffold statements to from bsb/import bsb

List which issues this resolves:

Closes #482

Tasks

• Added tests
• Updated documentation

Previous (less) or future (more)?

Usecase:

The user wants to compile an hdf5 exploiting the morphologies, but it does not want to simulate it in NEURON and does not have NEURON installed.

Actual behaviour:

The user has to remove from the json the"NEURON" simulation part.

Desired behaviour:

The compilation should in principle work even if NEURON is not installed.

In order to print output during an MPI process stdout needs to operate in unbuffered mode. The reporting module can overwrite stdout if MPI is present.

Since point neuron models ignore any compartments given, there are many duplicate entries between the same neuron pairs in connectivity sets created with detailed connectivity strategies. To optimize, a reduced dataset with unique pairs and a parallel dataset with the amount of times they occur could be provided.

Impact on ConnectivitySet

The ConnectivitySets get_intersection method should be adapted to incorporate a get_intersection_iterator.

Impact on NEST adapter

Multi-contact connections are created only once. Unitary weights are tuned to get the right population firing rates, and then multiplied by the synaptic contact multiplicity to get the single connection weight (population weight will be an array)

Impact on NEURON adapter

None, the get_intersections function should produce the same result. To optimize, we should switch to the get_intersection_iterator to reduce memory footprint.

Not optimized number of possible searches..
probably n^(2)*log(n) searches , useless..

Fixes dbbs-lab/models-old#25

And wrap them in more information such as cell model, current context, ...

Fixes dbbs-lab/models-old#25

Time to split foolks.

Right now, hours of radio-silence pass when connecting GrCs to PCs. It would be useful to track the progress %.

as_cache should return an empty MorphologyCache based on the MR. MorphologyCaches should produce singleton morphologies that are immutable by default.

The report module requires a slight refactoring so that report listeners can be added. Reporting to screen should then be added as a default report listener. This should improve the design for a variety of future logging functions (better than the messy report file which is mostly obsolete when #495 is implemented)

In order to print output during an MPI process stdout needs to operate in unbuffered mode. The reporting module can overwrite stdout if MPI is present.

They're a mess, should become similar to the PlacementSet

@AliceGem is this a correct explanation of num_neurons? (git blame says you added it 😋) Could you also look into #471 for this PR?

They're a mess, should become similar to the PlacementSet

Since point neuron models ignore any compartments given, there are many duplicate entries between the same neuron pairs in connectivity sets created with detailed connectivity strategies. To optimize, a reduced dataset with unique pairs and a parallel dataset with the amount of times they occur could be provided.

Impact on ConnectivitySet

The ConnectivitySets get_intersection method should be adapted to incorporate a get_intersection_iterator.

Impact on NEST adapter

Multi-contact connections are created only once. Unitary weights are tuned to get the right population firing rates, and then multiplied by the synaptic contact multiplicity to get the single connection weight (population weight will be an array)

Impact on NEURON adapter

None, the get_intersections function should produce the same result. To optimize, we should switch to the get_intersection_iterator to reduce memory footprint.

Describe the work done

More plotting functions now return their produced figs, and fixed some other small bugs. PSTHs can now be stacked.

Not optimized number of possible searches..
probably n^(2)*log(n) searches , useless..

Currently the morphology_repository attribute is problematic: It looks for the specified file relative to the cwd and not to where the configuration was found or some other sane behavior, so when you compile from another folder it won't find the morphology repository anymore

SC and BC (not labelled) should be connected to PC (labelled) implicitly respecting the microzone division.
In the current neuron.json scaffold the segregation is ok, but we should check that this is consistent with morphologies orientations:

We should check that NEST-specific variables are properly set when running NEST simulations with the scaffold (e.g. local_num_threads or RNG variables).

Transform the Configuration and its current state back into a configuration tree (ignore breakpoints and multifiles for now)

For some connections, divergence values are not used for connectivity (e.g. for IO to PC connections), but it should be checked and a warning should be thrown.

• For IO to PC values see #104.

• For PC-DCN: "Because 30–50 Purkinje cells converge onto each CbN cell (Person & Raman, 2012a), the number of inhibitory inputs was held constant at 40 in all experiments." [https://physoc.onlinelibrary.wiley.com/doi/pdf/10.1113/JP274321]