kalekundert / exmemo Goto Github PK
View Code? Open in Web Editor NEWLab notebook for people who like the command line.
License: GNU General Public License v3.0
Lab notebook for people who like the command line.
License: GNU General Public License v3.0
Workspaces and experiment directories are defined by meeting certain requirements: having a name that matches a certain pattern, having certain subdirectories, containing certain files, etc.
When these requirements are not met, the error messages are not helpful. In the case of workspaces, it just says "this is not a workspace". In the case of experiments, it just doesn't find the experiment in question (and may find none at all), which leads to some confusing errors.
I was able to trigger a bunch of this behavior by cloning a workspace on a remote server. The workspace wasn't detected initially because none of the empty directories were cloned, so I had to make those directories manually. Maybe the root of the workspace should be defined only by the presence of the .exmemorc
file. Exmemo does probably assume that all those directories exists, but I could probably just check for them as needed, and either fail gracefully or create them on-the-fly.
The experiment I had made also wasn't detected because I'd committed some of the files in the experiment, but not the *.rst
file. Experiments are detected by (i) being a directory, (ii) prefixed with a date, (iii) containing an *.rst
file with a name matching the directory. So without the *.rst
file, the experiment wasn't detected. In this case, 'm not so sure what the best course of action is:
Maybe I could improve the error message somehow, i.e. detect if a directory with the right name would match, but is just missing its *.rst
file. That would at least tell the user what to do.
I could add another kind of dot-file to identify experiments (i.e. either one in each experiment, or one in the experiment root that serves as an index), but it would be just as easy to not commit that file too. It would also require the user to manage a magic file that they don't really understand, and it would clutter up their directories. I don't think this is a good idea.
I could relax the requirements and consider any date-prefixed directory to be an experiment. One problem I can see with this is that exmemo would like to know the name of the *.rst
file for each experiment, so it can open it up when you ask for it. Right now this always works, because the file and the directory have to have the same name (excluding the date prefix). If I get rid of this requirement, I'd have to guess at the name of the *.rst
file, although in most cases I could probably guess right.
Hey, I really dig this package! It's extremely well written.
I'm curious if this has been tested on windows?
When I try the exmemo note new "Large step with half twist"
, it just hangs. Also, when I create a new project, the documents
, data
, and protocols
folders aren't created. I'm not sure if this is a bug or not.
Similar to .. datatable::
, but for sequences. There's gotta be a JS library for this kind of thing.
The current method of translating keywords to paths (based on substring matching) has some annoying corner cases:
Names with several variants may be difficult to specify unambiguously. For example, I have protocols named "pcr" and "pcr_cloning". Typing just "pcr" is always ambiguous, even though I know that I mean just "pcr". This situation comes up farily often.
Sometimes I find myself running a command with an ambiguous keyword a bunch of times in a row, e.g. if I'm trying to get the arguments how I want them. This is annoying because unless I edit the command to make it unambiguous (which is annoying itself), I'm asked which path I mean each time. Some ways to address this:
I only think a certain file has a certain substring when it doesn't. See #3. It would be nice to provide custom keywords for certain paths, basically aliases.
I often create ad-hoc bibliographies using footnotes and :doi:
roles. This isn't so bad, but it does have some annoyances:
A custom bibliography directive could address both of these issues. The challenge would be integrating with the footnote system, so the same [AuthorYear]_
syntax could continue to be used in the document.
Alternatively, I could maybe play with the CSS to render the footnotes more nicely.
I often don't remember the exact name I used for an experiment, but I remember keywords like what kind of experiment it was, what mutant I was working with, etc. It might be nice if I could associate keywords with experiments using some sort of config file.
I've got to think more about how to organize all the Sphinx extensions associated with exmemo. I think a lot of them could be useful for other projects, and shouldn't require installing all of exmemo. Plus, Sphinx extensions normally live in sphinxcontrib
directory, which these don't. I also need documentation.
I had an idea for how to symlink data files into notebook directories, in such a way that (i) rearranging the notebook directories won't break the symlinks and (ii) no absolute paths are used (important for collaboration). The idea is to have a "root" symlink in the .expt
metadata directory associated with each experiment. All other symlinks will go through that one (e.g. .expt/root/data/gels/20210916_xyz.svg
). If the experiment is moved, the root symlink can be automatically updated.
A lot of exmemo commands perform recursive globs to find files matching a keyword. These globs take longer and longer----eventually becoming a bottleneck---as the project grows and more files need to be searched. Part of the problem is that querying the filesystem is fairly expensive. I think I could speed these searches up significantly if I maintained a cache of the names of the files in the project.
However, a cache wouldn't do me any good if I need to update it every time I run a command anyways. (I definitely can't assume that new files won't be created between commands, and it would be annoying to the user if it were possible for the cache to be stale.) So I think I'd have to run a daemon to make this work. Having to manage a daemon would also be annoying, but I think I could make stay behind the scenes pretty well:
The RST role for refering to experiments is way too verbose, but it's not obvious how I could make it simpler.
Also, it too hard to create new experiments, and there's no support for nested experiments.
Sometimes, when I find mistakes in my protocols, I realize that I've been making the same mistake in all the experiments I've done up to that point. I wonder if I can think of a good way to annotate that...
I think this more general name makes better sense.
I want a more flexible API for looking up experiments:
Despite wanting all this flexibility, I also need to think about how to provide a consistent experience to end-users.
I'm actually not sure if this directory is even part of exmemo, but in any case 'reagents' better reflects what this directory is actually for.
I often set out to do an experiment to answer question X, but some complications come up, and I have to do some more related experiments to get get to the bottom of things. Considering that I now want to organize experiments heirarchically, a natural way to handle this case would be to:
It'd be nice to have a command to do manage doing this automatically. Some names:
breakout
drill
expand
Curently, the lab notebook is hosted as a static site; it's basically just a graphical way to view notebook entries. However, it would be nice if the web interface was more interactive and more customized to the lab notebook application. Some specific features that would be nice:
The underlying directory structure would remain the single source of truth, and "edit" actions by the web server would directly operate on the file system. Some more thoughts about the implementation:
_build
directory.
*.rst
into nodes... toctree::
, :expt:
) might be hard./
: Dashboard-style views of current/planned/complete experiments./expt/<id>
: Show the requested experiment.
/tag/<name>
: Show all the experiments associated with the given tag.
/search/<query>
: Show a list of all the experiments matching the given query (with some context)..meta
directory to each experiment to hold information about ids and tags. Right now the only metadata is the .id
file in each experiment. Alternatively, I could keep the metadata in a single structured file (e.g. nested text) that contains ids and tags. A directory is probably more future-proof, though.
Probably the first thing to play with is the Flask/Sphinx linkage. If I can get that to work, the rest should be pretty straight-forward and incremental.
If multiple people are collaborating on a project, they could easily end up creating experiments with duplicate keys. I'll need a way to avoid this. Off the top of my head, one option would be to have each user specify how many users there are, and which user number they are. New experiment ids would be multiplied by the total number of users, then summed with the specific user number. That would work, but requires a lot of manual intervention.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.