Lyp could check, let's say every two weeks, for updates to lilypond. If the user is using a stable version, and a newer stable version is available, lyp could suggest an update. The same for an unstable version (which would be very useful, since a new unstable version is normally released every few weeks).

In order to be able to require arbitrary packages when compiling a file, add a --require/-r option to the lilypond command.

This will also necessitate changing the shorthand for the --raw option into -R.

When lyp is installed as a gem, gem binaries add an annoying additional latency of about 300ms. The only way to get rid of this latency is to rewrite the gem binary files. Lyp should detect that it's being used as a gem, and suggest the user run a builtin command to rewrite the binaries:

$ lyp -v
Lyp is being used as a ruby gem. To make the lyp and lilypond binaries faster, 
run the following command:

  lyp rewrite binaries

This message should be only displayed once.

After installing curl:

i5 pihentagy # wget -qO- https://git.io/getlyp | bash
Detecting latest lyp version...
Downloading https://github.com/noteflakes/lyp/releases/download/v1.3.1/lyp-1.3.1-linux-x86_64.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   605    0   605    0     0   1030      0 --:--:-- --:--:-- --:--:--  1030
100 6735k  100 6735k    0     0  1710k      0  0:00:03  0:00:03 --:--:-- 2109k
Extracting /tmp/lyp-release-installer/release.tar.gz
Could not find rugged-0.23.0 in any of the sources
Run `bundle install` to install missing gems.

Why not traveling ruby?

• The available compiled rugged gem is relatively outdated (0.22). To you git tags we need version 0.23 or later. There's no compiled rugged gem for Windows.
• The available version of ruby itself is relatively outdated, at 2.2.2.
• For any updates we are dependent on the traveling-ruby project, which has been practically neglected in the last year.
• The Rakefile is kinda ugly - lots of sh commands.

The alternative

If we wish to replace traveling ruby with something better, we need to have:

• platform-specific portable versions of Ruby, i.e. a Ruby that will work regardless its location.
• compiled rugged gem for each platform, bundled together with the corresponding Ruby.
• a simple way to build the portable Ruby for each platform: install Ruby from source, install the lyp dependencies, cleanup unneeded files, compress to tar archive.
• a simple way to package lyp itself, together with the pre-packaged version of lyp, into a compressed tar archive.

Plan of action

We will target three platforms: macOS, Windows, Linux X86-64 (the latter two will be built by spinning up small machines on AWS).

For each platform, a ruby installation will be maintained. Each installation will include a "raw version" and a "clean version". Each time the raw version is changed, it is copied to ruby.clean, and then unneeded files are removed. The clean version is what will be distributed.

Once we have the three archive files for the three platforms, we copy them into the lyp repository. To build a release, we prepare a release directory for each platform, unarchive the corresponding ruby, add the lyp files and some wrapper scripts, then create an archive and we're all done.

Tree structure

lyp
  bin - lyp binary files and wrappers
  lib - lyp source files
  ruby - a version of ruby

For each platform:

• Install ruby from source

• Create a bootstrapping ruby wrapper script - placed in the ruby bin directory

• Use wrapper to install lyp dependencies:

  ruby/bin/ruby_wrapper ruby/bin/gem install http-client rugged ...

• Cleanup and create archive:

  cp -R ruby ruby.original
  # remove all *.c *.a *.so *.o *.ri files
  # archive ruby
  mv ruby ruby.clean
  mv ruby.original ruby
  

Building a release

For each platform:

• Create a release-specific directory, i.e. dist/release/lyp-1.3.2-osx
• Open the ruby-2.4.0-osx archive to dist/release/lyp-1.3.2-osx/ruby
• Copy the lyp bin and lib directories to dist/release/lyp-1.3.2-osx
• Rename bin/lyp to bin/lyp.rb, bin/lilypond to bin/lilypond.rb
• Add wrapper scripts lyp, lilypond, lyp_env.sh (see here)
• Create archive of release

When a package is missing, show the location of the \require:

Missing package dependency "assert" in test.ly:2:1

      \require "assert"

Install any missing package dependencies by running: 'lyp resolve test.ly'

As this will necessitate additions to the dependency tree structure, combine work on this issue with #12.

Example:

\require "dummy"
\require "dummy>=0.1.0"

The reason is that for each leaf in the dependency tree, the dependencies hash maps package names to a dependency leaf which contains a clause (package ref) and matching versions. See resolver code.

The solution would be to keep an array of leaves for each package name, so for each instance of require a separate dependency leaf would be created.

Currently, there's no easy way to run any of the binary files bundled with lilypond, even though they are present for each version of lilypond installed by lyp. Example:

$ lyp exec musicxml2ly ...

# with version
$ lyp exec -F 2.19.31 musicxml2ly ...

# auto-install version of lilypond
$ lyp exec -nF 2.19.52 musicxml2ly ...

Add --music/-M (--music-relative/-m for relative mode) switch that allows entering music directly on the command line. Some examples:

# absolute mode
$ lilypond -M "c'4 d' e' f' g'1"
#=> compiles to a PDF with a random name

# -M/--music-relative - means \relative c'
$ lilypond -m "c4 d e f g1"

# produce a PNG snippet
$ lilypond -Sm "c4 d e f g1"

# set output file
$ lilypond -Sm "c4 d e f g1" -o my-snippet

Based on the work here: https://gist.github.com/ciconia/a786b461f7b6c2526ac0eeb42d3da8a0

Start a server:

# can specify a specific version of 
$ lyp serve -u2.19.37 -d # run as daemon

# can also specify port:
$ lyp serve -p 3213

# load packages on startup, in order to make compilation faster
$lyp serve -r assert -r house-style

Stop a server daemon (?):

# lyp stop-daemon -u2.19.37

Run client:

# check if any server is up (servers are registered in the lyp config file)
# if available (and has right version), use it
$ lyp compile <args> myfile.ly

Work left to do on server:

1. test it!
2. full support for command line options (as as full as possible - loglevel for example).

In order to facilitate updating of packages, without accumulating lots of old versions, an update command should be added:

$ lyp install [email protected]
$ lyp update assert
$ lyp list assert
   assert => (0.3.0)

Also, the install command should accept a -u or --update flag to delete any old versions of the installed package.

When installing via wget, it still requires curl

pihentagy@i5 ~ $ wget -qO- https://git.io/getlyp | bash
Detecting latest lyp version...
bash: line 4: curl: command not found
Could not get latest version from GitHub.
Downloading https://github.com/noteflakes/lyp/releases/download/v/lyp--linux-x86_64.tar.gz
--2017-01-01 20:18:00--  https://github.com/noteflakes/lyp/releases/download/v/lyp--linux-x86_64.tar.gz
Resolving github.com (github.com)... 192.30.253.112, 192.30.253.113
Connecting to github.com (github.com)|192.30.253.112|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2017-01-01 20:18:01 ERROR 404: Not Found.

An example here:

https://lists.gnu.org/archive/html/lilypond-devel/2011-06/msg00011.html

The package should provide a sane API for listening to global events and taking action upon stuff (like for example page breaks).

1. Rename lypack into lyp
2. Create a lyp-index repository. This repository will hold two files:
   • README.md - A description of the repository and how to contribute.
   • index.json - a JSON formatted list of packages, with each package having the following attributes: git URL, author, description.
3. Create a repository for the lyp website. The website will include a /packages page which will dynamically load the lyp index file, render it, and allow filtering by arbitrary text.

When working extensively with lyp or lilypond, an interactive mode might be beneficial. It cuts down on typing, and it also performs faster.

$ lyp
lyp> easy.ly
GNU LilyPond 2.19.52
Processing `/tmp/lyp/wrappers/easy.ly'
Parsing...
Interpreting music...
Preprocessing graphical objects...
Finding the ideal number of pages...
Fitting music on 1 page...
Drawing systems...
Layout output to `/var/folders/tz/jxcv5s0j091dfpxl3qbvpnf40000gn/T//lilypond-DjaSdE'...
Converting to `easy.pdf'...
Deleting `/var/folders/tz/jxcv5s0j091dfpxl3qbvpnf40000gn/T//lilypond-DjaSdE'...
Success: compilation successfully completed
lyp> pwd
/Users/sharon
lyp> watch scores/bwv35 # watches files in background
lyp> unwatch # stops watching

So for interactive mode to be useful a bunch of commands might be needed, but at least some of those might be useful also when invoking lyp non-interactively, for example the watch command.

Running lilypond scheme-sandbox fails because the lilypond wrapper treats the argument as a regular filename

When installing from gem, a lilypond stub binary will be created in the ruby bin directory, and will be on the path. When doing lyp list lilypond, lyp will list this binary as a system-installed version.

• scholarly
• lalily
• live-score
• auto-transpose
• page-layout (depends on edition-engraver)
• breaks (depends on oll-core)
• oll-core - keep only the nested options tree for a start
• curves

Done:

• lyp-roman-numerals
• lily-fonts - bravura and lilyjazz
• edition-engraver
• merge-rests-engraver

If I invoke a lyp-installed lilypond, lyp tries to process all parameters on its own, so e.g. "--version" or "--help" don’t work.

In order to beautify a bit the commands for including etc, introduce new names for the different include commands:

\incl "blah.ly"
\inclOnce "blah.ly"
\condIncl ##f "blah.ly"
\condInclOnce ##f "blah.ly"

The old commands pinclude etc should continue working. They should be deprecated (produce warnings) as of v1.5.0, and removed as of v2.0.0.

Instead of the usual:

$ lyp install paganini@dev:.

Provide a --dev switch so:

$ cd paganini
$ lyp install --dev

Add --auto-install-packages option (-A as shortcut), whereby before compiling the files, any packages not found will be automatically installed.

~$ sudo gem install lyp
[sudo] password for uliska: 
Fetching: highline-1.7.8.gem (100%)
Successfully installed highline-1.7.8
Fetching: ruby-progressbar-1.7.5.gem (100%)
Successfully installed ruby-progressbar-1.7.5
Fetching: thor-0.19.1.gem (100%)
Successfully installed thor-0.19.1
Fetching: mini_portile2-2.1.0.gem (100%)
Successfully installed mini_portile2-2.1.0
Fetching: nokogiri-1.6.8.rc2.gem (100%)
Building native extensions.  This could take a while...
ERROR:  Error installing lyp:
    ERROR: Failed to build gem native extension.

    /usr/bin/ruby2.1 extconf.rb
mkmf.rb can't find header files for ruby at /usr/lib/ruby/include/ruby.h

extconf failed, exit code 1

Gem files will remain installed in /var/lib/gems/2.1.0/gems/nokogiri-1.6.8.rc2 for inspection.
Results logged to /var/lib/gems/2.1.0/extensions/x86_64-linux/2.1.0/nokogiri-1.6.8.rc2/gem_make.out

It this a missing requirement?

Add bundle command which bundles all required dependencies and included files into a single target file, which could be compiled using plain lilypond.

Usage:

$ lyp bundle myfile.ly output.ly
# or:
$ lyp bundle myfile.ly
# which will write to myfile-bundle.ly

The bundle command should fail if font packages are used.

Original idea here:

https://github.com/joram-berger/lydiff

# compare two files
$ lyp diff file1.ly file2.ly

# compare two lilypond versions
$ lyp diff -a2.9.45 -b2.9.46 myfile.ly

The diff command should could provide, in addition to visual diff (using ImageMagic tools), a diff on metadata, such as number of pages, etc, or maybe just a diff of the lilypond stderr output.

I don't think it is a very good idea to have one tool manage both lilypond versions and lilypond code packages. I suggest following the Single Responsibility Principle and splitting lyp in two separate tools: a lilypond version manager allowing one to install several lilypond versions easily and switch between them, and a package manager, working simply with the lilypond currently in PATH. (Regardless of it being provided by the version manager or installed as a single system-wide lilypond instance.)

Add a --force-version or -F option for selecting the lilypond version according to the \version statement in the given file. For example:

$ lilypond -F myfile.ly

This can also be combined with the -n/--install option for automatically installing missing lilypond versions:

$ lilypond -nF myfile.ly

The command takes an input file, and flattens all included files into a target file. Usage:

$ lyp flatten myfile.ly output.ly

After successfully rendering my file with lilypond angelus.ly, I cannot flatten the .ly file.

lyp flatten angelus.ly 
No such file or directory @ rb_sysopen - /home/pihentagy/gdrive/kotta/Ckotta/darabok/AngelusPastoribus/deutsch.ly

It might be beneficial to offer a way to lock dependency versions a la ruby's bundler or nvm's shrinkwrap, and thus prevent problems that may arise from newer versions of dependencies (and also having to resolve the versions each time the user's file is compiled).

So suppose the user's file is called concerto.ly and it \requires some packages. The resolve command does the following:

• Parse the file for any referenced packages.
• Install any missing dependency or transitive dependency.
• Create a file called concerto.lock which is a YAML file containing:
  • A list of packages with specific versions.
  • A list of package specifiers and the actual version to use.

The next time the file is compiled, either using lilypond or lypack compile, lypack looks for the concerto.lock file. If the lock file exists, lypack still parses the user's files for dependencies, but uses the locked versions. If for some reason there's a discrepancy (the user's file or any included file has changed), the system signals an errors, and the user will need to run lypack resolve again.

This way, using a lock file is still optional (files can still be distributed around without too much hassle), but when a lock file is available, it ensures a repeatable build.

In order to make SVG output easier, add --svg option (like --pdf or --png), so:

$ lilypond --svg myfile.ly

# equivalent to:
$ lilypond -dbackend=svg myfile.ly

Running lilypond with -e/--evaluate option is fails with:

sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `/Users/sharon/.lyp/lilyponds/2.18.2/bin/lilypond -e#(display "hello") /dev/null'

(Besides, not sure /dev/null would work as file name)

Packages should have a way to test their functionality. User interface:

lyp install mypack
lyp test mypack

For package developers:

cd mypack
lyp test

lyp test should any files found using the wildcard test/*_test.ly.

There should also be some functionality for doing assertion type stuff. So, we create a small package dedicated to testing. This package includes some scheme functions for doing assertions.

In addition, package developers could use lyp test as part of integration test, for example on travis-ci. The assertions package also includes tests and a .travis.yml config file that could be used as a model.

This is related to #14 (support for non-repository packages). Lyp should allow installing a package under a different name. The format for installing from local path should be extended to allow arbitrary URLs:

# currently works: install from local files
$ lyp install foo@dev:bar

# add this: install from local files without version tag
$ lyp install foo:bar

# also add this: install from URL but rename the package
$ lyp install mypackage://foo.com/bar
# or:
$ lyp install mypackage:https://foo.com/bar

We should be able to do stuff like

\require assert

for includes, the file extension can be omitted.

\incl #'../music
\incl music

Search:

lyp snippet search TERM

Lookup:

lyp snippet lookup 366

Interactive:

lyp snippet -i

UI needs to be thought of a bit more...

In order to support more complex packages (like for example openlilylib), introduce support for sub-package modules:

\require "[email protected]/control_points"

In that case, instead of loading [email protected]/package.ly, lypack will load [email protected]/control_points.ly

A possible problem is a case where you have a package which package.ly file includes all its files. Requiring the package in the normal way, and later requiring a specific file of the package in either a user's file or another dependency, will lead to the same file being included twice.

Original issue here: lyp-packages/lilyjazz#1

Lyp will fail to install fonts on non-lyp installed versions of lilypond. This is because lilypond's share directory is at an unexpected location. This could also be because there is an existing lilypond shim bash script, which makes detecting the actual path to the lilypond binary more complicated.

There is, however, an easy way to detect the data dir for any installed version of lilypond:

#(display (ly:get-option 'datadir))

or:

$ lilypond -e"(display (ly:get-option 'datadir))" /dev/null  2>/dev/null

# for example:
=> /usr/share/lilypond/current

The fonts directory should then be under the given path, in the above case /usr/share/lilypond/current/fonts.

In addition, lilypond should be able to patch system-installed lilyponds. This can be done by doing a quick check when installing fonts, on the font.scm file. If it is not identical to the patched file, then it needs to be replaced.

Hi,
I'm trying to use lyp to install lilyjazz under Lubuntu 16.04 but it does not work:
After installing lyp via ruby gem (neither the curl nor the wget installation worked for me, they both aborted with the error: bash: line 52: syntax error: unexpected end of file.| gem required cmake and pkg-config before I could install lyp) I am not able to install any lyp-packages. The installations always abort with the error message:
Could not clone repository (please check that the package URL is correct.)

When I copy the URL into my webbrowser the right git opens so I guess that is not the problem.

Opposed to the General spec there is no ~/.lyp/bin folder (.lyp contains the folders ext, lilyponds, packages and a file settings.yml) on my machine also there is no file ~/.bash_profile so I guess that might cause trouble too.

I am not a software developer so I don't know what else you need to help me so feel free to ask.

Thanks, I like the idea of a Lilypond package manager very much

Sven

Sometimes a repository is overkill. Sometimes people don't want to use git. Lyp should be able to install packages from plain urls, or even from stdin.

# local file (already supported)
$ lyp install mypack@ly-code/mypack.ly

# from stdin
$ cat ly-code/mypack.ly | lyp install mypack -

# plain http(s)
$ lyp install http://myserver.com/mypack.ly

# archive on https
$ lyp install http://myserver.com/mypack.tar.gz

# gists
$ lyp install gist.github.com/ciconia/22e96b33463a8ae4b7df
$ lyp install gist:ciconia/22e96b33463a8ae4b7df

Currently lyp fails if trying to include a .ly file bundled with lilypond, or adding to the include search path using the --include command line switch. Lyp should take into account include search paths used by Lilypond. See also #43.

When --snippet is specified, the lilypond preprocessor should inject the following block before the user's file:

\paper {
  indent = 0\mm
  oddHeaderMarkup = ""
  evenHeaderMarkup = ""
  oddFooterMarkup = ""
  evenFooterMarkup = ""
}

The benchmark command takes arbitrary lilypond arguments, runs them on all installed versions of lilypond, then displays running times for each version:

$ lyp benchmark myfile.ly
#=>
  2.18.2 : 50.10s
  2.19.16: 35.25s 

The watch command is used for watching files or directories and automatically running lilypond whenever those files change. The watch should accept one or more directories or files to watch.

The watch command should also accept a --target option to signify the file to compile whenever any file changes. This would be useful when the user wishes to watch include files but to only compile a specific file.

$ lyp watch bwv78
17:38:13 Watching bwv78...
17:40:22 bwv78/oboe.ly changed, recompiling
...
17:45:51 bwv78/violino1.ly changed, recompiling
...
^C
# watch multiple directories, recompile specific file
$ lyp watch bwv78 house-style stdlib -t bwv78/score.ly
...

After successfully installing lilypond:

curl -sSL https://git.io/getlyp | bash
lyp install lilypond

I got the following error:

$ lyp compile angelus.ly 
Lyp 1.3.1
Invalid include file specified in /home/pihentagy/gdrive/kotta/Ckotta/darabok/AngelusPastoribus/angelus-V1.ly:2: 

  \include "deutsch.ly"

/home/egerkon/.lyp/lib/app/lib/lyp/resolver.rb:561:in `error'
/home/egerkon/.lyp/lib/app/lib/lyp/resolver.rb:556:in `error'
/home/egerkon/.lyp/lib/app/lib/lyp/resolver.rb:222:in `process_include_command'
/home/egerkon/.lyp/lib/app/lib/lyp/resolver.rb:187:in `block (2 levels) in process_lilypond_file'
/home/egerkon/.lyp/lib/app/lib/lyp/resolver.rb:184:in `scan'
/home/egerkon/.lyp/lib/app/lib/lyp/resolver.rb:184:in `block in process_lilypond_file'
/home/egerkon/.lyp/lib/app/lib/lyp/resolver.rb:182:in `each_line'
/home/egerkon/.lyp/lib/app/lib/lyp/resolver.rb:182:in `process_lilypond_file'
/home/egerkon/.lyp/lib/app/lib/lyp/resolver.rb:154:in `compile_dependency_tree'
/home/egerkon/.lyp/lib/app/lib/lyp/resolver.rb:80:in `resolve_package_dependencies'
/home/egerkon/.lyp/lib/app/lib/lyp/wrapper.rb:7:in `wrap'
/home/egerkon/.lyp/lib/app/lib/lyp/lilypond.rb:102:in `compile'
/home/egerkon/.lyp/lib/app/bin/lilypond:59:in `<main>'

Similar to the recently added --cropped switch, add a --snippet switch which is equivalent to --cropped --png -dresolution=600.

Add --all option to lyp test command to run package/local tests on all installed versions of lilypond.

Also, the lyp test command should normally not display the lilypond output (both STDOUT and STDERR) unless the --verbose option is given. It should however count assertions (displayed in test files by calling #(assert:summary)) and show a summary of the tests:

• Any failures (file/package/lilypond version)
• Number of assertions per package, total assertions
• Total running time

In order to allow version-specific "polyfills", add a \condRequire command and version predicates:

\condRequire #(lyp:ly-version< "2.19.13") "lyric-extenders"

Some version predicates:

#(lyp:ly-version< ver)
#(lyp:ly-version<= ver)
#(lyp:ly-version>= ver)

The website should have at least two pages: a main page, and a package listing page.

main page

One option is to take the README and convert it into a web page, either using the automatic github tools, or by integrating it into a Jekyll website, as described here: http://stackoverflow.com/questions/15214762/how-can-i-sync-documentation-with-github-pages

Another possibility is to create a two-column layout, where there's a sidebar on the left with a table of contents, and then the actual scrolling content on the right - largely based on the README, which is already quite task-oriented. (an example of this kind of layout is the influxdb docs, even though they're split into multiple pages).

For the package listing page, a simple list of packages, with the name linked to the package homepage, followed by the package description. A preliminary implementation here: lyp-packages/index#1

A navbar at the top of the page should have the following links:

• docs (main page)
• packages
• github icon (github page)

For example ls for list etc. Can take inspiration from nvm and other tools