Improve the installation layout about bsc HOT 8 OPEN

I would like to make progress on this issue, since it is holding up the packaging of BSC -- see issue #137 and (duplicate) issue #288.

I also think that it's related to issue #103 (adhering to the GNU standards for DESTDIR/PREFIX) and to PR #289 (support for specifying the LIBDIR) -- and we should fix those as well.

I would also add two requirements for whatever new layout we choose: (1) allow multiple versions to be installed at the same time (which can be done by including the version number in file/directory names), and (2) avoid using "bluespec" as the name for anything (both because there will be more tools and more repos around the language, and also to avoid association with the company Bluespec Inc).

I think that we will make progress by putting up concrete proposals (as @thoughtpolice has) and converging on a layout, however I would like to understand if there are any unsaid requirements that motivated that proposal. For example, if the same BSC version is installed for multiple OS/archs, are we trying to share the non-OS/arch-dependent files, to avoid duplication? Because here's a very basic proposal:

$PREFIX/bin
$PREFIX/$LIBDIR/bsc-$VERSION/

and that's it. The lib directory that BSC currently installs into was never intended to be the OS's lib directory, it was just BSC's own directory. And the files there can arguably be said to be for BSC's internal use, more than directly accessed by the user, so I have no qualms about hiding it all under a single directory. But I'm also amenable to moving files out of there and into other locations.

(Note that, if everything that the makefiles currently install into PREFIX was installed as-is into $PREFIX/$LIBDIR/bsc-$VERSION/ and symlinks for bsc/bluetcl are added to $PREFIX/bin/ pointing into $PREFIX/$LIBDIR/bsc-$VERSION/bin/, then everything would continue to work without any changes needed to the BSC codebase.)

If we want to put the non-arch-specific parts under $PREFIX/share/bsc-$VERSION/ (instead of lib), that's also possible.

I guess we could move the Tcl libraries out of a bsc directory and into wherever Tcl libraries are normally installed (as @thoughtpolice seems to have proposed). And we could also move the Bluesim headers into $PREFIX/include/bsc-$VERSION/ or similar -- although they're less likely to be used directly by users, and they have associated object files that would need to be put somewhere too if this changed.

As for other minor cleanups: I have no objection to giving hierarchy to the Verilog directories: a Verilog directory with generic, Vivado, and Quartus undernearth (although there might even need to be additional hierarchy, to support different variants for different Vivado versions, etc). I have no objection to changing Bluesim to bluesim. I have no objection to rethinking the tcllib hierarchy -- it currently has only one directory, to allow for future expansion, but that would probably be done as a separate tool in a separate repo (see the new bdw repo), so maybe the additional hierarchy is unneeded.

So, I could imagine, on the other end of the spectrum (requiring lots of codebase changes to bsc/bluetcl to find files), having this layout:

$PREFIX/bin                           # wrapper scripts
$PREFIX/bin/bsc-$VERSION              # core executables of the wrapper scripts

$PREFIX/share/bsc-$VERSION/Libraries               # bo files
$PREFIX/share/bsc-$VERSION/Verilog                 # Verilog files
$PREFIX/share/bsc-$VERSION/Verilog/generic         # (to avoid cluttering the parent directory?)
$PREFIX/share/bsc-$VERSION/Verilog/Vivado
$PREFIX/share/bsc-$VERSION/Verilog/Quartus
$PREFIX/share/bsc-$VERSION/exec                    # scripts used by bsc (unless these should go under `bin` somewhere?)
$PREFIX/share/bsc-$VERSION/tcltk/tcl${TCLVERSION}  # tclIndex and tcl files (unless there's a central place for tcl libraries?)

$PREFIX/share/doc/bsc-$VERSION/                    # copyright, manuals etc

$LIBDIR/bsc-$VERSION/SAT              # SAT libraries (loaded by bsc)
$LIBDIR/bsc-$VERSION/VPI              # VPI libraries (linked with user and generated code, for Verilog sim of designs with imported C)
$LIBDIR/bsc-$VERSION/bluesim          # Bluesim libraries (linked with generated code, for Bluesim sim)

$PREFIX/include/bsc-$VERSION/bluesim  # bluesim headers

from bsc.

We should also think about how where BSV libraries would be installed.

I recommend that BSV source libraries and .bo files be installed in a similar location.

from bsc.

I suggest something like:

$PREFIX/bin                                # bsc, bluesim, etc
$PREFIX/include                            # bluesim headers
$PREFIX/lib                                # bluesim.so, bluedpi.so, etc
$PREFIX/lib/bluespec/base                  # .bo files for prelude
$PREFIX/lib/bluespec/yices                 # sat objs, etc
$PREFIX/lib/bluespec/exec                  # sim scripts
$PREFIX/lib/tcl${VERSION}/bluespec         # bluespec tcl modules
$PREFIX/share/bluespec/verilog             # verilog synthesis files
$PREFIX/share/bluespec/verilog/vivado/     # synthesis files for Vivado
$PREFIX/share/bluespec/verilog/quartus/    # synthesis files for Quartus

It's definitely worth getting the general thrust of this in the right order. (This is in theory backwards incompatible with proprietary Bluespec, but I think we should take the chance to make some of these simple breaks now!)

from bsc.

Historically, the PREFIX was a dedicated directory where Bluespec tools would sit (not a general location like /usr/local) and the directory lib was merely a place for everything that the Bluespec tools needed. That is, a company might install the tools at /fileserver/tools/Bluespec-2019.09.A/ and users would add the bin directory to their path, but otherwise wouldn't need to know what was under lib. Because most of the files there are internally used by the Bluespec tools and didn't need to be in a location visible in the user's environment. For example, the Bluesim headers and libraries are only ever used by CC calls inside BSC; the user will never run any commands that need to see them (which is why it didn't much matter that the files sat in their own directory). Although some files need to be visible, like the Verilog files (although even that may not be necessary if we provided build tools that helped the user gather up all the Verilog for the next steps in their flow).

Prior to this open source release, the BSC tools required that users set BLUESPECDIR in their environment to point to that lib directory. There are many places inside the BSC tools where this path is used to construct paths to files that are user directly or as arguments to commands that get run internally. For the open source release, we wanted to eliminate this requirement, but we didn't have time to change BSC to not need the path (or even time to figure out how); so we compromised and used wrapper scripts in $PREFIX/bin that figure out their location and set BLUESPECDIR to the sibling lib directory.

All of this was similar to how other proprietary hardware design tools operate. For example, if you download Cadence or Synopsys or Xilinx tools, they install into their own directory, and users just need to add its bin to their path and usually set a variable like VCS_HOME or QTHOME or source a provided shell script that, in the case of Xilinx, sets XILINX_VIVADO to the path to the tools' directory. So Bluespec was following this structure, and not, say, the structure one is used to for tools like GCC and packages for individual workstations.

Now, though, it makes sense to move in that direction. We probably should structure the files under PREFIX to support installing into places like /usr/local (or at least to use the same naming conventions). And I agree that it's better to make any big breaks now.

I have no objection to @thoughtpolice 's structure. I might suggest that Bluesim headers go under include/bluespec, just because it's not necessary for them to be automatically in the user's include path. (Although, if it is in the path, then the BSC tools don't need BLUESPECDIR to help find it; so that's a trade off.) And I might suggest that yices still be SAT or else some general name for any of BSC's libraries not going into the top-level /lib for some reason. (Hopefully we can eventually make yices a separately installed requirement rather than installed by this repo, so maybe this directory will go away anyway.) And I would have expected exec to go under share since it's not libraries or "compiled things" (it's executables that get called internally by BSC).

Making this change won't just be a matter of changing the install commands, but also of updating the internals of BSC in every place where it uses BLUESPECDIR to construct a path. And we have to decide whether we keep BLUESPECDIR pointing to the lib next to bin (and just alter the relative paths from there) or whether to make more drastic changes to eliminate the need for BLUESPECDIR inside the tools.

from bsc.

I haven't reviewed the full details here but one minor nitpick that I want to throw out is: it would be nice to avoid weird irregularities with using Uppercase and lowercase in directory names. I mean, even aside from shenanigans with case sensitive filesystems, I think it just looks really weird! So this means it'd also rename e.g. s/Libraries/libraries (or maybe just call is prelude or stdlib?) and Verilog/Quartus to verilog/quartus, etc.

from bsc.

from bsc.

There was a fair amount of work done on Debian packaging, and to a lesser extent on rpm packaging, that addresses organizing the file layout in a way that is consistent with the Linux FHS while also allowing multiple versions of the compiler to be installed simultaneously. That configuration might be out of date, and didn't survive the migration to github, however.

from bsc.

I think we can have two separate steps: (1) create the BSC "glob" of files and (2) install that glob appropriately on some system. I would be happy getting the makefiles to do step 1 and then allowing people to contribute scripts, config files, etc that do step 2 for particular systems.

As I understand, the Debian packaging that @darius-bluespec mentions worked this way anyway: the BSC "inst" directory was created (step 1) and then the packaging files put from there to appropriate places. (In fact, I think it just moved the "inst" directory to /usr/share/PACKAGENAME, or possibly /usr/lib/PACKAGENAME, and created links as necessary to make files and subdirectories available in other locations?)

For @czeck, his step 2 may simply be to copy inst to /opt/bsc/bsc-$VERSION.

I think this 2-step separation has several benefits:

One, the makefiles don't get cluttered with handling all the differences between systems. (For example, the makefiles don't need to support renaming lib with a LIBDIR variable, because that becomes part of the second step.)

Two, it separates the work. We need to tag a version as 4.0.0, and I don't think that step 2 is blocking that. If we get step 1 done, then people can contribute steps 2 files at their leisure. And step 1 can be done without any of the special knowledge that's needed for step 2 (which I don't have).

So I'm happy if the makefiles continue to create a "glob", with some requirements, and people can then install that how they like, respecting the requirements. (An example "requirement" is that BSC looks for Verilog, Bluesim, VPI, etc files in locations relative to a root BLUESPECDIR location; the user doesn't have to set this in their environment if a wrapper around the binary does it, but the files still currently need to be located relative to the same root.)

We should enumerate what the requirements currently are. And, going forward, we can discuss changing those requirements. (For example, instead of hardcoding locations for everything under a single BLUESPECDIR root, maybe there's a config file somewhere that lists locations?)

I think that having this list of "requirements" as the "interface" between step 1 and step 2, means that I can move forward on step 1 (and tagging a commit as version 4.0.0 of BSC) without needing to be experienced in all the ways of installing onto various systems (which I am not), while the people who are (experienced in that) can contribute the files for doing step 2. (FYI, I'm happy to provide the old Debian files that we had, if someone wants to take that forward.)

The README (or any install docs) would also need to be updated to make this clear. That is, make clear that the install PREFIX isn't /usr or /usr/local (as things stand now) but could be a location like /opt/bsc/bsc-VERSION or /usr/share/bsc-VERSION or $HOME/my-tools/bsc-VERSION -- that sort of thing.

And, as Ed points out, end users also sometimes manually use the files in the BSC glob. That's currently done by knowing the locations of the files under the BLUESPECDIR root (which users hardcode into their scripts). If we change the "requirements" to allow distribution packagers to move files around, we'll need to provide ways for users to query the locations (in ways that can be called from the users' scripts).

from bsc.