clasp-developers / clasp Goto Github PK
View Code? Open in Web Editor NEWclasp Common Lisp environment
Home Page: https://clasp-developers.github.io/
clasp Common Lisp environment
Home Page: https://clasp-developers.github.io/
I've observed that the clasp build (which used Boost build) does not fail when encountering a fatal error, as it should.
This is easy to reproduce, but in any case, I encountered it just now. I build a basic Jessis chroot using debootstrap. Then I installed the clang 3.6 package from trunk. I has also some basic dev tools including gcc/g++ 4.9, but none of the other specific clasp dependencies.
This gives streams of errors which look as follows, but the build keeps going.
In file included from ../../src/core/foundation.h:671:
../../src/gctools/memoryManagement.h:39:10: fatal error: 'gc/gc.h' file not
found
This is because the Boehm garbage collector is not installed.
A build should fail fast. In this case, it should fail at the first fatal error. Anything else contravenes a basic law of software development, namely, FAIL FAST.
See the list of file listed in https://bitbucket.org/faheem/clasp-debian/src/tip/rules
under the target override_dh_auto_clean
: If these were moved into make clean
, they
could be removed from debian/rules
.
Here is a list of the files in the Clasp Debian package.
Are all of these needed? In particular, are the Boost Build files necessary?
As in:
(let* ((x 1)
(x (+ 1 x)) (print x))
A first step is to get necesary header and library files included in the Debian package. The libraries are easy, for now, anyway. They are static.
Here is a paste of all header files included in the clasp repos.
https://gist.github.com/605869a9136a8894d2aa
It would be helpful to have a list of top level headers that are required for building stuff (mostly C++ extensions, I suppose). Then one can presumably use some automated method for calculating a list of all header files that depend on those.
1.0
1
(type-of *)
DOUBLE-FLOAT
Guide for Ubuntu 14.04
1-Before doing anything run these commands in the terminal.
sudo apt-get install m4 g++ ncurses-dev libbz2-dev libgmpxx4ldbl
export BOOST_BUILD_PATH=”/home/{computer-name}/clasp/boost_build_v2”
2-Now, go to the externals-clasp folder and make a document called local.config
3-Find the local.config.linux file, copy everything from that file and place it inside of local.config
4-Inside of local.config comment out the GCC_TOOLCHAIN
5-Go to your externals-clasp folder and run make
6-Finally, go to your clasp folder that you cloned from github and run make inside of it.
Happy Lisping!
> (apropos "list-all")
COMPILER::LAMBDA-LIST-ALLOW-OTHER-KEYS
CORE::READ-LIST-ALLOW-CONSING-DOT
LIST-ALL-PACKAGES Function
READER-LIST-ALLOW-CONSING-DOT Function
> (apropos "list-all" :cmp)
zsh: segmentation fault (core dumped) /usr/local/clasp/bin/clasp_mps_o
> (find-package :cmp)
#<COMPILER>
> (apropos "list-all" (find-package :cmp))
zsh: segmentation fault (core dumped) /usr/local/clasp/bin/clasp_mps_o
Since documents are missing, apropos is a key feature.
For Wrapper handle semantics so that the wrapper takes ownership of a unique_ptr and gives up ownership when it returns the object to C++
Check out asttooling/translators.h - this code is currently commented out but it mentions the idea
// You will also need to make clbind::Wrappers do the right thing with std::unique_ptrs
//
template <>
struct from_object<std::vector<std::unique_ptr<clang::ASTUnit>>&>
{
typedef std::vector<std::unique_ptr<clang::ASTUnit>>& DeclareType;
std::vector<std::unique_ptr<clang::ASTUnit>> _Temp;
DeclareType _v;
from_object(core::T_sp o) : _v(_Temp)
{
if ( o.nilp() ) {
this->_v.clear();
return;
} else if ( core::VectorObjects_sp vo = o.asOrNull<core::VectorObjects_O>() ) {
this->_v.clear();
this->_v.resize(vo->length());
for ( int i(0), iEnd(vo->length()); i<iEnd; i++ ) {
this->_v[i] = vo->elt(i).as<clbind::Wrapper<clang::ASTUnit,std::unique_ptr<clang::ASTUnit> > >();
}
return;
}
SIMPLE_ERROR(BF("Could not convert argument %s into std::vector<clang::ASTUnit*>") % _rep_(o));
}
};
This should cause an "illegal function call error" but the reader is getting confused.
I'm working on some basic Debian packaging for Clasp. Preliminary tests suggest that Clasp is really close to buildable to Debian testing/unstable as is. There are a few relatlvely minor road bumps.
So, I'm creating this issue to track my progress.
There is some earlier discussion about this at #22
The most relevant thing there is that the Boehm package Clasp uses is a (minor) fork of upstream, differing by a small patch.
Current status: I've patched the Debian Boehm package with the Clasp patch. Preliminary tests suggest Clasp builds against it Ok.
Currently I'm trying to figure out how to create an orig.tar.gz
tarball. This is complicated because the Clasp Git subrepository contains git submodules of ASDF and MPS.
Reloading .bundle files is not possible atm. Reloading .lsp files works. According to CLHS
http://www.lispworks.com/documentation/HyperSpec/Body/f_load.htm#load
load should behave (almost) equally regarding source and compiled files.
Ensure that it->matcher is fixed.
KIND_GCVECTOR_gctools__GCVector_moveable_asttooling__RegMap__SymbolMatcherDescriptorPair_: {
// processing #S(CONTAINERALLOC :KEY "gctools::GCVector_moveable<asttooling::RegMap::SymbolMatcherDescriptorPair>" :NAME "GCVector_moveable" :LOCATION "/home/meister/Development/clasp/src/gctools/gcvector.h:8:5" :CTYPE #S(GCVECTOR-MOVEABLE-CTYPE :KEY "gctools::GCVector_moveable<asttooling::RegMap::SymbolMatcherDescriptorPair>" :NAME "GCVector_moveable" :ARGUMENTS (#S(GC-TEMPLATE-ARGUMENT :CORE:INDEX 0 :CTYPE #S(CXXRECORD-CTYPE :KEY "asttooling::RegMap::SymbolMatcherDescriptorPair" :NAME "SymbolMatcherDescriptorPair")))))
// parm0-ctype = #S(CXXRECORD-CTYPE :KEY "asttooling::RegMap::SymbolMatcherDescriptorPair" :NAME "SymbolMatcherDescriptorPair")
gctools::GCVector_moveable<asttooling::RegMap::SymbolMatcherDescriptorPair>* obj_gc_safe = reinterpret_cast<gctools::GCVector_moveable<asttooling::RegMap::SymbolMatcherDescriptorPair>*>(client);
for (gctools::GCVector_moveable<asttooling::RegMap::SymbolMatcherDescriptorPair>::iterator it = obj_gc_safe->begin(); it!=obj_gc_safe->end(); ++it) {
// A scanner for #S(CXXRECORD-CTYPE :KEY "asttooling::RegMap::SymbolMatcherDescriptorPair" :NAME "SymbolMatcherDescriptorPair")
SMART_PTR_FIX(it->Name);
POINTER_FIX(it->matcher); // <<<<<< THIS MUST BE FIXED
}
From: https://gist.github.com/jasom/5b7b40debea3558cf09e
You will need gcc:4.8 installed; as of writing this is keyword masked, so unless you are already running ~, you'll need to add sys-devel/gcc:4.8 ~amd64
to your package.keywords. Note that you do not need it to be the default gcc, as clasp allows specifying gcc and g++ executables
Other than that, see the other files in this gist for the config.
cd clasp-externals
make
cd ../clasp
make
I would really, really love to have SLIME integration with Clasp. I've wanted it since I first discovered SLIME and Clasp was just a crazy glimmer in my lazy eye. I've put some work into enabling SLIME support but I haven't yet done any actual work with SWANK (the part of SLIME that would run within Clasp).
Here's what is already incorporated into Clasp to support SLIME/SWANK.
That's as far as I got before I got distracted with getting the static analyzer and MPS integration working and releasing Clasp.
Note: Clasp is very similar to ECL internally. So the ECL SWANK code could be used as a starting point to write the Clasp SWANK code. I use all of the non-compiler parts of the ECL Common Lisp source code. I mimic a lot of the C functionality in ECL with C++ functionality in Clasp. Any differences between the CLHS and Clasp are bugs in Clasp. Any differences between how the exposed functions in Clasp and the exposed functions in ECL behave can be changed to make them behave more like ECL if it retains compatibility with the CLHS and makes it easier to get external code (like SWANK) to work with Clasp.
If anyone wants to pick this up where I left off I'm happy to provide assistance. I wouldn't be able to make any serious progress on SLIME integration before the new year. But if you have time and want to step up and participate - all accolades and praise will go to you!
I'll start:
The Debian package uses the script
https://bitbucket.org/faheem/clasp-debian/src/tip/git-archive-all.sh
(from https://github.com/meitar/git-archive-all.sh) to create an orig.tar.gz tarball
for use with package building. This script is a bit buggy, but could be fixed/improved.
I'd rather have a Python script, though.
In any case, I suggest this be moved into the main clasp repos, and perhaps a make target could
be added. Currently the command used to build the tarball is
./debian/rules get-orig-source
from the top level of the source directory, with the debian/
directory copied in
The tarball specific part of the get-orig-source
target could be made into a tarball
target
in the main makefile, for use with packaging generally. Namely, the following lines
./debian/git-archive-all.sh --prefix clasp-0.1/ ../clasp_0.1.orig.tar
gzip ../clasp_0.1.orig.tar
If the script was on the main level, this would look like
./git-archive-all.sh --prefix clasp-0.1/ ../clasp_0.1.orig.tar
gzip ../clasp_0.1.orig.tar
Though having the 0.1
hardwired in there is of course, undesirable.
Tthe lisp part of the build seems to run sequentially on one core. Parallelizing this would speed things up considerably. I'm don't know how difficult this would be to do, though.
The function INTEGER-LENGTH is not currently defined. It can be found in the HyperSpec here. It should return the number of bits required to represent the number that was given as a parameter, for example:
> (integer-length 5)
3
Any chance of a homebrew formula that encapsulates the installation steps?
Currently the Debian package has the followign patch.
https://bitbucket.org/faheem/clasp-debian/src/tip/patches/cmpbundle.lsp.patch
It would be nice if this could be got rid of. There was some talk of using clang where llc is
currently used, I believe.
The pack is ~700MB
I'm working on getting a list of files that we can delete to fix that. Will update once I have it, then Christian can run a filter against it.
The build system is using the python
command at the moment. This works fine for all distros that have a Python 2 executable under this command, however there are notable exceptions such as Arch Linux (where python
points to Python 3) and NixOS (where there is no python
command and the Python version has to be specified explicitly). PEP 394 suggests using python2
as command and substituting shebangs such as #!/usr/bin/python
and #!/usr/bin/env python
with #!/usr/bin/python2
and #!/usr/bin/env python2
.
Here's a git-format patch that fixes this issue and has been tested on an Arch Linux system:
From 9ae0761b9e72ce62a6a83d33647033aa9e33d8ca Mon Sep 17 00:00:00 2001
From: Vasilij Schneidermann <[email protected]>
Date: Wed, 1 Oct 2014 22:56:49 +0200
Subject: [PATCH] Python2 fixes
---
bundler.jam | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/bundler.jam b/bundler.jam
index ee95737..e148300 100644
--- a/bundler.jam
+++ b/bundler.jam
@@ -52,7 +52,7 @@ rule bundle ( dir : sources * : properties * : opt_props * : dbg_props * )
echo Scraping in $(dir) ;
- out = [ SHELL "cd $(dir); export PYTHONPATH=`pwd`:${PYTHONPATH}; python $(.common)/symbolScraper.py *.cc *.h" : exit-status ] ;
+ out = [ SHELL "cd $(dir); export PYTHONPATH=`pwd`:${PYTHONPATH}; python2 $(.common)/symbolScraper.py *.cc *.h" : exit-status ] ;
if $(out[2]) = 0 {
ECHO $(out[1]) ;
@@ -135,7 +135,7 @@ actions register-classes
{
cd $(>[1]:D)
bname=`basename $(>[1]:D)`
- python $(.common)/registerClasses.py $(<) $(<:D)/${bname}_initScripting_inc.h *Package.h otherPackageClasses.h *.h $(<:D)/*.h *.cc $(<:D)/*.cc > registerClasses.log
+ python2 $(.common)/registerClasses.py $(<) $(<:D)/${bname}_initScripting_inc.h *Package.h otherPackageClasses.h *.h $(<:D)/*.h *.cc $(<:D)/*.cc > registerClasses.log
}
actions ltoh
@@ -147,7 +147,7 @@ actions ltoh
actions ptoh
{
cd $(>[1]:D)
- python $(.common)/pump.py $(>[1]) $(<)
+ python2 $(.common)/pump.py $(>[1]) $(<)
}
actions ltocc
--
2.1.0
I noticed the mps git clone happens after the boehm build is complete. This really should happen early. If there are network problems, generally it is better to fail early/fast.
This should not be hard to rearrange, I think.
Note that for a build where one is not trying to create a package, this doesn't matter so much, because presumably the necessary boehm file has been generated before the mps one starts, but for a binary package build like Debian the package will simply not be built.
externals-clasp built successfully, then clasp failed to link some libraries (namely cord, expat, gc, gmp, readline and history (everything minus libz)).
All the libs were present but in different locations. libz was at common/lib, the rest at common/lib64.
Copy (or move) all the libs from common/lib64 to common/lib and retry make.
Specify --libdir for each library -> http://pastebin.com/uXGpnwMu (search for --libdir)
With the given fix clasp almost builds and links except for libgmp. See the log -> http://pastebin.com/kFkDwNrw. I can confirm that libgmpxx.so.4 exists and points to the right place. A simple LD_LIBRARY_PATH=~/clasp/externals-clasp/common/lib make
did the trick. No clue on this one (have not read the build scripts).
A happy clasp build.
Lowered the number of jobs to avert memory trashing.
The comments in local.config.Linux of externals.clasp state that one can set GCC_EXECUTABLE and GXX_EXECUTABLE if gcc and g++ are not under GCC_TOOLCHAIN/bin. This suggests that these can be used instead of the GCC_TOOLCHAIN setting. externals.clasp will indeed build if following this advice, including clang 3.6. However, the resultant clang will be unable to link any executables, failing to find crtbegin.o from the gcclib-dev Linux package. It is indeed necessary to set GCC_TOOLCHAIN to arrive at a usable clang, even if overriding the gcc executable locations. I suggest amending the comments to that effect. Also suggest to change Linux GCC_TOOLCHAIN to /usr instead of $(HOME)/local/gcc-4.8.3. /usr is where common distributions like Ubuntu will place the system gcc. In this manner the configuration would "work out of the box" for anyone with a recent gcc.
Best Regards
Chris Kohlhepp
I had to install libraries for ncurses-dev and libbz2-dev.
sudo apt-get install ncurses-dev
sudo apt-get install libbz2-dev
Sometimes if you "make" from the initial cloned repository or you "make clean; make" boost build will complain about a missing .h file. I think its core_scrape_flag.h but I'm not looking at the error right now.
You can control-C the build and make again and it goes away. It's something to do with the order things are built by boost build.
When in the repl, if you issue an EOF (C-d), clasp loops forever in the top level, spewing:
Unknown top level command: :EOF
After the second error is thrown I would expect this to become >>>
and also expect to see some mention of :r2
.
$ rlwrap /usr/lib/clasp/bin/clasp_boehm_o
Starting Clasp 0.1... loading image... it takes a few seconds
Could not find pathname: /home/faheem/.clasprc
Top level.
> (1 1)
conditions.lsp>>signal-simple-error base-condition = COMMON-LISP:CONTROL-ERROR format-control = In throwCatchThrow ../../src/llvmo/intrinsics.cc line 1303
COMMON-LISP:CONTROL-ERROR :initializers nil
args= nil
conditions.lsp>>signal-simple-error base-condition = COMMON-LISP:PROGRAM-ERROR format-control = No value supplied for the init-name ~S. args= (nil )@0x498bd18
Condition of type: SIMPLE-PROGRAM-ERROR
No value supplied for the init-name NIL.
Available restarts:
(use :r1 to invoke restart 1)
1. (RESTART-TOPLEVEL) Go back to Top-Level REPL.
Broken at frame[44] CORE::REP.
File: #<CORE:SOURCE-FILE-INFO #P"/usr/local/src/clasp/clasp-0.1/debian/build/usr/lib/clasp/Contents/Resources/lisp/kernel/lsp/top.lsp"> (Position #573)
>> (1 1)
conditions.lsp>>signal-simple-error base-condition = COMMON-LISP:CONTROL-ERROR format-control = In throwCatchThrow ../../src/llvmo/intrinsics.cc line 1303
COMMON-LISP:CONTROL-ERROR :initializers nil
args= nil
conditions.lsp>>signal-simple-error base-condition = COMMON-LISP:PROGRAM-ERROR format-control = No value supplied for the init-name ~S. args= (nil )@0x80e4b98
Debugger received error of type: SIMPLE-PROGRAM-ERROR
No value supplied for the init-name NIL.
Error flushed.
>>
Several files look like they're only included in the distribution accidentally.
will update if i notice more.
*READ-DEFAULT-FLOAT-FORMAT*
=> SINGLE-FLOAT
(type-of 1.2) => DOUBLE-FLOAT
incidentally, it should also be consulted when printing, non-default format should use the exponent marker, i.e. 1.2d0 when it's double-float, and 1.9f0 if it's SINGLE-FLOAT, and no marker if it's the same format.
The Clasp build should be able to proceed using the following steps:
.git
directories for the Clasp repos and the Git submodulesThis doesn't currently appear to be the case.
pkhuong in #sbcl suggested passing remaining arguments in the multiple_values array.
He also suggested setting up the lambda-list processing code so that it first processes all of the provided argument and then in a second pass processes the default/initial forms.
He said I can keep track of which arguments are missing in scalar registers.
This misleading documentation has been inherited from ecl apparently.
but no reason not to fix it here. I guess one could report it to ecl too,
in case anyone cares.
If I can figure out how to recompile only the common lisp sources,
then i can try patching this. It probably is not hard.
faheem@orwell:~/local/clasp/bin$ rlwrap ./clasp_boehm_o
Starting Clasp 0.1... loading image... it takes a few seconds
Could not find pathname: /home/faheem/.clasprc
Top level.
(1 1)
Condition of type: SIMPLE-ERROR
obj must be a symbol - instead it is: 1
Available restarts:
Broken at frame[29] CORE::REP.
File: #<CORE:SOURCE-FILE-INFO #P"/home/faheem/local/clasp/Contents/Resources/lisp/kernel/lsp/top.lsp"> (Position #573)
1
1
:r1
Can we break out the chaos going on in the building and installing into its own .md file and have the readme focus on normal readme info. I feel like all that information in the readme can be off-putting to potentional new contributors. Looking around at some high starred repos for good readme examples
https://github.com/brendangregg/perf-tools
https://github.com/peter-murach/tty
site-config.jam is being ignored, but I need a user-config.jam that uses clang. The using clang
conflicts with one the one from Jamroot.jam:9, since it specifies version 3.6. This kills the build before any targets are built.
Compiling clasp on 32bit systems is broken. size_t == uint
Which is 70 years too early, needs to add 2208988800 seconds.
mband asked for the following in #clasp. It would be nice if you could do a somewhat quick writeup giving us an overview of the system (targeted developers - I really would like to have the learning/discover barrier reduced a bit) - like the flow lisp input->compilation->llvm backend and some about where it's located in the sources.
(time (sleep 1))
real time : 0.001 secs
(time (sleep 1000))
real time : 1.000 secs
A user reported that at its peak clasp uses 11 Gb for building. This is too much.
Also, if the build grabs memory too fast, at least on Linux, the Out of Memory killer might be going into action.
I suspect that is what is happening on my box. I have 16gb, and I don't think
that I am running out of memory, but when I try to build mps, first the disk starts churning, and the red light on my box, which I think means the disk is being written to, stays lit for several minutes continuously, then the build gets killed. It has now been killed twice - the second time I just did "make clasp-mps". I dont; know exactly why, but it may be the OOM killer.
Clang produced a traceback and a preprocessed file, and I've submitted an issue to LLVM, but the problem may just be that the process is getting killed.
In /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include
There needs to be a symbolic link in that directory:
total 8
-rw-r--r-- 1 root wheel 6235 Sep 27 2013 FlexLexer.h
lrwxr-xr-x 1 root wheel 10 Apr 28 12:49 c++@ -> ../lib/c++
fry:include$ pwd
This is because the build system tells clang where the resources are with:
darwin:"-resource-dir /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/5.1"
The problem is that clang takes that resource-dir and changes it to these paths:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/5.1/include
and the include/c++/v1 path doesn't exist on OS X 10.9 Xcode 5.1!!!!!
But the files that it's looking for are there but in:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/c++
Boost Python converters have some good features.
Also, suppose one has a converter for T. Then, supposing one has a converter for Something<:T>, then one only has to specify a converter for Something
, and the T is taken care of transparently. One does not have to specify conversion of T. One might term this as automatic composition of converters. Here is an example for vector:
https://bitbucket.org/faheem/corrmodel/src/tip/cppvec_conv_pif.cc?at=default
Of course, this won't happen unless the converter can be templated on T in the first place.
When I try to build clasp on Ubuntu 14.04, it say config-cache.jam
is missing.
Once I put config-cache.jam
copied from other project on boost_build_v2/build
, it start compiling.
On Debian Wheezy system gcc had to be back ported (upgraded) to gcc 4.9.
(/ -1 2) => -1/2 ;; right
(/ 1 -2) => 1/-2 ;; Not right
(/ -1 -2) => -1/-2 ;; really Not right
(/ 10 20) => 10/20 ;; Not right
(coerce '(#\a #\b) 'string)
=>
Concrete ideas from issue #18 will be put here, once I get to it.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.