They can be tested using:

jupyter nbconvert --to notebook --execute --ExecutePreprocessor.timeout=60 --output out_file 3D\ Double\ Well.ipynb

One file triggers internal compiler error:

[ 63%] Building Fortran object src/tests/CMakeFiles/test_ofmd_fft.dir/test_ofmd_fft.f90.o
/home/certik/repos/hfsolver/src/tests/test_ofmd_fft.f90: In function ‘test_ofmd_fft’:
/home/certik/repos/hfsolver/src/tests/test_ofmd_fft.f90:127:0: internal compiler error: Segmentation fault
     X = X - L*floor(X/L)
 ^
0x90f44f crash_signal
    ../.././gcc/toplev.c:337
0x604e0d is_runtime_conformable
    ../.././gcc/fortran/trans-expr.c:7856
0x604e1b is_runtime_conformable
    ../.././gcc/fortran/trans-expr.c:7856
0x611aa4 gfc_trans_assignment_1
    ../.././gcc/fortran/trans-expr.c:8117
0x5e3fc5 trans_code
    ../.././gcc/fortran/trans.c:1639
0x637441 gfc_trans_simple_do
    ../.././gcc/fortran/trans-stmt.c:1443
0x637441 gfc_trans_do(gfc_code*, tree_node*)
    ../.././gcc/fortran/trans-stmt.c:1606
0x5e3dfa trans_code
    ../.././gcc/fortran/trans.c:1748
0x6033bb gfc_generate_function_code(gfc_namespace*)
    ../.././gcc/fortran/trans-decl.c:5653
0x5a2f80 translate_all_program_units
    ../.././gcc/fortran/parse.c:4953
0x5a2f80 gfc_parse_file()
    ../.././gcc/fortran/parse.c:5150
0x5e0025 gfc_be_parse_file
    ../.././gcc/fortran/f95-lang.c:212
Please submit a full bug report,
with preprocessed source if appropriate.
Please include the complete backtrace with any bug report.
See <http://gcc.gnu.org/bugs.html> for instructions.
make[2]: *** [src/tests/CMakeFiles/test_ofmd_fft.dir/test_ofmd_fft.f90.o] Error 1
make[1]: *** [src/tests/CMakeFiles/test_ofmd_fft.dir/all] Error 2
make: *** [all] Error 2

I have isolated and reported the bug at https://gcc.gnu.org/bugzilla/show_bug.cgi?id=62142

As reported here:

https://groups.google.com/d/topic/travis-ci/26n1N8Jiyb4/discussion

Below are all builds 80-117 (both failing and working). And all failing builds 1-79 (the builds that worked are not listed).

Before

Builds that fail:

https://travis-ci.org/#!/certik/hfsolver/jobs/2685436 ppp1.worker.travis-ci.org:php-1
https://travis-ci.org/#!/certik/hfsolver/jobs/2685336 ppp1.worker.travis-ci.org:php-3
https://travis-ci.org/#!/certik/hfsolver/jobs/2648071 ppp1.worker.travis-ci.org:php-2
https://travis-ci.org/#!/certik/hfsolver/jobs/2684505 ppp1.worker.travis-ci.org:php-2
https://travis-ci.org/#!/certik/hfsolver/jobs/2672574 ppp1.worker.travis-ci.org:php-4
https://travis-ci.org/#!/certik/hfsolver/jobs/2648071 ppp1.worker.travis-ci.org:php-2

https://travis-ci.org/#!/certik/hfsolver/jobs/2440462 ppp1.worker.travis-ci.org:php-3
https://travis-ci.org/#!/certik/hfsolver/jobs/2532876 ppp1.worker.travis-ci.org:php-4
https://travis-ci.org/#!/certik/hfsolver/jobs/2533937 ppp1.worker.travis-ci.org:php-1
https://travis-ci.org/#!/certik/hfsolver/jobs/2542619 ppp1.worker.travis-ci.org:php-4
https://travis-ci.org/#!/certik/hfsolver/jobs/2552088 ppp1.worker.travis-ci.org:php-1
https://travis-ci.org/#!/certik/hfsolver/jobs/2632309 ppp1.worker.travis-ci.org:php-5
https://travis-ci.org/#!/certik/hfsolver/jobs/2632693 ppp1.worker.travis-ci.org:php-1
https://travis-ci.org/#!/certik/hfsolver/jobs/2632798 ppp1.worker.travis-ci.org:php-2
https://travis-ci.org/#!/certik/hfsolver/jobs/2645498 ppp1.worker.travis-ci.org:php-4
https://travis-ci.org/#!/certik/hfsolver/jobs/2647667 ppp1.worker.travis-ci.org:php-5

Builds that worked:

https://travis-ci.org/#!/certik/hfsolver/jobs/2685312 ppp2.worker.travis-ci.org:php-4
https://travis-ci.org/#!/certik/hfsolver/jobs/2684638 ppp2.worker.travis-ci.org:php-4
https://travis-ci.org/#!/certik/hfsolver/jobs/2671106 ppp3.worker.travis-ci.org:php-4
https://travis-ci.org/#!/certik/hfsolver/jobs/2685299 ppp4.worker.travis-ci.org:php-4
https://travis-ci.org/#!/certik/hfsolver/jobs/2685451 ppp2.worker.travis-ci.org:php-5
https://travis-ci.org/#!/certik/hfsolver/jobs/2684484 ppp3.worker.travis-ci.org:php-4
https://travis-ci.org/#!/certik/hfsolver/jobs/2683882 ppp3.worker.travis-ci.org:php-2
https://travis-ci.org/#!/certik/hfsolver/jobs/2683363 ppp4.worker.travis-ci.org:php-4
https://travis-ci.org/#!/certik/hfsolver/jobs/2673000 ppp3.worker.travis-ci.org:php-3
https://travis-ci.org/#!/certik/hfsolver/jobs/2654465 ppp4.worker.travis-ci.org:php-4
https://travis-ci.org/#!/certik/hfsolver/jobs/2653859 ppp4.worker.travis-ci.org:php-5
https://travis-ci.org/#!/certik/hfsolver/jobs/2653549 ppp4.worker.travis-ci.org:php-2

After instrumentation:

Builds that fail:

https://travis-ci.org/#!/certik/hfsolver/builds/2699327 ppp1.worker.travis-ci.org:php-1
https://travis-ci.org/#!/certik/hfsolver/builds/2699331 ppp1.worker.travis-ci.org:php-5
https://travis-ci.org/#!/certik/hfsolver/builds/2699344 ppp1.worker.travis-ci.org:php-4
https://travis-ci.org/#!/certik/hfsolver/builds/2699346 ppp1.worker.travis-ci.org:php-3

Builds that work:

https://travis-ci.org/#!/certik/hfsolver/builds/2698791 ppp3.worker.travis-ci.org:php-4
https://travis-ci.org/#!/certik/hfsolver/jobs/2698841 ppp4.worker.travis-ci.org:php-5
https://travis-ci.org/#!/certik/hfsolver/builds/2698767 ppp2.worker.travis-ci.org:php-1
https://travis-ci.org/#!/certik/hfsolver/jobs/2698669 ppp4.worker.travis-ci.org:php-3
https://travis-ci.org/#!/certik/hfsolver/jobs/2699069 ppp2.worker.travis-ci.org:php-4
https://travis-ci.org/#!/certik/hfsolver/builds/2699096 ppp4.worker.travis-ci.org:php-5
https://travis-ci.org/#!/certik/hfsolver/builds/2699281 ppp2.worker.travis-ci.org:php-3
https://travis-ci.org/#!/certik/hfsolver/builds/2699283 ppp4.worker.travis-ci.org:php-3
https://travis-ci.org/#!/certik/hfsolver/builds/2699329 ppp4.worker.travis-ci.org:php-5
https://travis-ci.org/#!/certik/hfsolver/builds/2699340 ppp2.worker.travis-ci.org:php-1
https://travis-ci.org/#!/certik/hfsolver/builds/2699342 ppp2.worker.travis-ci.org:php-5
https://travis-ci.org/#!/certik/hfsolver/builds/2699353 ppp4.worker.travis-ci.org:php-4

Hardware

ppp1: Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz
ppp2: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
ppp3: Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz
ppp4: Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz