Assertion failure in readwrite about ctf HOT 10 CLOSED

Really? Doesn't this test just fill a nonsymmetric matrix with zeros? I get no error when running it with 48 MPI processors.

from ctf.

Hmm, fails for me. Let me try a clean pull.

On 1/8/16 4:59 PM, Edgar Solomonik wrote:

Really? Doesn't this test just fill a nonsymmetric matrix with zeros?
I get no error when running it with 48 MPI processors.

—
Reply to this email directly or view it on GitHub
https://github.com/solomonik/ctf/issues/13#issuecomment-170150972.

from ctf.

If it still fails let me know what compiler and flags you are you using.
Also, that ASSERT is a macro that I thought should not even trigger on a
build without -DDEBUG or -DVERBOSE.

On Sat, Jan 9, 2016 at 12:02 AM, Devin Matthews [email protected]
wrote:

Hmm, fails for me. Let me try a clean pull.

On 1/8/16 4:59 PM, Edgar Solomonik wrote:

Really? Doesn't this test just fill a nonsymmetric matrix with zeros?
I get no error when running it with 48 MPI processors.

—
Reply to this email directly or view it on GitHub
https://github.com/solomonik/ctf/issues/13#issuecomment-170150972.

—
Reply to this email directly or view it on GitHub
https://github.com/solomonik/ctf/issues/13#issuecomment-170151395.

from ctf.

This is compiled with "-g -O0 -DDEBUG=1".

On 1/8/16 5:04 PM, Edgar Solomonik wrote:

If it still fails let me know what compiler and flags you are you using.
Also, that ASSERT is a macro that I thought should not even trigger on a
build without -DDEBUG or -DVERBOSE.

On Sat, Jan 9, 2016 at 12:02 AM, Devin Matthews [email protected]
wrote:

Hmm, fails for me. Let me try a clean pull.

On 1/8/16 4:59 PM, Edgar Solomonik wrote:

Really? Doesn't this test just fill a nonsymmetric matrix with zeros?
I get no error when running it with 48 MPI processors.

—
Reply to this email directly or view it on GitHub
https://github.com/solomonik/ctf/issues/13#issuecomment-170150972.

—
Reply to this email directly or view it on GitHub
https://github.com/solomonik/ctf/issues/13#issuecomment-170151395.

—
Reply to this email directly or view it on GitHub
https://github.com/solomonik/ctf/issues/13#issuecomment-170152244.

from ctf.

Yes, this still happens after a clean pull (from master). My compile line is:

mpicxx -o test.x -std=c++11 -fopenmp -g -O0 -DDEBUG=1 -I/home1/dmatthews/src/aquarius/src/external/ctf/include -L/home1/dmatthews/src/aquarius/src/external/ctf/lib test.cxx -lctf -L/apps1/intel/13.1/lib/intel64 -Wl,--start-group /apps1/intel/13.1/mkl/lib/intel64/libmkl_intel_lp64.a /apps1/intel/13.1/mkl/lib/intel64/libmkl_core.a /apps1/intel/13.1/mkl/lib/intel64/libmkl_gnu_thread.a -Wl,--end-group -lpthread -lm

OpenMPI 1.6.5
GCC 4.8.2

Also, this is using flat MPI on 6 nodes with 8 cores each if that matters. config.mk is:

### LINK TIME LIBRARIES AND FLAGS
#libraries and flags for link time (irrelevant if only building CTF lib and not examples/tests)
BLAS_LIBS   =  
LDFLAGS     = 

### COMPILE TIME INCLUDES AND FLAGS
#C++ compiler 
CXX         = mpicxx -std=c++0x
#includes for compile time
INCLUDES    = 
#optimization flags, some intel compiler versions may run into errors when using -fast or -ipo
CXXFLAGS    = -O0 -g -fopenmp -Wall 
#command to make library out of object files
AR          = ar

#macros to be defined throughout the code
DEFS        = -D_POSIX_C_SOURCE=200112L -D__STDC_LIMIT_MACROS -DFTN_UNDERSCORE=1 

### Optional: PROFILING AND TUNING
#uncomment below to enable performance profiling
#DEFS       += -DPROFILE -DPMPI
#uncomment below to enable automatic performance tuning (loses reproducibility of results)
#Note: -DTUNE requires lapack (include -mkl or -llapack in BLAS_LIBS)
#DEFS       += -DTUNE

### Optional: DEBUGGING AND VERBOSITY
#uncomment below to enable CTF execution output (1 for basic contraction information on start-up and contractions)
#DEFS       += -DVERBOSE=1
#uncomment to set debug level to dump information about mapping and internal CTF actions and activate asserts
DEFS       += -DDEBUG=1

### FULL COMPILE COMMAND AND LIBRARIES
#used to compile all plain C++ files
FCXX        = $(CXX) $(CXXFLAGS) $(DEFS) $(INCLUDES)
#link-line for all executables
LIBS        = $(BLAS_LIBS) $(LDFLAGS)
#compiler for CUDA files (used to compile CUDA code only when -DOFFLOAD and -DUSE_CUDA are in DEFS, otherwise should be same as FCXX with -x c++)
OFFLOAD_CXX = $(CXX) -x c++ $(CXXFLAGS) $(DEFS) $(INCLUDES)

from ctf.

Very strange. Does the CTF test_suite work on 48 cores with this build
setup?

On Sat, Jan 9, 2016 at 12:15 AM, Devin Matthews [email protected]
wrote:

Yes, this still happens after a clean pull (from master). My compile line
is:

mpicxx -o test.x -std=c++11 -fopenmp -g -O0 -DDEBUG=1 -I/home1/dmatthews/src/aquarius/src/external/ctf/include -L/home1/dmatthews/src/aquarius/src/external/ctf/lib test.cxx -lctf -L/apps1/intel/13.1/lib/intel64 -Wl,--start-group /apps1/intel/13.1/mkl/lib/intel64/libmkl_intel_lp64.a /apps1/intel/13.1/mkl/lib/intel64/libmkl_core.a /apps1/intel/13.1/mkl/lib/intel64/libmkl_gnu_thread.a -Wl,--end-group -lpthread -lm

OpenMPI 1.6.5
GCC 4.8.2

Also, this is using flat MPI on 6 nodes with 8 cores each if that matters.
config.mk is:

LINK TIME LIBRARIES AND FLAGS#libraries and flags for link time (irrelevant if only building CTF lib and not examples/tests)

BLAS_LIBS =
LDFLAGS =

COMPILE TIME INCLUDES AND FLAGS#C++ compiler

CXX = mpicxx -std=c++0x#includes for compile time
INCLUDES = #optimization flags, some intel compiler versions may run into errors when using -fast or -ipo
CXXFLAGS = -O0 -g -fopenmp -Wall #command to make library out of object files
AR = ar
#macros to be defined throughout the code
DEFS = -D_POSIX_C_SOURCE=200112L -D__STDC_LIMIT_MACROS -DFTN_UNDERSCORE=1

Optional: PROFILING AND TUNING#uncomment below to enable performance profiling#DEFS += -DPROFILE -DPMPI#uncomment below to enable automatic performance tuning (loses reproducibility of results)#Note: -DTUNE requires lapack (include -mkl or -llapack in BLAS_LIBS)#DEFS += -DTUNE

Optional: DEBUGGING AND VERBOSITY#uncomment below to enable CTF execution output (1 for basic contraction information on start-up and contractions)#DEFS += -DVERBOSE=1#uncomment to set debug level to dump information about mapping and internal CTF actions and activate asserts

DEFS += -DDEBUG=1

FULL COMPILE COMMAND AND LIBRARIES#used to compile all plain C++ files

FCXX = $(CXX) $(CXXFLAGS) $(DEFS) $(INCLUDES)#link-line for all executables
LIBS = $(BLAS_LIBS) $(LDFLAGS)#compiler for CUDA files (used to compile CUDA code only when -DOFFLOAD and -DUSE_CUDA are in DEFS, otherwise should be same as FCXX with -x c++)
OFFLOAD_CXX = $(CXX) -x c++ $(CXXFLAGS) $(DEFS) $(INCLUDES)

—
Reply to this email directly or view it on GitHub
https://github.com/solomonik/ctf/issues/13#issuecomment-170153877.

from ctf.

Something is very messed up. The testsuite runs fine on one core, but on
48 I eventually wind up with an invalid datatype error on MPI_DOUBLE
(during a redistribution on a complex tensor...).

I suppose I'll try MPICH.

On 1/8/16 5:18 PM, Edgar Solomonik wrote:

Very strange. Does the CTF test_suite work on 48 cores with this build
setup?

On Sat, Jan 9, 2016 at 12:15 AM, Devin Matthews [email protected]
wrote:

Yes, this still happens after a clean pull (from master). My compile
line
is:

mpicxx -o test.x -std=c++11 -fopenmp -g -O0 -DDEBUG=1
-I/home1/dmatthews/src/aquarius/src/external/ctf/include
-L/home1/dmatthews/src/aquarius/src/external/ctf/lib test.cxx -lctf
-L/apps1/intel/13.1/lib/intel64 -Wl,--start-group
/apps1/intel/13.1/mkl/lib/intel64/libmkl_intel_lp64.a
/apps1/intel/13.1/mkl/lib/intel64/libmkl_core.a
/apps1/intel/13.1/mkl/lib/intel64/libmkl_gnu_thread.a -Wl,--end-group
-lpthread -lm

OpenMPI 1.6.5
GCC 4.8.2

Also, this is using flat MPI on 6 nodes with 8 cores each if that
matters.
config.mk is:

LINK TIME LIBRARIES AND FLAGS#libraries and flags for link time

(irrelevant if only building CTF lib and not examples/tests)
BLAS_LIBS =
LDFLAGS =

COMPILE TIME INCLUDES AND FLAGS#C++ compiler

CXX = mpicxx -std=c++0x#includes for compile time
INCLUDES = #optimization flags, some intel compiler versions may run
into errors when using -fast or -ipo
CXXFLAGS = -O0 -g -fopenmp -Wall #command to make library out of
object files
AR = ar
#macros to be defined throughout the code
DEFS = -D_POSIX_C_SOURCE=200112L -D__STDC_LIMIT_MACROS
-DFTN_UNDERSCORE=1

Optional: PROFILING AND TUNING#uncomment below to enable

performance profiling#DEFS += -DPROFILE -DPMPI#uncomment below to
enable automatic performance tuning (loses reproducibility of
results)#Note: -DTUNE requires lapack (include -mkl or -llapack in
BLAS_LIBS)#DEFS += -DTUNE

Optional: DEBUGGING AND VERBOSITY#uncomment below to enable CTF

execution output (1 for basic contraction information on start-up and
contractions)#DEFS += -DVERBOSE=1#uncomment to set debug level to dump
information about mapping and internal CTF actions and activate asserts
DEFS += -DDEBUG=1

FULL COMPILE COMMAND AND LIBRARIES#used to compile all plain C++

files
FCXX = $(CXX) $(CXXFLAGS) $(DEFS) $(INCLUDES)#link-line for all
executables
LIBS = $(BLAS_LIBS) $(LDFLAGS)#compiler for CUDA files (used to
compile CUDA code only when -DOFFLOAD and -DUSE_CUDA are in DEFS,
otherwise should be same as FCXX with -x c++)
OFFLOAD_CXX = $(CXX) -x c++ $(CXXFLAGS) $(DEFS) $(INCLUDES)

—
Reply to this email directly or view it on GitHub
https://github.com/solomonik/ctf/issues/13#issuecomment-170153877.

—
Reply to this email directly or view it on GitHub
https://github.com/solomonik/ctf/issues/13#issuecomment-170154333.

from ctf.

This + the test suite runs OK on Stampede (48 cores). I'll try MPICH on the other system and if that works I'll close the issue.

from ctf.

Test suite passes with MPICH. OpenMPI is some seriously messed up shit.

from ctf.

Well old OpenMPI implementations are notoriously buggy. I will switch
CTF back to using CXX instead of C for datatypes then.

On Mon, Jan 11, 2016 at 5:40 PM, Devin Matthews [email protected]
wrote:

Closed #13 https://github.com/solomonik/ctf/issues/13.

—
Reply to this email directly or view it on GitHub
https://github.com/solomonik/ctf/issues/13#event-511321396.

from ctf.