internal compiler error: in tsubst_copy, at cp/pt.c:13970 using gcc 6.2 about sonnet HOT 23 CLOSED

@kosklain: Yes, I am referring to the submodule and these are steps which worked for me together with the patch #6:

cd sonnet/tensorflow
git checkout r1.1
./configure
bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
sudo -H pip uninstall tensorflow
sudo -H pip install /tmp/tensorflow_pkg/tensorflow-1.1.0rc1-cp27-cp27mu-linux_x86_64.whl
cd ../
bazel build --config=opt //:install
bazel-bin/install /tmp/sonnet_pkg
sudo -H pip uninstall sonnet
sudo -H pip install /tmp/sonnet_pkg/sonnet-1.0-py2-none-any.whl

from sonnet.

Got it working after using r1.1 branch for tensorflow, recompiled and installed from source. The master branch for tensorflow did not work for me. Also had to augment LD_LIBRARY_PATH and use the patch #6, both are described above.

>>> snt.resampler(tf.constant([0.]), tf.constant([0.]))
Tensor("resampler/Resampler:0", shape=(1,), dtype=float32)

~/projects/tensor/sonnet/sonnet/examples$ python rnn_shakespeare_test.py 
2017-04-08 16:08:26.416213: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2017-04-08 16:08:26.416596: I tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 0 with properties: 
name: GeForce GT 750M
major: 3 minor: 0 memoryClockRate (GHz) 0.967
pciBusID 0000:02:00.0
Total memory: 1.95GiB
Free memory: 1.10GiB
2017-04-08 16:08:26.416616: I tensorflow/core/common_runtime/gpu/gpu_device.cc:908] DMA: 0 
2017-04-08 16:08:26.416622: I tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 0:   Y 
2017-04-08 16:08:26.416633: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GT 750M, pci bus id: 0000:02:00.0)
..
----------------------------------------------------------------------
Ran 2 tests in 59.776s

OK

from sonnet.

I've pushed the image to docker hub in case anyone wants to use it for experimentation

docker run -p 8888:8888 -it davemssavage/sonnet

from sonnet.

Ideally we want to not have to compile tensorflow (given that it can get complex and might take a lot of time) even with new compilers. We are resolving the issues for people that compile sonnet with gcc 5x, partly solved by using:

bazel build --config=opt --copt="-D_GLIBCXX_USE_CXX11_ABI=0" :install

For now I think it's safe to say that this issue will be resolved when #25 is resolved. Marking as duplicate to track progress there.

from sonnet.

Same here with gcc 5.4. This fixed it for me:

diff --git a/sonnet/cc/kernels/resampler_op.cc b/sonnet/cc/kernels/resampler_op.cc
index f0a5e64..32d5ae3 100644
--- a/sonnet/cc/kernels/resampler_op.cc
+++ b/sonnet/cc/kernels/resampler_op.cc
@@ -236,8 +236,8 @@ struct ResamplerGrad2DFunctor<CPUDevice, T>{
     memset(grad_data, 0, sizeof(T) * grad_data_size);
     memset(grad_warp, 0, sizeof(T) * grad_warp_size);
 
-    const int data_batch_stride = data_height * data_width * data_channels;
-    const int warp_batch_stride = num_sampling_points * 2;
+    int data_batch_stride = data_height * data_width * data_channels;
+    int warp_batch_stride = num_sampling_points * 2;
     const int output_batch_stride = num_sampling_points * data_channels;
     const T zero = static_cast<T>(0.0);
     const T one = static_cast<T>(1.0);

Even though installation seems to fail:

Python 2.7.12 (default, Nov 19 2016, 06:48:10) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sonnet as snt
>>> import tensorflow as tf
>>> snt.resampler(tf.constant([0.]), tf.constant([0.]))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "sonnet/python/ops/resampler.py", line 62, in resampler
    raise ImportError("_gen_resampler could not be imported.")
ImportError: _gen_resampler could not be imported.

from sonnet.

The same here. _resampler.so could not be loaded.

from sonnet.

There is a file _resampler.so-2.params but not the _resampler.so it is looking for. Can't build the wheel due to this error.

from sonnet.

The problem with _resampler.so is:

>>> import tensorflow as tf
>>> _gen_resampler = tf.load_op_library(tf.resource_loader.get_path_to_datafile("_resampler.so"))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/load_library.py", line 64, in load_op_library
    None, None, error_msg, error_code)
tensorflow.python.framework.errors_impl.NotFoundError: _resampler.so: cannot open shared object file: No such file or directory

After augmenting the LD_LIBRARY_PATH:

~/projects/tensor$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/python2.7/dist-packages/sonnet/python/ops

I am getting bit different problem:

>>> import tensorflow as tf
>>> _gen_resampler = tf.load_op_library(tf.resource_loader.get_path_to_datafile("_resampler.so"))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/load_library.py", line 64, in load_op_library
    None, None, error_msg, error_code)
tensorflow.python.framework.errors_impl.NotFoundError: /usr/local/lib/python2.7/dist-packages/sonnet/python/ops/_resampler.so: undefined symbol: _ZN10tensorflow15shape_inference16InferenceContext15WithRankAtLeastENS0_11ShapeHandleEiPS2_

from sonnet.

Instead of augmenting the LD_LIBRARY_PATH: cd out of the sonnet source directory before running python.

from sonnet.

@normanheckscher: Thanks, cd out of the sonnet source directory instead of changing LD_LIBRARY_PATH works for me.

from sonnet.

@roman3017 thanks for looking into this. When you mention "using r1.1 branch of tensorflow", are you referring to the version linked here to obtain the headers (i.e. the submodule)? If so, I will merge your resampler_op.cc fix and make our submodule point to the r1.1 branch to make sure everyone can compile without an issue.

from sonnet.

Can't use with python3?
I got error with TFr1.1 and patch #6:

Python 3.5.2 (default, Apr  9 2017, 13:56:10)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sonnet as snt
>>> import tensorflow as tf
>>> snt.resampler(tf.constant([0.]), tf.constant([0.]))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'sonnet' has no attribute 'resampler'

from sonnet.

An alternate approach from upgrading to TFr1.1 is included in this merge request:

#7

I've included the compile error fix from @roman3017 in that, can tidy that up assuming it's accepted

from sonnet.

I have used the method @roman3017 mentioned and compiled and installed both sonnet and tensorflow.
However, when running the example:

snt.resampler(tf.constant([0.]), tf.constant([0.]))

It gives the same error as @roman3017 :

Traceback (most recent call last):
File "", line 1, in
File "sonnet/python/ops/resampler.py", line 62, in resampler
raise ImportError("_gen_resampler could not be imported.")
ImportError: _gen_resampler could not be imported.

from sonnet.

@Kongsea that initial test works for me. I've even run the sonnet/examples/rnn_shakespeare.py example without getting that error.

Ubuntu 16.04 LTS
Latest CUDA and cuDNN from Nvidia
GTX 1080
TensorFlow compiled and installed (prior to compiling and installing sonnet) from the head with GPU enabled.

from sonnet.

@roman3017 I am trying to investigate why you needed to reinstall TensorFlow as well. Ideally, a person that has installed TensorFlow via the pip package shouldn't need to reinstall the library. To have more information, what version of TensorFlow was installed in your system before you uninstalled it?

from sonnet.

@normanheckscher Since my previous tensorflow version is 0.12, I both built sonnet and tensorflow 1.1 from source. Now tensorflow works well, but sonnet cannot work and _gen_resampler could not be imported.
My Ubuntu is 14.04, CUDA is 8.1 and cuDNN 5.1. I am using a Titan X.

from sonnet.

@normanheckscher Now it works. Seven hours ago, it did not work, but now it works. Nothing has changed. So wired.

from sonnet.

@kosklain: Originally I had installed the latest master branch of TF from the source, which was not working with sonnet for me when I just configured the headers.

from sonnet.

@kosklain thanks, I'll try the sonnet compile later with the latest.

Regarding your statement "Ideally we want to not have to compile tensorflow" that's fine but I hope tensorflow and sonnet will continue to make it possible (and not too difficult) for people to compile their own. I want to run whichever Linux I want (I don't use Centos or Ubuntu) and to be able to install GPU libraries and other components in the locations I prefer. Precompiled stuff often makes assumptions that don't work for certain users or for the way they prefer to do things. Thanks!

from sonnet.

@edrozenberg yes, I agree with you. As mentioned, we would not want users to be forced to compile tensorflow, but by no means this implies we will force such users to use pre-compiled binaries, both things should just be available as options.

from sonnet.

from sonnet.

@japan4415
Hello,
I have the same problem as you. Solved?

Thank you so much !

from sonnet.