Nauta compilation failure: Error 2 about nauta HOT 27 CLOSED

Hi Javier,
thanks for letting us know about the issue. Can you please tell me if you tried to build the tagged branch https://github.com/IntelAI/nauta/tree/v1.0.0-beta.0 or just develop branch https://github.com/IntelAI/nauta/tree/develop ?
Also if possible can you post full output from the build - I believe there was an error during building tensorflow at some point earlier.

from nauta.

Hello mgumowsk:

I have cloned using : #git clone --recursive https://github.com/IntelAI/nauta.git
and then I have run: #cd /nauta && make k8s-installer-build

make-k8s-installer-build.log.txt

from nauta.

Hi Javier,
in the provided log I can see an error about lack of free space:
Monday 18 February 2019 12:24:34 +0000 (0:00:00.349) 0:04:58.521 ******* fatal: [local]: FAILED! => {"ansible_job_id": "773893671130.9994", "attempts": 320, "changed": false, "finished": 1, "msg": "Error building 127.0.0.1:32768/nauta/rpm/kubernetes - code: None, message: Error processing tar file(exit status 1): write /root/rpmbuild/kubernetes/server/kubernetes-server-linux-amd64.tar.gz: no space left on device

Please verify that you have enough free disk space on the device, 20GB or more - the process of building docker images is very disk consuming.

from nauta.

Thank you, mgumowsk.
I have added additional 50 GB disk , where I have created a directory called nauta, and moved all files.
Now I don´t see space problem in the log but I still get the same error.
make-k8s-installer-build.log.txt

from nauta.

Hi Javier,
this time the error is different, we're investigating the issue and we'll let you know shortly about the solution.

from nauta.

Hi @fjmatos,
Probably something wrong with docker registry data was happened.
Please run make k8s-installer-clean which cleans temporary data and later try to build once again using make k8s-installer-build command.

from nauta.

Hi @ajoskowski
I have run , make k8s-installer-clean , and then make k8s-installer-build , but I get the same error.

ncluded: /nauta/tools/container-build/tasks/wait.yml for local

TASK [container-build : Wait for task shared/centos/rpm-packer] ***************************************
Wednesday 20 February 2019 17:22:51 +0000 (0:00:00.565) 0:20:24.637 ****
fatal: [local]: FAILED! => {"ansible_job_id": "138845617195.16433", "attempts": 1, "changed": false, "fmsg": "Error building 127.0.0.1:32772/nauta/shared/centos/rpm-packer - code: None, message: manifest fo772/nauta/shared/centos:1fe8ee8dfbb8871c149ec50114009b355ed639cd not found, logs: ['Step 1/11 : ARG BAS:7.4.1708', '\n', 'Step 2/11 : FROM ${BASE_IMAGE}', '\n']"}

I don´t understand why in the error los I see ".../nauta/shared/centos:..." , when my os is Ubuntu.

from nauta.

Javier,
You see centos, because this is a building process of docker images which are necessary for platform deployment. Currently we cannot reproduce your problem, but we are testing various scenarios of build run. When we find a reason of your problem, we will send you a tip.

One more question, did you follow the steps which are described here? Do you have at least 12 GB of RAM?

from nauta.

@ajoskowski
I have followed the guidelines in the documentation.
My kubernetes master node got 12 GB Ram, 2 GB in the worker nodes.
50 GB HD in the master, 10 GB HD in the worker nodes.
I´m running make in the master.
All nodes connect fine because I can deploy containers.
I have checked installation and I got all the docker packages.
So I think I meet all the requirements.

from nauta.

Hi @fjmatos,
Last time we implemented some changes in installer build process. Now it should be more stable and quicker. Please pull new changes from develop branch and try to run make k8s-installer-clean and later make k8s-installer-build commands. If you encounter on any problem, please give us a feedback with logs.

from nauta.

Hi @ajoskowski
I´m not sure if I´m using development branch, but this are the steps I have taken.

#git clone --recursive https://github.com/IntelAI/nauta.git
Cloning into 'nauta'...
remote: Enumerating objects: 13, done.
remote: Counting objects: 100% (13/13), done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 20630 (delta 5), reused 9 (delta 4), pack-reused 20617
Receiving objects: 100% (20630/20630), 31.20 MiB | 10.81 MiB/s, done.
Resolving deltas: 100% (11005/11005), done.
#make k8s-installer-clean
08813b84506e
#make k8s-installer-build

but I get the same error.

make-k8s-installer-build.log.txt

from nauta.

@fjmatos , you are using old version of code, I see that in logs you have included:

TASK [cache-registry-init : Start cache registry container] ***********************************************
Wednesday 27 February 2019  11:12:29 +0000 (0:00:00.048)       0:00:03.006 ****
changed: [local]

which is a part of removed code.

Please show me git log command output. Hash of your top commit should be 57dcdc7.

If you've other hash, please run the following commands:

git checkout develop
git pull
git reset --hard origin/develop

from nauta.

This is the output of the checkout:

#git checkout develop
Already on 'develop'
Your branch is up-to-date with 'origin/develop'.

git-log.txt

from nauta.

@fjmatos , you are not up-to-date with origin/develop.
Please run git pull && git reset --hard origin/develop.

from nauta.

@ajoskowski , I´m still getting some errors.
git-reset-output.txt
make-k8s-installer-build.log.txt

from nauta.

You encountered No space left on device once again.

from nauta.

I got a 50 GB disk partition for /nauta, so the process must create temporary files in / , where I have 2.5 GB free.
I´ll try somehow increase free space in /

from nauta.

After almost 4 hours of compilation, I get a new error.
I see in the log file: "fatal error: openssl/opensslv.h: No such file or directory"
This seems to be related with libssl-dev package, but I cannot find anything about it in the How_to_Build_Nauta guide.
These are ssl packages that I got installed in my Ubuntu:
libssl1.0.0
libgnutls-openssl27
openssl

I have installed libssl-dev and launched the compilation again.

make-k8s-installer-build.log.txt

from nauta.

Still getting errors

make-k8s-installer-build.log.txt

from nauta.

Hello!
Now you run into problem related with CPU flags.
It's from your logs:

The TensorFlow library was compiled to use AVX instructions, but these aren't available on your machine.

Can you share your cat /proc/cpuinfo output?
I assume you build nauta on some kind of VM where your CPU not supports required flags. You need CPU with AVX and SSE flags enabled. If you build on VM maybe your virtualization mechanism allows to set missing CPU flags?

from nauta.

Hello @mzylowski
Thank you for your information. In the requirements, I just checked that my VM CPU was compatible with sse sse2 instructions.
I´m looking into the bios of my host (dell R510) and it looks not be able to support AVX , so I´ll try to compile nauta in a VM placed in a host with this type of flags.

from nauta.

Hello @mzylowski
I have installed a VM with the right flags.
With 100 GB HD for /var/lib/docker , and 12 GB RAM, but I´m still getting some errors.

k8s_installer_build.log
cpuinfo.txt

from nauta.

@fjmatos , it looks like q connection problem in your infrastructure, because you can see in logs that "connection reset by peer" error was encountered. In my opinion you should try to run make k8s-installer-clean && make k8s-installer-build once again. You can check also proxy settings if problem still occurs.

from nauta.

I have run again, but now I get another kind of errors.

k8s_installer_build.log

from nauta.

@fjmatos are you behind a corporate proxy server? If yes, did you set a proxy settings? The simplest way to set proxy settings is exporting http_proxy, HTTP_PROXY, https_proxy, HTTPS_PROXY, no_proxy and NO_PROXY environment variables with valid addresses.

from nauta.

I'm closing that issue, as it was resolved off-line

from nauta.

@AdamTumi What was the solution?

from nauta.