Comments (8)
what's the output of nvidia-smi
in your environment?
from aresdb.
Output of nvidia-smi:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.48 Driver Version: 390.48 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla V100-SXM2... On | 00000000:15:00.0 Off | 0 |
| N/A 35C P0 46W / 300W | 6MiB / 32510MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla V100-SXM2... On | 00000000:16:00.0 Off | 0 |
| N/A 35C P0 41W / 300W | 6MiB / 16160MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 2 Tesla V100-SXM2... On | 00000000:3A:00.0 Off | 0 |
| N/A 33C P0 44W / 300W | 6MiB / 32510MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 3 Tesla V100-SXM2... On | 00000000:3B:00.0 Off | 0 |
| N/A 36C P0 42W / 300W | 6MiB / 16160MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 4 Tesla V100-SXM2... On | 00000000:89:00.0 Off | 0 |
| N/A 34C P0 41W / 300W | 6MiB / 16160MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 5 Tesla V100-SXM2... On | 00000000:8A:00.0 Off | 0 |
| N/A 36C P0 43W / 300W | 6MiB / 16160MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 6 Tesla V100-SXM2... On | 00000000:B2:00.0 Off | 0 |
| N/A 34C P0 43W / 300W | 6MiB / 32510MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 7 Tesla V100-SXM2... On | 00000000:B3:00.0 Off | 0 |
| N/A 35C P0 42W / 300W | 6MiB / 16160MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 3413 G /usr/lib/xorg/Xorg 5MiB |
| 1 3413 G /usr/lib/xorg/Xorg 5MiB |
| 2 3413 G /usr/lib/xorg/Xorg 5MiB |
| 3 3413 G /usr/lib/xorg/Xorg 5MiB |
| 4 3413 G /usr/lib/xorg/Xorg 5MiB |
| 5 3413 G /usr/lib/xorg/Xorg 5MiB |
| 6 3413 G /usr/lib/xorg/Xorg 5MiB |
| 7 3413 G /usr/lib/xorg/Xorg 5MiB |
+-----------------------------------------------------------------------------+
from aresdb.
that's wired.. the error is during initializing. AresDB has not copied anything to device memory yet.
looks like you were running on bare metal (whithout docker) but I was not able to reproduce. (I ran the same make rule with same cuda version and driver version, and was able to start the server properly).
several things I would try:
- see if you can run any sample cuda app.
- maybe try after killing xorg processes. (although it shouldn't matter)
from aresdb.
I tried a number of CUDA samples, including bandwidthTest, deviceQuery, histogram, and more without error.
Linux version: Ubuntu 16.04.6 LTS
Could the trouble have to do with the Linux version, or is there a recommended version of nvidia and cuda to work with Ubuntu 16.04.6 LTS maybe?
from aresdb.
not likely. I tested in the same linux version.
from aresdb.
Still having trouble. Has anyone encountered this error from make test-cuda?
[----------] 4 tests from UnaryTransformTest
[ RUN ] UnaryTransformTest.CheckInt
Exception happend when doing UnaryTransform:parallel_for failed: invalid device function
terminate called after throwing an instance of 'thrust::system::system_error'
what(): parallel_for failed: invalid device function
Aborted (core dumped)
CMakeFiles/test-cuda.dir/build.make:60: recipe for target 'CMakeFiles/test-cuda' failed
make[3]: *** [CMakeFiles/test-cuda] Error 134
make[3]: Leaving directory '/nvme0n1/go2/src/github.com/uber/aresdb'
CMakeFiles/Makefile2:388: recipe for target 'CMakeFiles/test-cuda.dir/all' failed
make[2]: *** [CMakeFiles/test-cuda.dir/all] Error 2
make[2]: Leaving directory '/nvme0n1/go2/src/github.com/uber/aresdb'
CMakeFiles/Makefile2:395: recipe for target 'CMakeFiles/test-cuda.dir/rule' failed
make[1]: *** [CMakeFiles/test-cuda.dir/rule] Error 2
make[1]: Leaving directory '/nvme0n1/go2/src/github.com/uber/aresdb'
Makefile:262: recipe for target 'test-cuda' failed
make: *** [test-cuda] Error 2
from aresdb.
I just realized I didn’t mention this before, but I am running in GPU mode, so am running the cmake command ‘cmake -DQUERY_MODE=DEVICE’. I am considering the Docker implementation as well, but I saw the cmake command does not specify DQUERY_MODE. Does the Docker version run in GPU or CPU mode?
from aresdb.
I just realized I didn’t mention this before, but I am running in GPU mode, so am running the cmake command ‘cmake -DQUERY_MODE=DEVICE’. I am considering the Docker implementation as well, but I saw the cmake command does not specify DQUERY_MODE. Does the Docker version run in GPU or CPU mode?
QUERY_MODE=DEVICE
is the one you want to set with a GPU machine.
if QUERY_MODE
is missing the make file will set it base on whether the build machine has GPU card (here)
The docker version should be in GPU mode because it's on top of nvidiadocker.
from aresdb.
Related Issues (20)
- Ares Connector: Multiple table points to the same schema
- Update services.Service with placement.Placement in m3cluster HOT 1
- docker build error HOT 2
- upsertBatch need validate string length
- docker file build error HOT 1
- Useless field assignment in `query_handler.go` HOT 1
- Useless field assignment in `aql_nonaggr_batchexecutor.go`
- replace rdkafka with sarama to mitigate compiling issue HOT 1
- Simplify c++ binder to pass in data type as parameter which using pointer to pass data HOT 1
- Error in build docker 96% finished [Built target arrow_cuda_objlib] HOT 1
- Failed to use swagger in docker HOT 2
- CUDA 9.2? HOT 1
- Error populating query using the examples HOT 1
- kaungmyat182804.wordpress
- kaungmyat182804.wordpress
- AresDB on Arm/Arm64
- Java Support HOT 2
- Status of aresdb
- how should i get the production access approval?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aresdb.