Comments (6)
Hi,
May I ask what is your running environment (including hardware and software)? Under different environments, the performance of this model varies. If the environment is the same as what we describe in the Readme.md file, the current default settings should give you the model exactly the same as the pretrained model we released here.
from aggcn.
I have also uploaded the logs and the config.
If the running environment is the same as we described here, the output should be the same as in the logs.txt. The best model is the one we reported and released.
from aggcn.
Hi,
My python version is 3.6.5, pytorch is 1.1, CUDA is 10.0. I'm using GTX 1080Ti. I trained another 5 times, and the mean F1 is around 67.5% (+- 0.3%). I fully understand that the software and hardware will lead to different performance, but didn't expect so large difference.
Also, could you tell me the mean and std of F1 score in your experiments? It's important for measuring the stability of the model and a concrete comparison to other methods.
from aggcn.
Sorry for the late reply.
Yes, I do test the model under similar settings as yours. It seems that the loss is different from the first epoch (1.254588 v.s. 1.24539). These minor differences will start to accumulate, which eventually lead to a different model (around 67.5%). For now, we couldn't figure it out the reason behind this. For the model stats, I will update you later, since I am kind of occupied by the visa stuff...
For the mean and std of F1 score in my experiments, the stats is 68.2% +- 0.5%. Thank you for pointing out this issue! We deeply appreciate that.
Also, we will update this score on our paper, for a fair an concrete comparison to other methods.
from aggcn.
Hi,
I have run the training as well and get similar results as reported by @wzhouad :
Final Score:
Precision (micro): 70.780%
Recall (micro): 63.308%
F1 (micro): 66.836%
OS: openSUSE Leap 15.0.
GPUs: RTX 2080 Ti
cuda verison: 10
Python version: Python 3.6.8
Package Version
certifi 2019.6.16
cffi 1.12.3
mkl-fft 1.0.12
mkl-random 1.0.2
numpy 1.16.4
pip 19.1.1
pycparser 2.19
setuptools 41.0.1
torch 1.1.0
tqdm 4.32.2
wheel 0.33.4
from aggcn.
Hi @marchbnr ,
As stated in the Readme of this repo, we can't guarantee the performance of this repo when you run it under totally different settings (software and hardware). We also released the training log and pre-trained model.
For now, we might not able to find out the cause of this issue, since it involved too many variables (versions of GPU, CUDA, pytorch, etc,.)
from aggcn.
Related Issues (20)
- some error about:"RuntimeError: cuda runtime error (100) : " HOT 4
- About "M identical blocks" HOT 8
- Why the number of classes on the SemEval2010-Task 8 is only 10? HOT 2
- some question about your paper HOT 2
- 请问代码中的mlp output layer 是用来干嘛的 HOT 1
- RuntimeError: cuda runtime error (38) : no CUDA-capable device is detected at ..\src\THC\THCGeneral.cpp:70 HOT 2
- About replacing data sets HOT 9
- I found an error: train.py: error: argument --id: expected one argument
- RuntimeError: cuda runtime error (100) : no CUDA-capable device is detected at /tmp/pip-req-build-ufslq_a9/aten/src/THC/THCGeneral.cpp:50 HOT 1
- how to get standford_head and stanford_deprel for cross-sentence data HOT 3
- F1=0 HOT 1
- the Final Score HOT 2
- how to preprocess the dataset HOT 5
- 关于AGGCN模型细节的问题 HOT 1
- Why did you configure the first densely connected layer with GraphConvLayer? HOT 1
- What is the function of tensor "denom" ? HOT 3
- environment error
- How to test the n-ary relation extraction part of the experiment?
- 适用于中文数据集吗? HOT 3
- 关于模型中的M个block的问题
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aggcn.