Comments (3)
This is my code
from disent.metrics import metric_dci, metric_mig
import torch
from torch.utils.data import DataLoader
from disent.metrics import metric_dci, metric_mig, metric_factor_vae
import torch
from torch.utils.data import DataLoader
from disent.dataset import DisentDataset
from model.β_vae import BetaVAE_H
from disent.dataset.data import DSpritesData
from disent.dataset.transform import ToImgTensorF32
def train():
device = torch.device("cuda:0" if torch.cuda.is_available() else 'cpu')
data = DSpritesData()
dataset = DisentDataset(data, transform=ToImgTensorF32(), augment=None)
dataloader = DataLoader(dataset=dataset, batch_size=32, shuffle=True)
checkpoint = torch.load('./checkpoints/beta_vae50.pt', map_location=device)
model = BetaVAE_H().to(device)
model.load_state_dict(checkpoint['model'])
model.eval()
# we cannot guarantee which device the representation is on
get_repr = lambda x: model.mlp_encoder(x.to(device))
# evaluate
return {
**metric_dci(dataset, get_repr, num_train=20, boost_mode='sklearn'),
**metric_mig(dataset, get_repr, num_train=20),
**metric_factor_vae(dataset, get_repr, num_train=20),
}
a_results = train()
print('beta=4: ', a_results)
from disent.
Hi there, and thank you!
Unfortunately docs for this are sparse. I understand this is not the most ideal, would gladly accept PRs to fix this.
However, for context, the mig, dci and factor vae scores are largely based on those from https://github.com/google-research/disentanglement_lib (Default values should be similar) From what I remember without looking at the code num_train
and batch_size
affect the sample size of underlying data that is used to compute the metrics. Too little data and the metrics will be inaccurate, too much and processing time will be too much. Often for metrics during training I would lower these values and then do a final larger compute at the end with the default values.
from disent.
hydra config experiments metrics:
https://github.com/nmichlo/disent/tree/8f061a87076adeae8d6e5b0fa984b660cd40e026/experiment/config/metrics
actual code that selects these:
Lines 208 to 209 in 8f061a8
metric wrapper:
- see
compute
andcompute_fast
disent/disent/metrics/utils.py
Lines 52 to 109 in 8f061a8
fast version kwargs:
Line 48 in 8f061a8
Line 45 in 8f061a8
disent/disent/metrics/_factor_vae.py
Line 47 in 8f061a8
NOTE: kwargs for fast versions were arbitrarily chosen. The standard versions should follow kwargs from disentanglement_lib
.
NOTE: batch_size
is like batch size from dataset loaders, the model is often used within these metrics and is run on the GPU if possible.
from disent.
Related Issues (20)
- Windows, Macos & GPU Tests
- Standardise Loss Reduction Modes HOT 1
- Adopt a Lightning-Flash Style API for Frameworks HOT 1
- [Q]: What's the native way to split datasets into train, validation and test? HOT 4
- [Q]: Command to run with defaults? HOT 4
- [FEATURE]: Experiment Config Override Support & Custom Code Registry HOT 3
- [FEATURE]: Allow Metrics To Directly Accept Frameworks & Datasets HOT 1
- [DOCS]: Add Experiment Examples HOT 1
- [BUG]: Investigate Schedule Interaction With Validation & Test Data
- [FEATURE]: Model Saving and Checkpointing HOT 11
- [FEATURE]: Add to Registry from Configs HOT 1
- [FEATURE]: Simplifying your configs using hydra-zen HOT 2
- [BUG]: Automatic downloads not working for MPI3D and dSprites HOT 3
- Averaging in AdaGVAE HOT 2
- [FEATURE]: Add DMS metric HOT 5
- [FEATURE]: Add IRS Metric HOT 2
- [FEATURE]: wandb model checkpoint support
- visualisation with trained model HOT 3
- [BUG]: Frameworks not loading as LightningModule when using trainer.fit
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from disent.