- Install Homebrew
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
- Install git
brew install git
- Clone this repository
mkdir -p ~/Projects
cd ~/Projects
git clone https://github.com/hinson/DLLecture
Windows 10 (Ubuntu subsystem)
- Install git
apt install git
- Clone this repository
mkdir -p ~/Projects
cd ~/Projects
git clone https://github.com/hinson/DLLecture
{ }
means a placeholder in the following.
- Create a key file for ssh
ssh-keygen
- Add a public key to the remote server's authorized_keys file
brew install ssh-copy-id
ssh-copy-id {user name}@{remote IP}
Windows 10 (Ubuntu subsystem)
Use the subsystem environment to invoke linux commands in the following.
ssh-keygen
ssh-copy-id {username}@{remote IP}
ssh -l{user name} {remote IP}
If you want to change the password, use
passwd
brew cask install osxfuse
brew install sshfs
mkdir -p ~/Mounts/{mount point}
sshfs {username}@{remote IP}:/home/{username} ~/Mounts/{mount point}
When you want to unmount a driver,
diskutil umount ~/Mounts/{mount point}
Install sshfs for Windows by following this.
Notice: The storage of /home/*
in the remote server is shared by all users at this time. If you want to handle big data files (eg. >=1GB ), please consider to additionally mount {data}
volumes for your usage. Let's take this as an exercise.
-
To easily switch multiple versions of Python (2 or 3)
-
To install required libraries for your projects without system privilege
-
Reference: https://github.com/pyenv/pyenv
curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bash_profile
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bash_profile
echo 'eval "$(pyenv init -)"' >> ~/.bash_profile
echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.bash_profile
source ~/.bash_profile
brew install pyenv pyenv-virtualenv
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bash_profile
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bash_profile
echo 'if command -v pyenv 1>/dev/null 2>&1; ' \
'then eval "$(pyenv init -)"; fi' >> ~/.bash_profile
echo 'if which pyenv-virtualenv-init > /dev/null; ' \
'then eval "$(pyenv virtualenv-init -)"; fi' >> ~/.bash_profile
exec "$SHELL"
- To easily manage python libraries for data science and machine learning
- Install Anaconda 3 to adapt to different environments by various projects.
pyenv install anaconda3-5.1.0
cd ~/Projects/DLLecture
pyenv local anaconda3-5.1.0
conda create -n DLLecture --clone root
conda activate DLLecture
If you reopen the terminal, only the following commands need.
cd ~/Projects/DLLecture
conda activate DLLecture
Use v1.15
conda install tensorflow==1.15
conda install tensorflow-gpu==1.15
pip install tensorboardX
conda install pytorch torchvision ignite cudatoolkit=10.1 -c pytorch
It may take a while to download...
- A good interactive notebook environment for research
- Open a python shell
cd ~/Projects/DLLecture
python
- Generate pass code
from notebook.auth import passwd; passwd()
-
Then copy the pass code
type:salt:hashed-password
after inputing and verifying your password. -
Type
exit()
to quit the shell.
jupyter notebook --generate-config
vim ~/.jupyter/jupyter_notebook_config.py
- Three modes in
vim
: Command, Input, Line. The default mode is Command. - In Command mode, type
/
and item string to search the item invim
. - Type
i
to enter Input mode. you can modify the contents in Input mode. - Press
esc
back to Command mode anytime. - In Command mode, type
:wq
to save your change and quit.
These items need to be set. To set new value, delete the #
in front of the items firstly.
c.NotebookApp.ip = 'localhost' # '{remote IP}' for DL-box
c.NotebookApp.password = u'{paste the above pass code}'
c.NotebookApp.port = {your port} # 8888 for your PC or unused one in 1024~65535 for DL-box
c.NotebookApp.open_browser = False
jupyter notebook
Then openhttp://localhost:8888/
in the browser and login with your password.
byobu new -s jupyter
jupyter notebook
Press F6
to detach from the session where the program will keep running even if you logout.
If you want to enter a session again, use byobu attach -t {session name}
or just run byobu
to select one. For more details about byobu
, see http://byobu.co.
Now, openhttp://{remote IP}:{your port}/
in the browser and login with your password.
- Install kernel to let the virtual environment work in Jupyter.
cd ~/Projects/DLLecture
python -m ipykernel install --user --name DLLecture
Open a new terminal.
cd ~/Projects/DLLecture
tensorboard --logdir ./runs
Then openhttp://localhost:6006/
in the browser.
byobu new -s tensorboard
cd ~/Projects/DLLecture
tensorboard --logdir ./runs --port {your another port}
Press F6
to detach from the session.
Then openhttp://{remote IP}:{your another port}/
in the browser.
-
The home of Jupyter is the location where
jupyter
was executed, i.e.,~/Projects/DLLecture
in this example. -
Click the
New
in the right top coner to create a new folder, then check the folder and rename it toruns
for tensorboard's data loading. -
Click
New
to create a newDLLecture
notebook. -
Copy the following code (edited from this demo) into the first shell, then press
shift
+return
to run it,
import torch
import torchvision
import numpy as np
import torchvision.models as models
from torchvision import datasets
from tensorboardX import SummaryWriter
logdir = './runs'
resnet18 = models.resnet18(False)
writer = SummaryWriter(logdir)
sample_rate = 44100
freqs = [262, 294, 330, 349, 392, 440, 440, 440, 440, 440, 440]
for n_iter in range(100):
dummy_s1 = torch.rand(1)
dummy_s2 = torch.rand(1)
# data grouping by `slash`
writer.add_scalar('data/scalar1', dummy_s1[0], n_iter)
writer.add_scalar('data/scalar2', dummy_s2[0], n_iter)
writer.add_scalars('data/scalar_group', {'xsinx': n_iter * np.sin(n_iter),
'xcosx': n_iter * np.cos(n_iter),
'arctanx': np.arctan(n_iter)}, n_iter)
dummy_img = torch.rand(32, 3, 64, 64) # output from network
if n_iter % 10 == 0:
x = torchvision.utils.make_grid(dummy_img, normalize=True, scale_each=True)
writer.add_image('Image', x, n_iter)
dummy_audio = torch.zeros(sample_rate * 2)
for i in range(x.size(0)):
# amplitude of sound should in [-1, 1]
dummy_audio[i] = np.cos(freqs[n_iter // 10] * np.pi * float(i) / float(sample_rate))
writer.add_audio('myAudio', dummy_audio, n_iter, sample_rate=sample_rate)
writer.add_text('Text', 'text logged at step:' + str(n_iter), n_iter)
for name, param in resnet18.named_parameters():
writer.add_histogram(name, param.clone().cpu().data.numpy(), n_iter)
# needs tensorboard 0.4RC or later
writer.add_pr_curve('xoxo', np.random.randint(2, size=100), np.random.rand(100), n_iter)
writer.add_graph(resnet18, torch.randn(1, 3, 224, 224))
dataset = datasets.MNIST('mnist', train=False, download=True)
images = dataset.test_data[:100].float()
label = dataset.test_labels[:100]
features = images.view(100, 784)
writer.add_embedding(features, metadata=label, label_img=images.unsqueeze(1))
# export scalar data to JSON for external processing
writer.export_scalars_to_json("./all_scalars.json")
writer.close()
Wait until it finishes, and see the result on TensorBoard.