Code Monkey home page Code Monkey logo

oc's Introduction

Usage

Install requirements

yum install -y epel-release
yum install -y ansible python-netaddr git

IMPORTANT: Make sure that you installed the epel version. The extras repository has ansible 2.4, which is incompatible with this playbook!

Clone this repo

git clone --recursive https://bitbucket.versatushpc.com.br/scm/opencattus/deployment.git
cd deployment
git submodule init
git submodule update

Runninng

  • Before begin make sure to have, at last, the list of mac of the nodes.

  • Edit nodes.yaml. The variable structures need stay the same. Here is a list of variables and their meanings

    variable required desc
    mac yes mac addess of the node
    ipmi_user no IPMI username, if ommited the default* one is used
    ipmi_password no IPMI password, if ommited the default* one is used
    slurm.Sockets only for Slurm Sockets slurm.conf entry
    slurm.CoresPerSocket only for Slurm CoresPerSocket slurm.conf entry
    slurm.ThreadsPerCore only for Slurm ThreadsPerCore slurm.conf entry

    * default here means the value present in answers-<date>.yaml.

  • Run ansible-playbook config.yaml to create configuration file

    • Answer the questions to configure the environment.
    • A new file named answers-.yaml will be created, where <date> is the date for today with the hour. This file contains all your answers to all previous questions, you will use this in the next steps.

  • Run ansible-playbook all.yaml -e @answer-file.yaml to run all phases in sequence

    Or run each of these steps in order:

  • Run ansible-playbook deploy.yaml -e @answer-<date>.yaml to install the base system

  • Add the image ansible-playbook add-image.yaml -e @answers-<date>.yaml

  • To configure infiniband ansible-playbook infiniband.yaml -e @answers-<date>.yaml

  • Install OpenHPC ansible-playbook basic-openhpc.yaml -e @answers-<date>.yaml

  • If using Slurm, install Slurm ansible-playbook slurm.yaml -e @answers-<date>.yaml

  • If using PBS, install PBS ansible-playbook pbs.yaml -e @answers-<date>.yaml

  • To install ganglia ansible-playbook ganglia.yaml -e @answers-<date>.yaml

  • To configure postfix ansible-playbook postfix.yaml -e @answers-<date>.yaml

  • To configure spack ansible-playbook spack.yaml -e @answers-<date>.yaml

  • To install TexLive ansible-playbook latex-support.yaml -e @answers-<date>.yaml

  • Finally to create nodes, run: ansible-deploy define-nodes.yaml -e @answers-<date>.yaml. The nodes will be defined in the order they appear in the file.

Running single role

ansible -m include_role -a 'name=<ROLE_NAME>' sms -e @answers-...yaml

Running single playbook

ansible -m include_tasks -a 'roles/xcat/tasks/install.yaml' sms -e @answers-...yaml

Questions

name prompt default choices help
cluster_name Hostname? headnode
domain_name Domain name? cluster.example.com
ipaadmin_password FreeIPA admin password
ipadm_password FreeIPA Directory Manager password FreeIPA Directory Manager password

It uses the same password as admin password entered before. This is used
for authenticating to LDAP.
nodes_prefix Node prefix? node
nodes_separator Node name separtor (optional)
nodes_padding Node name padding? 3
sms_eth_external External interface? eth0
sms_eth_internal Internal interface? eth1
sms_network Management network? 172.26.0.0/16 Management network is the primary network. Is the one by
which nodes boot and where services like DHCP, DNS, TFTP
talk.
sms_ip Head node management network internal IP?
sms_network_ip_start Management network first node IP address This is the Management Network first node IP address. Remaning nodes
will be addressed by subsequent address.

Expects an IP in the form 10.1.0.1
xcat_dhcp_dynamicrange xCAT DHCP Dynamic range? Dynamic range used for xCAT during node discovery.

Expects a IP range like 1.1.1.1-1.1.1.10
service_network Enter the service network 172.25.0.0/16 This is Service Network. Is the network used for BMC services and alike. It
expects to receive a CIDR notation address like 10.1.0.0/16. The network address
for the nodes are inferred from this network, starting from 10.1.0.2 up to
as many as nodes are defined on cluster.csv
service_network_ip_start Service network first node IP address This is BMC address of the first node. The reimaing nodes
will be named with subsequent IP address.

Expects an IP address like 10.2.0.1
ipmi_user What is the default IPMI username ADMIN
ipmi_password IPMI default password This can be overriden by node basis. The password
is setup for IPMI connection together with IPMI username
asked before
name prompt default choices help
--- --- --- --- ---
cluster_name Hostname? headnode
domain_name Domain name? cluster.example.com
ipaadmin_password FreeIPA admin password
ipadm_password FreeIPA Directory Manager password FreeIPA Directory Manager password

It uses the same password as admin password entered before. This is used
for authenticating to LDAP.
nodes_prefix Node prefix? node
nodes_separator Node name separtor (optional)
nodes_padding Node name padding? 3
sms_eth_external External interface? eth0
sms_eth_internal Internal interface? eth1
sms_network Management network? 172.26.0.0/16 Management network is the primary network. Is the one by
which nodes boot and where services like DHCP, DNS, TFTP
talk.
sms_ip Head node management network internal IP?
sms_network_ip_start Management network first node IP address This is the Management Network first node IP address. Remaning nodes
will be addressed by subsequent address.

Expects an IP in the form 10.1.0.1
xcat_dhcp_dynamicrange xCAT DHCP Dynamic range? Dynamic range used for xCAT during node discovery.

Expects a IP range like 1.1.1.1-1.1.1.10
service_network_enabled Configure service network? yes yes, no Network used for BMC/IPMI services
service_network Enter the service network 172.25.0.0/16 This is Service Network. Is the network used for BMC services and alike. It
expects to receive a CIDR notation address like 10.1.0.0/16. The network address
for the nodes are inferred from this network, starting from 10.1.0.2 up to
as many as nodes are defined on cluster.csv
service_network_ip_start Service network first node IP address This is BMC address of the first node. The reimaing nodes
will be named with subsequent IP address.

Expects an IP address like 10.2.0.1
ipmi_user What is the default IPMI username ADMIN
ipmi_password IPMI default password This can be overriden by node basis. The password
is setup for IPMI connection together with IPMI username
asked before
queue_system Choose a queue system none none, slurm, pbs
pbs_default_place PBS default place shared shared, scatter PBS default.place option
postfix_install Install postfix? yes yes, no Enable installation and configuration of postfix
postfix_profile Choose a profile for postfix installation? local relay, local, sasl Select the profile. There are 3 options
local) All messages are delivered locally only
relay) Messages are relayed to another MTA
sasl) Messages are relayed to another MTA, but it
uses user/password to connect to it
postfix_relay_server Type the MTA to relay to? The relay address, ex: relay-smtp.example.com
postfix_mynetworks_more Custom trusted networks? Comma separated list, localhost and cluster networks are already added by default
Ex: 192.168.123.0/24, 10.0.0.0/8
postfix_relay_server Relay server?
postfix_relay_port Relay port? 25
postfix_sasl_user SMTP user?
postfix_sasl_password SMTP password?
postfix_sasl_server SMTP server?
postfix_sasl_port SMTP port? 25
certificate_install Create certificate yes yes, no
ib_stack Infiniband stack none none, upstream, mellanox none => Do not install Infiniband at all
upstream => The main stream OFED stack, opensource and provided by default repos
mellanox => The Mellanox OFED stack
ib_network Application network 172.27.0.0/16 This is the network used by applications to talk, is a high speed network.

Expcets a network address like 172.27.0.0/16
ib_network_ip_start Infiniband network first node IP address
name prompt default choices help
--- --- --- --- ---
cluster_name Hostname? headnode
domain_name Domain name? cluster.example.com
ipaadmin_password FreeIPA admin password
ipadm_password FreeIPA Directory Manager password FreeIPA Directory Manager password

It uses the same password as admin password entered before. This is used
for authenticating to LDAP.
nodes_prefix Node prefix? n
nodes_separator Node name separtor (optional)
nodes_padding Node name padding? 2
sms_eth_external External interface? eth0
sms_eth_internal Internal interface? eth1
sms_network Management network? 172.26.0.0/16 Management network is the primary network. Is the one by
which nodes boot and where services like DHCP, DNS, TFTP
talk.
sms_ip Head node management network internal IP?
sms_network_ip_start Management network first node IP address This is the Management Network first node IP address. Remaning nodes
will be addressed by subsequent address.

Expects an IP in the form 10.1.0.1
xcat_dhcp_dynamicrange xCAT DHCP Dynamic range? Dynamic range used for xCAT during node discovery.

Expects a IP range like 1.1.1.1-1.1.1.10
service_network_enabled Configure service network? yes yes, no Network used for BMC/IPMI services
service_network Enter the service network 172.25.0.0/16 This is Service Network. Is the network used for BMC services and alike. It
expects to receive a CIDR notation address like 10.1.0.0/16. The network address
for the nodes are inferred from this network, starting from 10.1.0.2 up to
as many as nodes are defined on cluster.csv
service_network_ip_start Service network first node IP address This is BMC address of the first node. The reimaing nodes
will be named with subsequent IP address.

Expects an IP address like 10.2.0.1
ipmi_user What is the default IPMI username ADMIN
ipmi_password IPMI default password This can be overriden by node basis. The password
is setup for IPMI connection together with IPMI username
asked before
queue_system Choose a queue system none none, slurm, pbs
pbs_default_place PBS default place shared shared, scatter PBS default.place option
slurm_partition_name Slurm default partition name batch The Slurm partition name
postfix_install Install postfix? yes yes, no Enable installation and configuration of postfix
postfix_profile Choose a profile for postfix installation? local relay, local, sasl Select the profile. There are 3 options
local) All messages are delivered locally only
relay) Messages are relayed to another MTA
sasl) Messages are relayed to another MTA, but it
uses user/password to connect to it
postfix_relay_server Type the MTA to relay to? The relay address, ex: relay-smtp.example.com
postfix_mynetworks_more Custom trusted networks? Comma separated list, localhost and cluster networks are already added by default
Ex: 192.168.123.0/24, 10.0.0.0/8
postfix_relay_server Relay server?
postfix_relay_port Relay port? 25
postfix_sasl_user SMTP user?
postfix_sasl_password SMTP password?
postfix_sasl_server SMTP server?
postfix_sasl_port SMTP port? 25
certificate_install Create certificate yes yes, no
ib_stack Infiniband stack none none, upstream, mellanox none => Do not install Infiniband at all
upstream => The main stream OFED stack, opensource and provided by default repos
mellanox => The Mellanox OFED stack
ib_network Application network 172.27.0.0/16 This is the network used by applications to talk, is a high speed network.

Expcets a network address like 172.27.0.0/16
ib_network_ip_start Infiniband network first node IP address
timezone Select timezone America/Sao_Paulo
name prompt default choices help
--- --- --- --- ---
cluster_name Hostname? headnode
domain_name Domain name? cluster.example.com
ipaadmin_password FreeIPA admin password
ipadm_password FreeIPA Directory Manager password FreeIPA Directory Manager password

It uses the same password as admin password entered before. This is used
for authenticating to LDAP.
nodes_prefix Node prefix? n
nodes_separator Node name separtor (optional)
nodes_padding Node name padding? 2
sms_eth_external External interface? eth0
sms_eth_internal Internal interface? eth1
sms_network Management network? 172.26.0.0/16 Management network is the primary network. Is the one by
which nodes boot and where services like DHCP, DNS, TFTP
talk.
sms_ip Head node management network internal IP?
sms_network_ip_start Management network first node IP address This is the Management Network first node IP address. Remaning nodes
will be addressed by subsequent address.

Expects an IP in the form 10.1.0.1
xcat_dhcp_dynamicrange xCAT DHCP Dynamic range? Dynamic range used for xCAT during node discovery.

Expects a IP range like 1.1.1.1-1.1.1.10
service_network_enabled Configure service network? yes yes, no Network used for BMC/IPMI services
service_network Enter the service network 172.25.0.0/16 This is Service Network. Is the network used for BMC services and alike. It
expects to receive a CIDR notation address like 10.1.0.0/16. The network address
for the nodes are inferred from this network, starting from 10.1.0.2 up to
as many as nodes are defined on cluster.csv
service_network_ip_start Service network first node IP address This is BMC address of the first node. The reimaing nodes
will be named with subsequent IP address.

Expects an IP address like 10.2.0.1
ipmi_user What is the default IPMI username ADMIN
ipmi_password IPMI default password This can be overriden by node basis. The password
is setup for IPMI connection together with IPMI username
asked before
queue_system Choose a queue system none none, slurm, pbs
pbs_default_place PBS default place shared shared, scatter PBS default.place option
slurm_partition_name Slurm default partition name batch The Slurm partition name
postfix_install Install postfix? yes yes, no Enable installation and configuration of postfix
postfix_profile Choose a profile for postfix installation? local relay, local, sasl Select the profile. There are 3 options
local) All messages are delivered locally only
relay) Messages are relayed to another MTA
sasl) Messages are relayed to another MTA, but it
uses user/password to connect to it
postfix_relay_server Type the MTA to relay to? The relay address, ex: relay-smtp.example.com
postfix_mynetworks_more Custom trusted networks? Comma separated list, localhost and cluster networks are already added by default
Ex: 192.168.123.0/24, 10.0.0.0/8
postfix_relay_server Relay server?
postfix_relay_port Relay port? 25
postfix_sasl_user SMTP user?
postfix_sasl_password SMTP password?
postfix_sasl_server SMTP server?
postfix_sasl_port SMTP port? 25
certificate_install Create certificate yes yes, no
ib_stack Infiniband stack none none, upstream, mellanox none => Do not install Infiniband at all
upstream => The main stream OFED stack, opensource and provided by default repos
mellanox => The Mellanox OFED stack
ib_network Application network 172.27.0.0/16 This is the network used by applications to talk, is a high speed network.

Expcets a network address like 172.27.0.0/16
ib_network_ip_start Infiniband network first node IP address
timezone Select timezone America/Sao_Paulo
name prompt default choices help
--- --- --- --- ---
cluster_name Hostname? headnode
domain_name Domain name? cluster.example.com
ipaadmin_password FreeIPA admin password
ipadm_password FreeIPA Directory Manager password FreeIPA Directory Manager password

It uses the same password as admin password entered before. This is used
for authenticating to LDAP.
nodes_prefix Node prefix? n
nodes_separator Node name separtor (optional)
nodes_padding Node name padding? 2
sms_eth_external External interface? eth0
sms_eth_internal Internal interface? eth1
sms_network Management network? 172.26.0.0/16 Management network is the primary network. Is the one by
which nodes boot and where services like DHCP, DNS, TFTP
talk.
sms_ip Head node management network internal IP?
sms_network_ip_start Management network first node IP address This is the Management Network first node IP address. Remaning nodes
will be addressed by subsequent address.

Expects an IP in the form 10.1.0.1
xcat_dhcp_dynamicrange xCAT DHCP Dynamic range? Dynamic range used for xCAT during node discovery.

Expects a IP range like 1.1.1.1-1.1.1.10
service_network_enabled Configure service network? yes yes, no Network used for BMC/IPMI services
service_network Enter the service network 172.25.0.0/16 This is Service Network. Is the network used for BMC services and alike. It
expects to receive a CIDR notation address like 10.1.0.0/16. The network address
for the nodes are inferred from this network, starting from 10.1.0.2 up to
as many as nodes are defined on cluster.csv
service_network_ip_start Service network first node IP address This is BMC address of the first node. The reimaing nodes
will be named with subsequent IP address.

Expects an IP address like 10.2.0.1
ipmi_user What is the default IPMI username ADMIN
ipmi_password IPMI default password This can be overriden by node basis. The password
is setup for IPMI connection together with IPMI username
asked before
queue_system Choose a queue system none none, slurm, pbs
pbs_default_place PBS default place shared shared, scatter PBS default.place option
slurm_partition_name Slurm default partition name batch The Slurm partition name
postfix_install Install postfix? yes yes, no Enable installation and configuration of postfix
postfix_profile Choose a profile for postfix installation? local relay, local, sasl Select the profile. There are 3 options
local) All messages are delivered locally only
relay) Messages are relayed to another MTA
sasl) Messages are relayed to another MTA, but it
uses user/password to connect to it
postfix_relay_server Type the MTA to relay to? The relay address, ex: relay-smtp.example.com
postfix_mynetworks_more Custom trusted networks? Comma separated list, localhost and cluster networks are already added by default
Ex: 192.168.123.0/24, 10.0.0.0/8
postfix_relay_server Relay server?
postfix_relay_port Relay port? 25
postfix_sasl_user SMTP user?
postfix_sasl_password SMTP password?
postfix_sasl_server SMTP server?
postfix_sasl_port SMTP port? 25
certificate_install Create certificate yes yes, no
ib_stack Infiniband stack none none, upstream, mellanox none => Do not install Infiniband at all
upstream => The main stream OFED stack, opensource and provided by default repos
mellanox => The Mellanox OFED stack
ib_network Application network 172.27.0.0/16 This is the network used by applications to talk, is a high speed network.

Expcets a network address like 172.27.0.0/16
ib_network_ip_start Infiniband network first node IP address
timezone Select timezone America/Sao_Paulo
name prompt default choices help
--- --- --- --- ---
cluster_name Hostname? headnode
domain_name Domain name? cluster.example.com
ipaadmin_password FreeIPA admin password
ipadm_password FreeIPA Directory Manager password FreeIPA Directory Manager password

It uses the same password as admin password entered before. This is used
for authenticating to LDAP.
nodes_prefix Node prefix? n
nodes_separator Node name separtor (optional)
nodes_padding Node name padding? 2
sms_eth_external External interface? eth0
sms_eth_internal Internal interface? eth1
sms_network Management network? 172.26.0.0/16 Management network is the primary network. Is the one by
which nodes boot and where services like DHCP, DNS, TFTP
talk.
sms_ip Head node management network internal IP?
sms_network_ip_start Management network first node IP address This is the Management Network first node IP address. Remaning nodes
will be addressed by subsequent address.

Expects an IP in the form 10.1.0.1
xcat_dhcp_dynamicrange xCAT DHCP Dynamic range? Dynamic range used for xCAT during node discovery.

Expects a IP range like 1.1.1.1-1.1.1.10
service_network_enabled Configure service network? yes yes, no Network used for BMC/IPMI services
service_network Enter the service network 172.25.0.0/16 This is Service Network. Is the network used for BMC services and alike. It
expects to receive a CIDR notation address like 10.1.0.0/16. The network address
for the nodes are inferred from this network, starting from 10.1.0.2 up to
as many as nodes are defined on cluster.csv
service_network_ip_start Service network first node IP address This is BMC address of the first node. The reimaing nodes
will be named with subsequent IP address.

Expects an IP address like 10.2.0.1
ipmi_user What is the default IPMI username ADMIN
ipmi_password IPMI default password This can be overriden by node basis. The password
is setup for IPMI connection together with IPMI username
asked before
queue_system Choose a queue system none none, slurm, pbs
pbs_default_place PBS default place shared shared, scatter PBS default.place option
slurm_partition_name Slurm default partition name batch The Slurm partition name
postfix_install Install postfix? yes yes, no Enable installation and configuration of postfix
postfix_profile Choose a profile for postfix installation? local relay, local, sasl Select the profile. There are 3 options
local) All messages are delivered locally only
relay) Messages are relayed to another MTA
sasl) Messages are relayed to another MTA, but it
uses user/password to connect to it
postfix_relay_server Type the MTA to relay to? The relay address, ex: relay-smtp.example.com
postfix_mynetworks_more Custom trusted networks? Comma separated list, localhost and cluster networks are already added by default
Ex: 192.168.123.0/24, 10.0.0.0/8
postfix_relay_server Relay server?
postfix_relay_port Relay port? 25
postfix_sasl_user SMTP user?
postfix_sasl_password SMTP password?
postfix_sasl_server SMTP server?
postfix_sasl_port SMTP port? 25
certificate_install Create certificate yes yes, no
ib_stack Infiniband stack none none, upstream, mellanox none => Do not install Infiniband at all
upstream => The main stream OFED stack, opensource and provided by default repos
mellanox => The Mellanox OFED stack
ib_network Application network 172.27.0.0/16 This is the network used by applications to talk, is a high speed network.

Expcets a network address like 172.27.0.0/16
ib_network_ip_start Infiniband network first node IP address
timezone Select timezone America/Sao_Paulo
name prompt default choices help
--- --- --- --- ---
cluster_name Hostname? headnode
domain_name Domain name? cluster.example.com
ipaadmin_password FreeIPA admin password
ipadm_password FreeIPA Directory Manager password FreeIPA Directory Manager password

It uses the same password as admin password entered before. This is used
for authenticating to LDAP.
nodes_prefix Node prefix? n
nodes_separator Node name separtor (optional)
nodes_padding Node name padding? 2
sms_eth_external External interface? eth0
sms_eth_internal Internal interface? eth1
sms_network Management network? 172.26.0.0/16 Management network is the primary network. Is the one by
which nodes boot and where services like DHCP, DNS, TFTP
talk.
sms_ip Head node management network internal IP?
sms_network_ip_start Management network first node IP address This is the Management Network first node IP address. Remaning nodes
will be addressed by subsequent address.

Expects an IP in the form 10.1.0.1
xcat_dhcp_dynamicrange xCAT DHCP Dynamic range? Dynamic range used for xCAT during node discovery.

Expects a IP range like 1.1.1.1-1.1.1.10
service_network_enabled Configure service network? yes yes, no Network used for BMC/IPMI services
service_network Enter the service network 172.25.0.0/16 This is Service Network. Is the network used for BMC services and alike. It
expects to receive a CIDR notation address like 10.1.0.0/16. The network address
for the nodes are inferred from this network, starting from 10.1.0.2 up to
as many as nodes are defined on cluster.csv
service_network_ip_start Service network first node IP address This is BMC address of the first node. The reimaing nodes
will be named with subsequent IP address.

Expects an IP address like 10.2.0.1
ipmi_user What is the default IPMI username ADMIN
ipmi_password IPMI default password This can be overriden by node basis. The password
is setup for IPMI connection together with IPMI username
asked before
queue_system Choose a queue system none none, slurm, pbs
pbs_default_place PBS default place shared shared, scatter PBS default.place option
slurm_partition_name Slurm default partition name batch The Slurm partition name
postfix_install Install postfix? yes yes, no Enable installation and configuration of postfix
postfix_profile Choose a profile for postfix installation? local relay, local, sasl Select the profile. There are 3 options
local) All messages are delivered locally only
relay) Messages are relayed to another MTA
sasl) Messages are relayed to another MTA, but it
uses user/password to connect to it
postfix_relay_server Type the MTA to relay to? The relay address, ex: relay-smtp.example.com
postfix_mynetworks_more Custom trusted networks? Comma separated list, localhost and cluster networks are already added by default
Ex: 192.168.123.0/24, 10.0.0.0/8
postfix_relay_server Relay server?
postfix_relay_port Relay port? 25
postfix_sasl_user SMTP user?
postfix_sasl_password SMTP password?
postfix_sasl_server SMTP server?
postfix_sasl_port SMTP port? 25
certificate_install Create certificate yes yes, no
ib_stack Infiniband stack none none, upstream, mellanox none => Do not install Infiniband at all
upstream => The main stream OFED stack, opensource and provided by default repos
mellanox => The Mellanox OFED stack
ib_network Application network 172.27.0.0/16 This is the network used by applications to talk, is a high speed network.

Expcets a network address like 172.27.0.0/16
ib_network_ip_start Infiniband network first node IP address
timezone Select timezone America/Sao_Paulo

oc's People

Contributors

dhilst avatar viniciusferrao avatar

Watchers

Ernedin Zajko avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.