Select

should use is null, not = null because NULL is a special marker to indicate a data value does not exist, not a data value. So that cannot use = to check for equality.

Output NULL

To output null as a value, should use another select clause as a shell, for example:

select (
	select distinct salary as SecondHighestSalary
	from Employee
	order by salary desc
	limit 1 offset 1
) as SecondHighestSalary

The clause will output the second higher salary but cannot output "NULL" if the database is null. The shell clause will transfer them to the null.

Additional, to assign a variable, using = is Ok.

left and right join are outer join.

What happens after left joins? Example: [[183. Customers Who Never Order]]

Create table If Not Exists Customers (id int, name varchar(255))
Create table If Not Exists Orders (id int, customerId int)
Truncate table Customers
insert into Customers (id, name) values ('1', 'Joe')
insert into Customers (id, name) values ('2', 'Henry')
insert into Customers (id, name) values ('3', 'Sam')
insert into Customers (id, name) values ('4', 'Max')
Truncate table Orders
insert into Orders (id, customerId) values ('1', '3')
insert into Orders (id, customerId) values ('2', '1')

# if run
SELECT *
FROM Customers
LEFT JOIN Orders
ON Customers.id = Orders.customerId

# result are
{"headers": ["id", "name", "id", "customerId"], "values": [[1, "Joe", 2, 1], [2, "Henry", null, null], [3, "Sam", 1, 3], [4, "Max", null, null]]}

Without ON

In the left or right join, ON is necessary. However, in MySQL and join, ON is optional. In this case, using join without on will produce a cross join.

Condition in ON or WHERE

In the inner join, write conditions in on or where has no differences. However, if using a left join or right join, there is a difference. For example:

SELECT *
FROM dbo.Customers AS CUS 
LEFT JOIN dbo.Orders AS ORD 
ON CUS.CustomerID = ORD.CustomerID
WHERE ORD.OrderDate >'20090515'

SELECT *
FROM dbo.Customers AS CUS 
LEFT JOIN dbo.Orders AS ORD 
ON CUS.CustomerID = ORD.CustomerID
AND ORD.OrderDate >'20090515'

ON is a filter when producing the temp table, in the left join case it will return all left items and ignore the date condition in the above example. Different from ON, WHERE will execute after the temp table is produced. Thus it will delete all items that do not meet the conditions.

combine two varchar

concat('a', 'b')
in group: group_concate(distinct name order by name separator ',')

upper case and lower case

upper('a') and lower('a')

select char in varchar

1. select n char from left or right side left(name, n) or right(name, n)
2. select all chars from nTH to mTH from the left substr(name, n, m)

length

length('aaaa')

agg functions

count(name)
in group also could use count(distinct name)

ROUND numbers

rounds a number to a specified number of decimal places
round(num, 2) or round(235.415, -1) = 240.000

It is a good code-writing habit for code understanding and development.

Why

Over time, code evolves, whereas code comments go stale.

& Document

Self-documenting code does not obviate the need for documentation. Consider README files and Swagger docs. We still need those. Also, sometimes it might still make sense to leave code comments or drop a link to a confluence doc in the code in the code.

Example

Version B is a self-document code.

Version A:

// The cellular range from the tower is 5 km.
// Let's assume there will be cell coverage in a circular area 
// where the tower is at the center. A simple model.
// Taking pi to the nearest hundredth, the coverage area will
// then be calculated as follows.
let x = 5.0
let y = 3.14 * pow(x, 2)

Version B:

let cellularRangeFromTowerKm = 5.0
let coverageAreaKm = circleArea(radius: cellularRangeFromTowerKm)

function circleArea(radius: Double) -> Double {
    let pi = 3.14
    let area = pi * pow(x, 2)
    return area
}

To ensure information security, we need the hash function to encrypt them.

Type

Three types of hash functions:

• SHA1: sha-1 will generate 160-bit (20-byte) 40 int hash.
• SHA2: 256, 224, and so on...
• SHA3: Subset of the broader family of algorithms called Keccak. Not included in this course.

Salted

a password hashing technique that is used to secure passwords. In a salted hash, a unique random value, known as a "salt," is generated for each password and is stored along with the hashed password. This technology will make the password hard to be cracked especially for easy password. However, it will not 100% ensure the security.

Generative Deep Learning

Introduction to Generative Adversarial Networks (GANs) (对抗生成网络)

Combined by a forger network and an expert network.

• Generator network
  • Input: random vector (a random point at a latent space)
  • Output: synthetic image
• Discriminator Network
  • Input: image (real or synthetic)
  • Output: predict where this image comes from.

What is a latent space?

A vector space that cannot be observed.

Difference

The GANS different forms of other training setting up, the optimization minimum is not fixed. With the GANS, every step tries to take down the hill changes the entire landscape. There exists an equilibrium balance between two processes or forces, not a minimum optimization process.

Also, in this case, GANS hard to train.

Schematic Implement

This is a deep convolutional GANS (DCGANs).

1. generator maps vectors (latent_dim) to shape of image (32, 32, 3).
2. discriminator maps image shape (32, 32, 3) to a binary 'real' probability.
3. gan chains the generator and discriminator together. gan(x) = discriminator(generator(x)).
4. train the discriminator by real and fake images and labels.
5. train the generator to fool the discriminator. Move the weight of the generator that makes discriminator classify generated image as a real image.

Tricks

1. tanh as the last activation in the generator.
2. sample point the latent space using a normal distribution.
3. Stochasticity improves robustness. GANs may be stuck in any sort of way because of the resulting balanced equilibrium.
   1. dropout in discriminator (and generator).
   2. adding random noise to labels of the discriminator.
4. sparse gradient hinder GANs training different from other deep learning technology. Here are two methods to decrease:
   1. change maxpooling to stried convolution for downsampling.
   2. change ReLu to LeakyReLu which relax sparsity constants by allowing small negative activation value.
5. To fix the checkerboard artifact caused by unequal coverage of the pixel space in the generator, the stridden Conv2DTranpose or Conv2D in both generator and discriminator.
6. When adversarial loss increases and discriminative loss tends to 0 (discriminator end up dominating the generator), reduce the discriminator learning rate, and increase the dropout level of the discriminator.

The Generator

# import
import keras
from kerasimport layers
import numpy as np

# input shape
latent_dim = 32
height = 32
width = 32
channels = 3

# module
generator_input = keras.Input(shape=(latent_dim, ))
# transform the input into a 16*16 128 channel feature map
x = layers.Dense(128 * 16 * 16)(generator_input)
x = layers.LeakyReLu()(x)
x = layers.Reshape((16 * 16 * 128)) (x)

x = layers.Conv2D(256, 5, padding='same')(x)
x = layers.LeakyReLu()(x)

# upsamples to 32*32
x = layers.Conv2DTranspose(256, 4, strides=2, padding='same')(x)
x = layers.LeakyReLu()(x)

x = layers.Conv2D(256, 5, padding='same')(x)
x = layers.LeakyReLu()(x)
x = layers.Conv2D(256, 5, padding='same')(x)
x = layers.LeakyReLu()(x)

# produces 32*32 1 channel feature map
x = layers.Conv2D(channels, 7, activation='tanh', padding='same')(x)
generator = keras.models.Model(generator_input, x)
generator.summary()

The Discriminator

discriminator_input = layers.Input(shape=(height, width, channels))
x = layers.Conv2D(128, 3)(discriminator_input)
x = layers.LeakyReLU()(x)
x = layers.Conv2D(128, 4, strides=2)(x)
x = layers.LeakyReLU()(x)
x = layers.Conv2D(128, 4, strides=2)(x)
x = layers.LeakyReLU()(x)
x = layers.Conv2D(128, 4, strides=2)(x)
x = layers.LeakyReLU()(x)
x = layers.Flatten()(x)

# dropout layer
x = layers.Dropout(0.4)(x)

# output layer
x = layers.Dense(1, activation='sigmoid')(x)

# instantiates module
discriminator = keras.models.Model(discriminator_input, x)
discriminator.summary()

# using gradient clipping by value in the optimizer / to stabilize, use learning rate decay
discriminator_optimizer = keras.optimizers.RMSprop(lr=0.0008, clipvalue=1.0, decay=1e-8)
discriminator.compile(optimizer=discriminator_optimizer, loss='binary_crossentropy')

The GANs

# only applies to gan model
discriminator.trainable = False

gan_input = keras.Input(shape=(latent_dim, ))
gan_output = discriminator(generator(gan_input))
gan = keras.models.Model(gan_input, gan_output)

gan_optimizer = keras.optimizers.RMSprop(lr=0.0004, clipvalue=1.0, decay=1e-8)
gan.compile(optimizer=gan_optimizer, loss='binary_crossentropy')

How to train the DCGAN

import os
from keras.preprocessing import image

# laod cifar-10
(x_train, y_train), (_, _) = keras.datasets.cifar10.load_data()

x_train = x_train.reshape(
  (x_train.shape[0],) +
  (height, width, channels)).astype('float32') / 255.

iterations = 10000
batch_size = 20
save_dir = 'your_dir'

start = 0
for step in range(iterations):
  random_latent_vectors = np.random.normal(size=(batch_size, latent_dim))

generated_images = generator.predict(random_latent_vectors)

stop = start + batch_size
real_images = x_train[start: stop]
combined_images = np.concatenate([generated_images, real_images])

labels = np.concatenate([np.ones((batch_size, 1)), np.zeros((batch_size, 1))])
labels += 0.05 * np.random.random(labels.shape)

d_loss = discriminator.train_on_batch(combined_images, labels)

random_latent_vectors = np.random.normal(size=(batch_size, latent_dim))

misleading_targets = np.zeros((batch_size, 1))

a_loss = gan.train_on_batch(random_latent_vectors, misleading_targets)

start += batch_size
if start > len(x_train) - batch_size:
  start = 0

if step % 100 == 0:
  gan.save_weights('gan.h5')

  print('discriminator loss:', d_loss)
  print('adversarial loss:', a_loss)

  img = image.array_to_img(generated_images[0] * 255., scale=False)
  img.save(os.path.join(save_dir, 'generated_frog' + str(step) + '.png'))
  img = image.array_to_img(real_images[0] * 255., scale=False)
  img.save(os.path.join(save_dir, 'real_frog' + str(step) + '.png'))

Python pathlib 使用小结

在 python 中，文件读写，包括日志，导入文件等都需要用到文件的地址，而对于不同的系统，比如 windows 和 mac ，使用 python 原生的相关函数和字符串，比如 join 进行控制很是困难，常常会遇到各种各样的不兼容问题。因此使用 pathlib 进行地址管理简化了工作，很是重要。

引用

from pathlib import Path

p = Path('log')

在当前目录下，返回一个交 log 的文件路径。这一步只是初始化了 pathlib 文件的路径，并没有对实际的文件系统有任何影响。

建立文件 touch

p.touch(exist_ok=True)

这一步建立了一个名为 log 的文件，实际的文件内容为空。

• exist_ok DEFAULT=True 这个参数检查的是文件是否存在。如果值为 False ，且文件已经存在的话，会出现 FileExistsError。

建立文件夹 mkdir

p.mkdir(parents=True, exist_ok=True)

这一步建立了一个名为 log 的文件夹。

• parents DEFAULT=False 查看是否存在上级目录。如果值为 True ，且p中没有指明上级目录，会出现 FileNotFoundError。
• exist_ok 同 touch 。

读写文件 open

和 os.open 一样，pathlib 也有自己的 open 函数。

with p.open('w') as f:
	f.write('this is a message')

具体使用方法和 python 原生的一样。

如果必须使用 Python 原生的 open 函数，则要注意 window 和 mac 不同的地方在于：

# windows
with open(p.name, 'w') as f:
	f.write('this is windows message')

# Mac os
with open(p, 'w') as f:
	f.write('this is mac message')

# linux 我没用过

地址相连 join

Path 中有关于 join 的函数 joinpath ，但是一般的，也有比较简单地，有局限性的方法。比如直接在建立 path object 的时候进行 join。

这种方式返回了一个新的 object，而不是更改了原有的，可以有局限性的使用啦。

plog = Path('log', 'first-log')
pa = Path(plog, 'a')

其他函数

pathlib 还有许多其他的 os 函数，具体如下链接。

Type

• System Testing: carried out on the whole system in the context of either system requirement specifications, functional requirement specifications, or in a contest of both.
• E2E Testing: test whether the flow of software from the initial stage to the final stage is behaving as expected.
• Unit Testing: a software development process in which the smallest testable parts of an application, called units, are individually and independently scrutinized for proper operation.
• Integration Testing: a type of software testing in which the different units, modules, or components of a software application are tested as a combined entity.

Code Coverage

It is a software testing metric that determines the number of lines of code that are successfully validated under a test procedure, which in turn, helps in analyzing how comprehensively a software is verified.

The only factor that affects the code coverage, is whether it is tested, not success or fail.

start new:

tmux

start new with session name:

tmux new -s myname

attach:

tmux a  #  (or at, or attach)

attach to named:

tmux a -t myname

list sessions:

tmux ls

kill session:

tmux kill-session -t myname

Kill all the tmux sessions:

tmux ls | grep : | cut -d. -f1 | awk '{print substr($1, 0, length($1)-1)}' | xargs kill

Copy from https://gist.github.com/MohamedAlaa/2961058

PATH

echo $PATH to display the variables.

And use PATH DEFAULT=${PATH}:/path/to/file to add a new one.

pwd

shows the current dictionary.

find

find a file with given name:

find . -name file_name

cat

cat could read the file content and output as a standard output stream.

alias

Create a shortcut of a command, for example:

alias lll = ls -a

more, less, most

less = more + backward navigation

most = more + less + some other features

which

which ls display the location of executables.

| Pipeline

Receive the input stream and output as the input of another command.

history | grep ls

> Direction

Receive the input stream and overwrite to the file.

"aaa" > a.py

chmod

chmod 777 main.py add permission for the file to be executable.

the second parameter is person add/remove permission.

• User
  • u: user
  • g: group
  • o: other
• +/-
• Permission
  • r: read
  • w: write
  • e: execute

whoami

existential crisis, to show the user name.

Run executable file in terminal

PATH

Type in ./clean instead of clean. Or, if you don't want to type the ./ prefix every time, add the present working directory (the dot) to your PATH environment variable (e.g. in your ~/.bash_profile or its equivalent in another shell).

Cent OS 7/8 配置静态IP

使用工具

• dhclient
• ifconfig / ip addr show

过程

链接网络

首先使用各种方法（我用的是WiFi）连接网络，确保可以访问。

测试：

ping baidu.com

使用dhclient分配静态IP地址

sudo dhclient

如果成功没有报错，那么就用ifconfig 或者是ip addr show 查看。

以下是我分配后的输出：

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
2: enp0s5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:1c:42:c4:45:85 brd ff:ff:ff:ff:ff:ff
    inet 这里是你的ip地址/24 brd ip前三位.255 scope global noprefixroute enp0s5
       valid_lft forever preferred_lft forever
    inet6 2601:1c0:6f00:7860::3ae0/128 scope global dynamic noprefixroute 
       valid_lft 598525sec preferred_lft 598525sec
    inet6 2601:1c0:6f00:7860:8954:52ad:3c10:54c1/64 scope global dynamic noprefixroute 
       valid_lft 199966sec preferred_lft 199966sec
    inet6 fe80::e853:11f4:2acd:3d00/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

修改网络配置文件

知道你的ip地址之后，需要在 /etc/sysconfig/network-scripts/ifcfg-NAME 中，修改网络配置。 NAME 中的值是和前面 2: enp0s5: <BROADCAST,MULTICAST,UP,LOWER_UP> 中的一样的。比如我的就是 enp0s5 啦。

这里记得切换成为超级用户。

su root

打开文件，其中我们需要修改，或者加上的是以下几行：

BOOTPROTO=static
ONBOOT=yes
IPADDR=写上的ip地址
NETMASK=255.255.255.0
GATEWAY=ip前面三位.1
DNS1=119.29.29.29

GATEWAY 可以写为你的ip前面三位.1。比如我的 ip 是 119.29.29.40，GATEWAY 可以写为 119.29.29.1。

DNS 这里写一个知名的就好了，例如：

• 奇安信：114.114.114.114
• Google：8.8.8.8
• 微步（OneDNS）：117.50.10.10

重启网络服务

这一步CentOS 7 和 8 略有不同：

CentOS 7

sudo systemctl restart network.service

CentOS 8

sudo systemctl restart NetworkManager.service

测试

看看能不能：

ping baidu.com

或者是用其他的电脑：

ping 你的IP

参考资料

CodeSheep：建议人手一套：个人专属多节点Linux环境打造，Linux操作系统学习实验环境安装配置视频教程

人家讲的比我详细多了哈哈哈，我这就是加了两个命令。

Hello, as a Pythonista, sometimes I have been confused about how to use import correctly and efficiently. This article will cover all things I know about it.

What do we import?

In an application, we often use import to import another module or package and let the following code could use them. So in the first part, I will discuss what modules and packages are.

A module is an object that serves as an organizational unit of Python code. It has a namespace

• module: A python object
  • have a naming space.
  • In coding, it often corresponds to a .py file.
  • a concept of Python runtime.
• package: a module with __path__ attribute.
  • In general, it often corresponds to a dictionary.
  • CAN contain other submodules and subpackage.

Agile has becomes a famous interactive process for developing software.

Why Agile

Why not Waterfall

In a waterfall, every stage depends on the previous stage.

• Con: no variation, adaptability, or other error once a waterfall project is set in motion.
• Pro: clear, productive, good time management, and available progress tracing.

Agile is more flexibility, adaptivity, and error tolerable.

Types

Some agile types include Scrum, Lean, and Extreme Programming (XP). This course talks about scrum.

Scrum

There are several roles, backlogs, and Ceremonies in the scrum.

Points

<10 points per task. Fab number is better.

Velocity comparisons by number of points completed per sprint is a senseless way to compare teams because the meaning of 1 point / 3 points / 5 points / etc. is determined by each team internally. One team's typical 3 pointer could be much harder than another team's 5 pointer.

Roles

• Product Owner: maximize the return the business gets on investments. Including salary, company renting fee, facilities, software, and maintenance.
  • They own the product backlog.
  • orders (prioritizes) the items in the product backlog.
• Scrum Master: a coach who guides the team to an over-high level of cohesiveness, self-organization, and performance.
  • They set the points.
  • scrum expert and advisor
  • impediment bulldozer
  • facilitator
• Team Member: how work done. What tech and tool to use. Who does which task.

Artifacts

• Product: list of deliverables for a product.
• Sprint: the TODO lists of a sprint.
• Turndown Chart: A burn chart shows how work remains on a given day. It is a good idea to start daily standup with it so that the development team can get timely real-time feedback about whether they are on track or not.
• Task Board: Tasks of to do, doing and done.

Ceremonies

• Sprint Planning Meeting: 2h/week
  • marks the beginning of the sprint.
  • commit to a set of deliverables for the sprint.
  • identifies the tasks that must be completed in order to deliver the agreed upon user stories
• Daily Scrum: < 10 mins, every week day
  • hold this meeting at the start of their work day
  • inspect and adapt the work the team members are doing
• Story Time: 1h/week
  • product owner will do: Define and Refine Acceptance Criteria
  • discussing and improving the stories in your product backlog, which contains all the stories for future sprints.
• Sprint Review: 30-60-120/sprint
  • invite any and all stakeholders.
  • demonstrating the stories that did get done.
  • stakeholders will have feedback and ideas, and the product owner and the team members will gather this feedback.
• Retrospective: 1-2 h/sprint
  • Only the team.
  • focus on what was learned during the sprint, and how that learning can be applied to make some improvement.

Semantic Versioning

It is an industry convention.

Major.Minor.Patch: 1.2.3

• Major: breaking changes of API.
• Minor: Changing function level, like adding some functionality and the function signature.
• Patch: Bug Fixing.

Flow

Two types of flow: Git Flow and GitHub Flow.

Git Flow

It has several branches lives infinitely.

GitHub Flow

It has a main branch that lives infinitely, and every branch should merge with it.

Merge

Commit

Edit commit before push

git commit --amend

Hooks

Like a real hook, Git Hook will run a specific script before or after important git actions, such as committing, rebasing, and pushing.

Where is it

All the hook files are stored in the .git/hooks:

applypatch-msg.sample     post-update.sample        pre-merge-commit.sample   pre-receive.sample        update.sample
commit-msg.sample         pre-applypatch.sample     pre-push.sample           prepare-commit-msg.sample
fsmonitor-watchman.sample pre-commit.sample                pre-rebase.sample         push-to-checkout.sample

How to write?

Open post-update.sample; here is an example:

#!/bin/sh
#
# An example hook script to prepare a packed repository for use over
# dumb transports.
#
# To enable this hook, rename this file to "post-update".

exec git update-server-info

Actually, not only bash script works, Python, Lua, or other script language work. Remember to add the path of exec file at the first line.

#!/bin/sh
#!/bin/python3
#!/usr/bin/env python3

Exit Value

In some pre check, the exit value could control whether the next step will run. If exit(0) is used in the pre-commit hook, the commit will achieve success. However, if the hook exit with -1(exit(-1)), the commit will be prevented until the requirements are met. Here is an example:

#!/usr/bin/env python3

import os

res = os.system('black ./tensor/ --check')

if res == 0:
    exit(0)
else:
    print('Some codes need to be format! please fix it.')

exit(-1)

In this example, the script uses black to check whether the code is formatted or not. If the code is unformatted, it will prevent the commit.

Usage

Remove the .sample in the file name; the hook will automatically run at a certain time.

PR check

Before the merge, during the PR check, we could run the automatic test, including unit testing and integration testing.

Merge

The default merge is the normal merge.

A review of Kafka.

What is Kafka?

Kafka is a multi-partition, multi-replica distributed messaging queue based on ZooKeeper coordination.

Architecture

Kafka uses producers to receive messages and store them in the cluster, then other software gets those messages by using consumers (or consumer groups). In the cluster, there are different brokers (an independent server) store all the information of different partitions. Below is a UML (from there):

1. Zookeeper: store metadata and manage the service discovery for all brokers in the cluster.
2. Producer: will create the message and send it to a certain topic.
3. Consumer: other software used for getting messages from a subscribed topic.
4. Broker: is an independent server. One or more brokers make up the cluster.
5. Consumer group: one or more consumers make up a consumer group. One partition has one consumer per group.
6. Topic: is a queue to organize messages. Also, a topic replicated one or more partitions.

Why Kafka?

Kafka could receive messages from multiple producers and store them as a uniform data structure. Also, it allows multiple consumers or consumer groups to read data from multiple topics. For different consumer groups, they could read messages on the same topic independently. For different consumers in one consumer group, Kafka could ensure that every message could be read and only could be read one time. Those two reasons are based on the message transfer mechanism.

Also, data storage is the next reason. Kafka will directly store data on the disk and allow consumers to read them asynchronously. In this case, Kafka could ensure that every message will not be missed.

The scalability is the third reason. Kafka allows users to add or delete brokers (topics, partitions) by themselves. Also, high performance is the fourth reason.

Relationship

Zookeeper.

Zookeeper has four functions in this progress. First, it maintains the configuration of topics. It stores and manages the metadata of existing topics, the number of partitions, the location of replicas, and so on. Secondly, Zookeeper will elect a new controller when an old controller is shutting down to make sure there is one and only one leader broker exists and all other follower brokers agree on that.

Controller: is one (only one in the Kafka ecosystem) of the brokers that have the responsibility to maintain the leader-follower relationship across all the partitions.

How does Zookeeper elect a new controller?

Broker

GLM

GLM is a set of C++ classes and functions to fill in the programming gaps in writing the basic vector and matrix mathematics for OpenGL applications. isn’t really a library.

Why use that?

The OpenGL overlords have “deprecated” some of the OpenGL functions we have been using to perform transformations. In the desktop world, it means that the use of such functions is discouraged. In Vulkan and in the mobile world of OpenGL-ES, it means those functions are gone.

Data type

glm::vec3: 3 components vector of floating-point numbers.
glm::mat3: 3 columns of 3 components matrix of floating-point numbers.
glm::mat4: 4 columns of 4 components matrix of floating-point numbers.
glm::vec3: 3 components vector of floating-point numbers.

vec3 could present coordinates, color (RGB), or others. vec4 could present color (RGBT) or something others.

Vertex Buffer Objects

What are they

an OpenGL feature that provides methods for uploading vertex data to the video device for non-immediate-mode rendering. Store vertex coordinates and vertex attributes on the graphics card.

The OpenGL Rendering Context (also called “the state”) contains all the characteristic information necessary to produce an image from geometry. This includes the current transformation, color, lighting, textures, where to send the display, etc.

where they live

GPU

what the advantage of using them is

Every time you go to redraw, coordinates will be pulled from GPU memory instead of CPU memory, avoiding a significant amount of bus latency.

GL_ARRAY_BUFFER vs. GL_ELEMENT_ARRAY_BUFFER

• GL_ELEMENT_ARRAY_BUFFER: index buffer
• GL_ARRAY_BUFFER: vertex buffer
• glDrawElements: draw primitives using bound element array buffer.
• glDrawArrays: draw primitives using bound vertex buffers

Shadows

Two-pass algorithm:

1. the First pass renders from the point of view of the light source and records depths from there

• a view of the light source
• a depth view from the light source, using Z-buffer as a deep shadow map.

2. the Second pass uses those depths to determine what fragments are not visible to the light and thus are in a shadow. If a fragment is in a shadow, render it with Ambient light only.

Geometric Modeling

Meshes

Mesh: a 3D object representation is consisting of a collection of vertices and polygons. It is a way to organize those faces and vertexes.

If it’s in nice neat rows like this, it is called a Regular Mesh.

If not, it is irregular mesh, Triangular Irregular Network, or TIN.

Edit

It can be smooth, also it can be edited.

• Pulling on a single Vertex
• Pulling on a Vertex with Proportional Editing Turned On
• 3D Boolean Operators
• Volume Sculpting(体积雕刻): sculpt Lattices(点阵).

Curve Sculpting

Another way to model.

Bézier Curve Sculpting:

Bézier surfaces exist and the bicubic ones are sculpted with 16 points.

Catmull-Rom Curve Sculpting: for any four points, it will only draw the line between the second and third points.

t goes from 0.0 to 1.0

Constructive Solid Geometry CSG

3D Venn Diagrams, Boolean

Metaball

Metaballs that can "glom" together if they get close enough

displacement texture

Using displacement textures in the vertex shader (Moon example)

The 3D moon displacement was geometrically modeled by: Reading a displacement texture in the vertex shader.

Image texture from the fragment shader？？？

L-systems

An L-system models geometry by: Creating a procedural equation and applying it recursively.

Simplified Euler's rule

relating Faces, Edges, and Vertices (F - E + V = 2)

Forward Kinematics

Input is the animation parameters.

The general pattern for computing [M n/g] = [M 1/g] * [M 2/1] * ... * [M n-1/n-2] * [M n/n-1]

Write it from left to right, say it from right to left: It’s because computer graphics has adopted the convention that the coordinates are multiplied on the right side of the matrix. In the program, write it up to down, and say it down to up.

Rendering

Rendering Equation

Light Shining from a point P = Light emitted by that point + Reflectivity * Σ(Light arriving from all other points)

Local versus Global illumination

Generally, OpenGL uses local illumination.

OpenGL rendering

• OpenGL: Starts at the Object, Works Towards the Eye
• ray-tracing: Starts at the eye, works towards the object (painter)

Z-buffer

Add an extension to the framebuffer to store the depth of each pixel. This is called a Depth-buffer or Z-buffer. Only allow pixel stores when the depth of the incoming pixel is closer to the viewer than the pixel that is already there.

Ray-tracing

• pro: highly realistic images
• con: computationally intensive: computation complexity is about square of object number

light transport

• to light source

• reflection

• refraction

intersectng

Both t’s are complex: ray missed the sphere
Both t’s are real and identical: ray is tangent to the sphere
Both t’s are real and different: ray goes through the sphere

Physically-Based Rendering (PBR).

Radiosity

Based on the idea that all surfaces gather light intensity from all other surfaces.

The light energy leaving surface i equals the amount of light energy generated by surface i plus surface i’s reflectivity times the amount of light energy arriving from all other surfaces

• Radiosity methods track rate at which energy (radiosity) leaves [diffuse] surfaces
• Determine equilibrium of light energy in a view-independent way
• Allows for diffuse interreflection, color bleeding, and walkthroughs
• Difficult to handle specular objects, mirrors
• large computational and storage cost.

Subsurface scattering

Subsurface Scattering mathematically models light bouncing around within an object before coming back out.

what materials is it good for creating the illusion of? skin, wax, milk, paraffin, etc.

Path-Tracing

This distribution of bounced light rays is called the Bidirectional Reflectance Distribution Function, or BRDF.

For a given material, the BRDF behavior of a light ray is a function of 4 variables: the 2 spherical coordinates of the incoming ray and the 2 spherical coordinates of the outgoing ray.
The outgoing light energy in the outgoing BRDF’s is always less than or equal to the amount of light that shines in.

Screen Space Ambient Occlusion (SSAO)

Animation

Key-framing: what it is, why do it this way
Forward kinematics: what it is, why do it this way
Inverse kinematics (IK): what it is, why do it this way
Particle systems: what they are, the three elements of doing it (Emit, Display, Update), what effects you can create this way
Rigid Body Physics: why do it this way, F=ma, springs, dampers, chains/strings, cloth
Functional animation: what it is, how it is done, why do it this way, its use in collision avoidance
Motion Capture (MoCap): what it is, why do it this way

Stencil Buffer

What is it? Separate Framebuffer, or, You Can Think of it as being Per-Pixel

How does it work?

What are some of the things you can do with it?

• While drawing into the Back Buffer, you can write values into the Stencil Buffer at the same time.
• While drawing into the Back Buffer, you can do arithmetic on values in the Stencil Buffer at the same time.
• The Stencil Buffer can be used to write-protect certain parts of the Back Buffer.

3D Printing

“3D Printing” is defined as some sort of “additive” process. (Additive manufacturing is also sometimes called Stereolithography.)

The 3D Printing Geometry File: STL

In this particular file, these coordinates are in units of inches.
Some 3D Printers use inches, most now seem to use millimeters.
there are 25.4 mm/inch

Object Modeling Rule

• The object must be a mesh and consist only of triangles.
• The object must be a legal solid. It must have a definite inside and a definite outside. It can’t have any missing face pieces.
• You can’t make a compound object by simply overlapping two objects in 3D. If you want both shapes together, do a Boolean union on them so that they become one complete, legal object.
• Vertex-to-Vertex Rule: Each edge in the mesh must bound 2 and only 2 triangles

Overhangs:

Some 3D printers handle this by leaving unused material in place to support the overhangs
Some 3D printers handle this by using software to add “support structures” to the overhangs

Simplified Euler's Formula

(F - E + V = 2)

Putting the Eye Position on an Orbiting Body

1. Put the eye-position at the corner of its Equator and Prime Meridian (xe = EARTH_RADIUS_MILES, ye = 0., ze = 0.)

2. Set the look-at position to be on a straight-line east of the eye- position: (xl = EARTH_RADIUS_MILES, yl = 0., zl = -100.)

3. Set the up-vector to be: (xu = 100., yu = 0., zu = 0.)

4. Spin the Earth by EarthSpinAngle about its Y axis

5. Translate the Earth by EARTH_ORBITAL_RADIUS_MILES in its X direction

6. Revolve the Earth by EarthOrbitAngle about the Sun’s Y axis

Stereographics

Horizontal parallax:

Vertical parallax: This works OK if you are doing orthographic projections, but if you use perspective, you will achieve a nasty phenomenon called vertical parallax。

Plane of zero parallax
Non-symmetric viewing volumes
Different ways of channeling the images into each eye
Lenticular
ChromaDepth