Code Monkey home page Code Monkey logo

notes's Introduction

notes

记录开发过程中的点点滴滴。

notes's People

Contributors

v5tech avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

notes's Issues

The Status Bar

1. 修改状态栏样式:

applicationDidFinishLaunching方法中添加

[[UIApplication sharedApplication] setStatusBarStyle: UIStatusBarStyleBlackTranslucent];

其可选的值为UIStatusBarStyleDefault, UIStatusBarStyleBlackTranslucentUIStatusBarStyleBlackOpaque

2. 隐藏状态栏:

applicationDidFinishLaunching方法中添加

[[UIApplication sharedApplication] setStatusBarHidden:YES animated:NO];

超级工具箱Super Tools Suite

网上找到几个不错的工具集

生成器(Generator)

  • JSON数据生成器:

http://www.json-generator.com/
http://jsonplaceholder.typicode.com/

  • Password密码生成器:用于各种系统用户的初始密码

http://strongpasswordgenerator.com/

  • CSS Sprite生成器:减少服务器请求合并零散的小图

http://www.spritecow.com/

  • GIF 图像生成器

http://gifmake.com/

  • Icon/Favicon 图标生成器

http://www.genfavicon.com/

  • QRcode 二维码生成器

QR Code generator http://www.unitaglive.com/qrcode
草料二维码 http://cli.im/
联图网 http://www.liantu.com/
易拍酷 http://www.epcool.com/

  • Placeholder 占位图像生成器

(1)http://dummyimage.com/
用PHP写的,基于MIT License代码开源。可以设置图像的宽度、高度、背景色、前景色、图像格式、文本。
URI格式:
http://dummyimage.com/宽度x高度/背景色/前景色.图像格式&文本
例如:
http://dummyimage.com/320x240/333333/00a2ff.png&text=RENSANNING

(2)http://placehold.it/
功能基本等同于dummyimage.com
例如:
http://placehold.it/320x240/333333/00a2ff.png&text=RENSANNING

(3)http://fpoimg.com/
除了不能设置图像格式,只能输出PNG外其他功能基本等同于dummyimage.com,就是URI的格式稍 有不同。
例如:
http://fpoimg.com/320x240?text=RENSANNING&bg_color=333333&text_color=00a2ff

(4)http://placehold.jp/
可以设置:宽度、高度、前景色、背景色、文字大小、图像格式(PNG/JPG)、文字、CSS
例如:
http://placehold.jp/46/cc9999/993333/320x240.png?text=RENSANNING

(5)http://placekitten.com/
这个除了能设置宽度和高度外,有趣的是图像都是猫。
比如:
http://placekitten.com/320/240

(6)http://placeimg.com/
这个除了能设置宽度和高度外,还可以指定输出图像的分类(animals、arch、nature、people、 tech),图像的颜色(grayscale,sepia)
例如:
http://placeimg.com/320/240/tech/sepia

(7)http://lorempixel.com/

(8)http://uifaces.com/

  • Random Color Generator

http://www.kareno.org/js/colors/

整形器(Formatter/Beautifier)

  • HTML/XML/CSS 整形器

http://www.xmltoolbox.com/

  • JavaScript 整形器

http://jsbeautifier.org/

  • JSON 整形器

http://jsonformat.com/

  • SQL 整形器

http://www.dpriver.com/pp/sqlformat.htm

JavaScript压缩混淆

  • JSMin

http://www.crockford.com/javascript/jsmin.html

  • YUI Compressor

http://yui.github.io/yuicompressor/

  • UglifyJS

http://lisperator.net/uglifyjs/

  • CSSTidy

http://csstidy.sourceforge.net/

  • TinyPNG

https://tinypng.com/

  • JPEGmini

http://www.jpegmini.com/

转换器(Converter)

  • 文档转换器(PDF-Word)

http://convert.neevia.com/pdfconvert/

  • 图像(JPG-PNG)

http://image.online-convert.com/
http://www.faststone.org/FSResizerDetail.htm

  • XML-JSON

http://xmlgrid.net/jsonXml.html

测试器(Tester/Validator)

  • HTML/CSS/Javascript

http://jsfiddle.net/
http://jsbin.com/
http://codepen.io/

  • 正则表达式

http://regexpal.com/

其他

  • 美化器(Prettifier/Highlighter)

http://prismjs.com/

  • 取色器Color Picker

http://www.colorpicker.com/

  • 加密解密(MD5、SHA1、SHA-256、Base64)

http://onlinemd5.com/
http://www.sha1-online.com/
http://www.everpassword.com/sha-256-generator
http://www.base64decode.org/

  • 编码转换(Ascii/Native、Unicode)

http://native2ascii.net/
http://www.branah.com/unicode-converter

  • Escape(SQL、CSV、HTML/XML、URL)

http://webtools.drunkvision.net/easyweb/

  • PDF 合并分割

http://www.splitpdf.com/
http://www.pdfsam.org/

linux下查看http 并发和 tcp连接数

linux查看httpd进程数

ps -ef | grep httpd | wc -l

查看Apache的并发请求数及其TCP连接状态

netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'

(这条语句是从新浪互动社区事业部技术总监王老大那儿获得的,非常不错)

返回结果示例:

LAST_ACK 5
SYN_RECV 30
ESTABLISHED 1597
FIN_WAIT1 51
FIN_WAIT2 504
TIME_WAIT 1057

其中的SYN_RECV表示正在等待处理的请求数;ESTABLISHED表示正常数据传输状态;TIME_WAIT表示处理完毕,等待超时结束的请求数。

来源http://blog.s135.com/post/269/

linux并发连接数查看

1、查看Web服务器(Nginx Apache)的并发请求数及其TCP连接状态:

netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'
netstat -n|grep  ^tcp|awk '{print $NF}'|sort -nr|uniq -c

或者:

netstat -n | awk '/^tcp/ {++state[$NF]} END {for(key in state) print key,"t",state[key]}'

返回结果一般如下:

LAST_ACK 5 (正在等待处理的请求数)
SYN_RECV 30
ESTABLISHED 1597 (正常数据传输状态)
FIN_WAIT1 51
FIN_WAIT2 504
TIME_WAIT 1057 (处理完毕,等待超时结束的请求数)

其他参数说明:

CLOSED:无连接是活动的或正在进行
LISTEN:服务器在等待进入呼叫
SYN_RECV:一个连接请求已经到达,等待确认
SYN_SENT:应用已经开始,打开一个连接
ESTABLISHED:正常数据传输状态
FIN_WAIT1:应用说它已经完成
FIN_WAIT2:另一边已同意释放
ITMED_WAIT:等待所有分组死掉
CLOSING:两边同时尝试关闭
TIME_WAIT:另一边已初始化一个释放
LAST_ACK:等待所有分组死掉

2、查看Nginx运行进程数

ps -ef | grep nginx | wc -l

返回的数字就是nginx的运行进程数,如果是apache则执行

ps -ef | grep httpd | wc -l

3、查看Web服务器进程连接数:

netstat -antp | grep 80 | grep ESTABLISHED -c

4、查看MySQL进程连接数:

ps -axef | grep mysqld -c

来源http://itnihao.blog.51cto.com/1741976/830365

Swift Code 碎片整理

Swift Code 碎片整理

1. NSURLConnection

var url = NSURL(string: "http://www.stackoverflow.com")
var request = NSURLRequest(URL: url)

NSURLConnection.sendAsynchronousRequest(request, queue: NSOperationQueue(), completionHandler: {(response: NSURLResponse!, data: NSData!, error: NSError!) in
        println(NSString(data: data, encoding: NSUTF8StringEncoding))
})

2. UIAlertView

let alertView:UIAlertView = UIAlertView()
alertView.title="提示"
alertView.message="恭喜你,你中奖了!"
alertView.addButtonWithTitle("确定")
alertView.show()

或者

var alert:UIAlertController = UIAlertController(title: "提示", message: "恭喜你,你中奖了!", preferredStyle: UIAlertControllerStyle.Alert)
var action: UIAlertAction = UIAlertAction(title: "确定", style: UIAlertActionStyle.Default, handler: {
    ac -> Void in
})
alert.addAction(action)
self.presentViewController(alert, animated: true, completion: nil)

3. navigationItem

//设置导航栏按钮(左边)
let leftBarButtonItem:UIBarButtonItem = UIBarButtonItem(title: "主页", style: UIBarButtonItemStyle.Plain, target: self, action: Selector("showMain"))
self.navigationItem.leftBarButtonItem = leftBarButtonItem

//设置导航栏按钮(右边)
let rightBarButtonItem:UIBarButtonItem = UIBarButtonItem(title: "设置", style: UIBarButtonItemStyle.Plain, target: self, action: Selector("showSetting"))
self.navigationItem.rightBarButtonItem = rightBarButtonItem

//设置导航栏名称
self.navigationItem.title="全国主要城市PM2.5"



func showMain(){

//SCLAlertView().showTitle(self,  title: "提示", subTitle: "你点击了主页", duration: 2.0, style: SCLAlertViewStyle.Info)

var alert = UIAlertController(title: "提示", message: "你点击了主页", preferredStyle: UIAlertControllerStyle.Alert)

alert.addAction(UIAlertAction(title: "确定", style: UIAlertActionStyle.Default, handler: {

    (UIAlertAction) -> Void in

        println("你点击了确定按钮")

    }))

self.presentViewController(alert, animated: true, completion: nil)

}

func showSetting(){
//SCLAlertView().showTitle(self,  title: "提示", subTitle: "你点击了设置", duration: 2.0, style: SCLAlertViewStyle.Info)

var alert = UIAlertController(title: "提示", message: "你点击了设置", preferredStyle: UIAlertControllerStyle.Alert)

alert.addAction(UIAlertAction(title: "确定", style: UIAlertActionStyle.Default, handler: {

    (UIAlertAction) -> Void in

        println("你点击了确定按钮")

    }))

self.presentViewController(alert, animated: true, completion: nil)
}

为CentOS配置yum源

为CentOS配置yum源

1. 首先备份/etc/yum.repos.d/CentOS-Base.repo

mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup

2. 添加源

2.1 添加163

下载对应版本repo文件, 放入/etc/yum.repos.d/(操作前请做好相应备份)

163源

2.2 添加sohu

下载repo文件, 放入/etc/yum.repos.d/(操作前请做好相应备份)

CentOS-Base-sohu.repo

sohu源

2.3 添加**科学技术大学源

首先备份CentOS-Base.repo

mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup

下载对应版本的CentOS-Base.repo, 放入/etc/yum.repos.d/

  • CentOS 5
# CentOS-Base.repo
#
# The mirror system uses the connecting IP address of the client and the
# update status of each mirror to pick mirrors that are updated to and
# geographically close to the client.  You should use this for CentOS updates
# unless you are manually picking other mirrors.
#
# If the mirrorlist= does not work for you, as a fall back you can try the 
# remarked out baseurl= line instead.
#
#

[base]
name=CentOS-$releasever - Base - mirrors.ustc.edu.cn
baseurl=http://mirrors.ustc.edu.cn/centos/$releasever/os/$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os
gpgcheck=1
gpgkey=http://mirrors.ustc.edu.cn/centos/RPM-GPG-KEY-CentOS-5

#released updates 
[updates]
name=CentOS-$releasever - Updates - mirrors.ustc.edu.cn
baseurl=http://mirrors.ustc.edu.cn/centos/$releasever/updates/$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates
gpgcheck=1
gpgkey=http://mirrors.ustc.edu.cn/centos/RPM-GPG-KEY-CentOS-5

#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras - mirrors.ustc.edu.cn
baseurl=http://mirrors.ustc.edu.cn/centos/$releasever/extras/$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras
gpgcheck=1
gpgkey=http://mirrors.ustc.edu.cn/centos/RPM-GPG-KEY-CentOS-5

#packages used/produced in the build but not released
[addons]
name=CentOS-$releasever - Addons - mirrors.ustc.edu.cn
baseurl=http://mirrors.ustc.edu.cn/centos/$releasever/addons/$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=addons
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5

#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-$releasever - Plus - mirrors.ustc.edu.cn
baseurl=http://mirrors.ustc.edu.cn/centos/$releasever/centosplus/$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=centosplus
gpgcheck=1
enabled=0
gpgkey=http://mirrors.ustc.edu.cn/centos/RPM-GPG-KEY-CentOS-5

#contrib - packages by Centos Users
[contrib]
name=CentOS-$releasever - Contrib - mirrors.ustc.edu.cn
baseurl=http://mirrors.ustc.edu.cn/centos/$releasever/contrib/$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=contrib
gpgcheck=1
enabled=0
gpgkey=http://mirrors.ustc.edu.cn/centos/RPM-GPG-KEY-CentOS-5
  • CentOS 6
# CentOS-Base.repo
#
# The mirror system uses the connecting IP address of the client and the
# update status of each mirror to pick mirrors that are updated to and
# geographically close to the client.  You should use this for CentOS updates
# unless you are manually picking other mirrors.
#
# If the mirrorlist= does not work for you, as a fall back you can try the 
# remarked out baseurl= line instead.
#
#

[base]
name=CentOS-$releasever - Base - mirrors.ustc.edu.cn
baseurl=http://mirrors.ustc.edu.cn/centos/$releasever/os/$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os
gpgcheck=1
gpgkey=http://mirrors.ustc.edu.cn/centos/RPM-GPG-KEY-CentOS-6

#released updates 
[updates]
name=CentOS-$releasever - Updates - mirrors.ustc.edu.cn
baseurl=http://mirrors.ustc.edu.cn/centos/$releasever/updates/$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates
gpgcheck=1
gpgkey=http://mirrors.ustc.edu.cn/centos/RPM-GPG-KEY-CentOS-6

#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras - mirrors.ustc.edu.cn
baseurl=http://mirrors.ustc.edu.cn/centos/$releasever/extras/$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras
gpgcheck=1
gpgkey=http://mirrors.ustc.edu.cn/centos/RPM-GPG-KEY-CentOS-6

#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-$releasever - Plus - mirrors.ustc.edu.cn
baseurl=http://mirrors.ustc.edu.cn/centos/$releasever/centosplus/$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=centosplus
gpgcheck=1
enabled=0
gpgkey=http://mirrors.ustc.edu.cn/centos/RPM-GPG-KEY-CentOS-6

#contrib - packages by Centos Users
[contrib]
name=CentOS-$releasever - Contrib - mirrors.ustc.edu.cn
baseurl=http://mirrors.ustc.edu.cn/centos/$releasever/contrib/$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=contrib
gpgcheck=1
enabled=0
gpgkey=http://mirrors.ustc.edu.cn/centos/RPM-GPG-KEY-CentOS-6

https://lug.ustc.edu.cn/wiki/mirrors/help/centos

**科学技术大学源

3. 运行yum makecache生成缓存

$ yum makecache

Gradle fails with “Ambiguous method overloading for method java.io.File#<init>”

android studio编译android项目时报以下错误:

D:\android-sdk-windows-x86_64\workspace\google-io-2014\app>gradle

FAILURE: Build failed with an exception.

* Where:
Build file 'D:\android-sdk-windows-x86_64\workspace\google-io-2014\app\build.gra
dle' line: 17

* What went wrong:
A problem occurred evaluating project ':app'.
> Ambiguous method overloading for method java.io.File#<init>.
Cannot resolve which method to invoke for [null, class java.lang.String] due to
overlapping prototypes between:
        [class java.lang.String, class java.lang.String]
        [class java.io.File, class java.lang.String]

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output.

BUILD FAILED

Total time: 4.365 secs

在项目的目录下创建local.properties其内容为:

sdk.dir=D\:\\android-sdk-windows-x86_64\\sdk

即:设置sdk.dir的目录为ANDROID_SDK的安装目录

参考链接
http://stackoverflow.com/questions/23811315/gradle-fails-with-ambiguous-method-overloading-for-method-java-io-fileinit

mongo-hadoop项目使用pig将数据导入mongodb及使用load data从本地将数据导入到mysql

CREATE DATABASE logs;

USE logs;
CREATE TABLE weblogs(
    md5             VARCHAR(32),
    url             VARCHAR(64),
    request_date    DATE,
    request_time    TIME,
    ip              VARCHAR(15)
);

若出现ERROR 1148 (42000): The used command is not allowed with this MySQL version

mysql> LOAD DATA LOCAL INFILE 'weblog_entries.txt' INTO TABLE weblogs
    -> FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n';
ERROR 1148 (42000): The used command is not allowed with this MySQL version

使用如下方式连接mysql

mysql -uroot -proot --local_infile=1

使用LOAD DATA LOCAL从本地数据文件导入数据库表中

mysql> use logs;
Database changed
mysql> LOAD DATA LOCAL INFILE 'weblog_entries.txt' INTO TABLE weblogs
    -> FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n';
Query OK, 3000 rows affected (0.04 sec)
Records: 3000  Deleted: 0  Skipped: 0  Warnings: 0

mysql> select count(*) from weblogs;
+----------+
| count(*) |
+----------+
|     3000 |
+----------+
1 row in set (0.01 sec)

mysql> 

禁用Xcode的index(索引服务)功能

使用Xcode打开一些体积较大的工程时,会一直停留或卡在indexing阶段,在终端使用下面的命令禁用index(索引服务)。

  • turn off Indexing 禁用index(索引服务)功能
sudo defaults write com.apple.dt.XCode IDEIndexDisable 1
  • turn Indexing on 开启index(索引服务)功能
sudo defaults write com.apple.dt.XCode IDEIndexDisable 0

Swift方法调用参数名称的省略

  • 初始化方法 强制要求所有参数名称
  • 类型的init方法 强制要求所有参数名称
  • 类方法 匿名第一个参数,并强制要求其他参数名称
  • 实例方法 匿名第一个参数,并强制要求其他参数名称
  • 全局方法 匿名所有参数名称

使用rbenv安装ruby 2.2.0

  • 依赖环境
sudo apt-get install libffi-dev
sudo apt-get update
sudo apt-get install -y git-core curl zlib1g-dev build-essential \
                     libssl-dev libreadline-dev libyaml-dev libsqlite3-dev sqlite3 \
                     libxml2-dev libxslt1-dev libcurl4-openssl-dev software-properties-common
  • 安装rbenv
cd ~
git clone git://github.com/sstephenson/rbenv.git .rbenv
echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc
echo 'eval "$(rbenv init -)"' >> ~/.bashrc
git clone git://github.com/sstephenson/ruby-build.git ~/.rbenv/plugins/ruby-build
echo 'export PATH="$HOME/.rbenv/plugins/ruby-build/bin:$PATH"' >> ~/.bashrc
exec $SHELL
  • 安装ruby
rbenv install 2.2.0
rbenv global 2.2.0
  • Gem提速
echo "gem: --no-ri --no-rdoc" > ~/.gemrc
  • 安装rails
gem install rails -v 4.2.0
rbenv rehash # 导出 gem 中提供的系统命令
  • 查看rails版本
rails -v
  • 安装MySQL
sudo apt-get install mysql-server  mysql-client  libmysqlclient-dev

elasticsearch-river-jdbc

#1. 创建元数据

PUT 127.0.0.1:9200/_river/jdbc_river/_meta
{
    "type" : "jdbc",
    "schedule" : "0 0-59 0-23 ? * *",
    "jdbc" : [
    {
        "url" : "jdbc:mysql://127.0.0.1:3306/hibernate_search",
        "user" : "root",
        "password" : "",
        "sql" : "SELECT * FROM book",
        "index": "book",
        "type": "book"
        },
    {
        "url" : "jdbc:mysql://127.0.0.1:3306/hibernate_search",
        "user" : "root",
        "password" : "",
        "sql" : "SELECT * FROM author",
        "index": "author",
        "type": "author"
        }
    ]
}

注:以上示例将同时创建book和author两张表的元数据及数据索引。
#2. 查看book索引

GET 127.0.0.1:9200/book/_search

#3. 查看author索引

GET 127.0.0.1:9200/author/_search

#4. 搜索

POST 127.0.0.1:9200/author/_search
{
  "query": {"query_string": {
    "default_field": "name",
    "query": "赵凯"
  }}
}

#5. 高亮查询

POST 127.0.0.1:9200/book/_search
{
  "query" : { "term" : { "description" : "经验" }},
    "highlight" : {
        "pre_tags" : ["<color>", "<em>"],
        "post_tags" : ["</color>", "</em>"],
        "fields" : {
            "description" : {}
        }
    }
}

#6. 删除元数据

DELETE 127.0.0.1:9200/_river/jdbc_river

#7. 删除索引

DELETE 127.0.0.1:9200/book
DELETE 127.0.0.1:9200/author

#8. 刷新元数据

GET  127.0.0.1:9200/_river/_refresh

#9. 刷新索引

GET  127.0.0.1:9200/book/_refresh
GET  127.0.0.1:9200/author/_refresh

#10. 查看river jdbc plugin 状态

http://127.0.0.1:9200/_river/jdbc/*/_state?pretty

ffmpeg视频合并、格式转换、截图

使用ffmpeg合并MP4文件

ffmpeg -i "Apache Sqoop Tutorial Part 1.mp4" -c copy -bsf:v h264_mp4toannexb -f mpegts intermediate1.ts
ffmpeg -i "Apache Sqoop Tutorial Part 2.mp4" -c copy -bsf:v h264_mp4toannexb -f mpegts intermediate2.ts
ffmpeg -i "Apache Sqoop Tutorial Part 3.mp4" -c copy -bsf:v h264_mp4toannexb -f mpegts intermediate3.ts
ffmpeg -i "Apache Sqoop Tutorial Part 4.mp4" -c copy -bsf:v h264_mp4toannexb -f mpegts intermediate4.ts
ffmpeg -i "concat:intermediate1.ts|intermediate2.ts|intermediate3.ts|intermediate4.ts" -c copy -bsf:a aac_adtstoasc "Apache Sqoop Tutorial.mp4"

视频分割

ffmpeg -ss 00:00:00 -t 00:00:03 -y -i test.mp4 -vcodec copy -acodec copy test1.mp4  

参数解释

说明:上面的这个例子是将test.mp4视频的前3秒,重新生成一个新视频。
-ss 开始时间,如: 00:00:00,表示从0秒开始,格式也可以00:00:0
-t 时长,如: 00:00:03,表示截取3秒长的视频,格式也可以00:00:3
-y 如果文件已存在强制替换;
-i 输入,后面是空格,紧跟着就是输入视频文件;
-vcodec copy 和 -acodec copy表示所要使用的视频和音频的编码格式,这里指定为copy表示原样拷贝;

使用ffmpeg转换flv到mp4

ffmpeg -i out.flv -acodec copy -vcodec copy out.mp4
ffmpeg -i out.flv -c:v libx264 out.mp4

使用ffmpeg截图

ffmpeg -ss 00:10:00 -i "Apache Sqoop Tutorial.mp4" -y -f image2 -vframes 1 test.png

ffmpeg -ss 10 -i input.flv -y -f image2  -vframes 100 -s 352x240 b-%03d.jpg
ffmpeg -i test.mp4 -y -f mjpeg -ss 3 -t 1  test1.jpg
ffmpeg -i test.mp4 -y -f image2 -ss 3 -vframes 1 test1.jpg

上面二个例子都表示,在第三秒的时候,截图。

参数解释:

-i  输入文件
-y  覆盖
-f  生成图片格式
-ss 开始截图时间 seconds or in hh:mm:ss[.xxx] 如果截图开始时间越接近篇尾,所花费的时间就会越长
-vframes  截图帧数 或者 使用 -t : 截图时长 seconds, or hh:mm:ss[.xxx]
-s  图片宽高比
b-%3d.jpg 格式化文件命名,会生成 b-001.jpg,b-002.jpg 等。

注意:把-ss 10放到第一个参数的位置,速度比放到放到其他位置快,且不会出现如下错误

添加水印

  • 水印局中
ffmpeg -i out.mp4 -i [email protected] -filter_complex overlay="(main_w/2)-(overlay_w/2):(main_h/2)-(overlay_h)/2" output.mp4

参数解释

-i out.mp4(视频源)
-i [email protected](水印图片)
overlay 水印的位置
output.mp4 输出文件

翻转和旋转

翻转

水平翻转语法: -vf hflip

ffplay -i out.mp4 -vf hflip

垂直翻转语法:-vf vflip

ffplay -i out.mp4 -vf vflip

旋转

语法:transpose={0,1,2,3}

0:逆时针旋转90°然后垂直翻转

1:顺时针旋转90°

2:逆时针旋转90°

3:顺时针旋转90°然后水平翻转

将视频顺时针旋转90度

ffplay -i out.mp4 -vf transpose=1

将视频水平翻转(左右翻转)

ffplay -i out.mp4 -vf hflip

顺时针旋转90度并水平翻转

ffplay -i out.mp4 -vf transpose=1,hflip

添加字幕

有的时候你需要给视频加一个字幕(subtitle),使用ffmpeg也可以做。一般我们见到的字幕以srt字幕为主,在ffmpeg里需要首先将srt字幕转化为ass字幕,然后就可以集成到视频中了(不是单独的字幕流,而是直接改写视频流)。

ffmpeg -i my_subtitle.srt my_subtitle.ass
ffmpeg -i inputfile.mp4 -vf ass=my_subtitle.ass outputfile.mp4

但是值得注意的是:

my_subtitle.srt需要使用UTF8编码,老外不会注意到这一点,但是中文这是必须要考虑的;

将字幕直接写入视频流需要将每个字符渲染到画面上,因此有一个字体的问题,在ass文件中会指定一个缺省字体,例如Arial,但是我们首先需要让ffmpeg能找到字体文件,不然文字的渲染就无从谈起了。ffmpeg使用了fontconfig来设置字体配置。你需要首先设置一下FONTCONFIG_PATH或者FONTCONFIG_FILE环境变量,不然fontconfig是无法找到配置文件的,这一点请参看这篇文章,如果你设置的是FONTCONFIG_PATH,那把配置文件保存为%FONTCONFIG_PATH%/font.conf即可,然后你可以在font.conf文件中配置字体文件的路径之类的。

Windows下为fontconfig设置如下的环境变量

FC_CONFIG_DIR=C:\ffmpeg
FONTCONFIG_FILE=font.conf
FONTCONFIG_PATH=C:\ffmpeg
PATH=C:\ffmpeg\bin;%PATH%

下面是一个简单的Windows版font.conf文件。

<?xml version="1.0"?>
<fontconfig>

<dir>C:\WINDOWS\Fonts</dir>

<match target="pattern">
   <test qual="any" name="family"><string>mono</string></test>
   <edit name="family" mode="assign"><string>monospace</string></edit>
</match>

<match target="pattern">
   <test qual="all" name="family" mode="not_eq"><string>sans-serif</string></test>
   <test qual="all" name="family" mode="not_eq"><string>serif</string></test>
   <test qual="all" name="family" mode="not_eq"><string>monospace</string></test>
   <edit name="family" mode="append_last"><string>sans-serif</string></edit>
</match>

<alias>
   <family>Times</family>
   <prefer><family>Times New Roman</family></prefer>
   <default><family>serif</family></default>
</alias>
<alias>
   <family>Helvetica</family>
   <prefer><family>Arial</family></prefer>
   <default><family>sans</family></default>
</alias>
<alias>
   <family>Courier</family>
   <prefer><family>Courier New</family></prefer>
   <default><family>monospace</family></default>
</alias>
<alias>
   <family>serif</family>
   <prefer><family>Times New Roman</family></prefer>
</alias>
<alias>
   <family>sans</family>
   <prefer><family>Arial</family></prefer>
</alias>
<alias>
   <family>monospace</family>
   <prefer><family>Andale Mono</family></prefer>
</alias>
<match target="pattern">
   <test name="family" mode="eq">
      <string>Courier New</string>
   </test>
   <edit name="family" mode="prepend">
      <string>monospace</string>
   </edit>
</match>
<match target="pattern">
   <test name="family" mode="eq">
      <string>Courier</string>
   </test>
   <edit name="family" mode="prepend">
      <string>monospace</string>
   </edit>
</match>

</fontconfig>

下面这个是Linux系统下改版过来的

<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<!-- /etc/fonts/fonts.conf file to configure system font access -->
<fontconfig>
<!-- 
    Find fonts in these directories
-->
<dir>C:/Windows/Fonts</dir>
<!--
<dir>/usr/X11R6/lib/X11/fonts</dir>
-->
<!--
    Accept deprecated 'mono' alias, replacing it with 'monospace'
-->
<match target="pattern">
    <test qual="any" name="family"><string>mono</string></test>
    <edit name="family" mode="assign"><string>monospace</string></edit>
</match>

<!--
    Load per-user customization file, but don't complain
    if it doesn't exist
-->
<include ignore_missing="yes" prefix="xdg">fontconfig/fonts.conf</include>

<!--
    Load local customization files, but don't complain
    if there aren't any
-->
<include ignore_missing="yes">conf.d</include>
<include ignore_missing="yes">local.conf</include>

<!--
    Alias well known font names to available TrueType fonts.
    These substitute TrueType faces for similar Type1
    faces to improve screen appearance.
-->
<alias>
    <family>Times</family>
    <prefer><family>Times New Roman</family></prefer>
    <default><family>serif</family></default>
</alias>
<alias>
    <family>Helvetica</family>
    <prefer><family>Arial</family></prefer>
    <default><family>sans</family></default>
</alias>
<alias>
    <family>Courier</family>
    <prefer><family>Courier New</family></prefer>
    <default><family>monospace</family></default>
</alias>

<!--
    Provide required aliases for standard names
    Do these after the users configuration file so that
    any aliases there are used preferentially
-->
<alias>
    <family>serif</family>
    <prefer><family>Times New Roman</family></prefer>
</alias>
<alias>
    <family>sans</family>
    <prefer><family>Arial</family></prefer>
</alias>
<alias>
    <family>monospace</family>
    <prefer><family>Andale Mono</family></prefer>
</alias>
</fontconfig>

http://blog.raphaelzhang.com/2013/04/video-streaming-and-ffmpeg-transcoding/

嵌入字幕

在一个MP4文件里面添加字幕,不是把 .srt 字幕文件集成到 MP4 文件里,而是在播放器里选择字幕,这种集成字幕比较简单,速度也相当快

ffmpeg -i input.mp4 -i subtitles.srt -c:s mov_text -c:v copy -c:a copy output.mp4

希望字幕直接显示出来,其实也不难

ffmpeg -i subtitle.srt subtitle.ass
ffmpeg -i input.mp4 -vf ass=subtitle.ass output.mp4

http://blog.neten.de/posts/2013/10/06/use-ffmpeg-to-burn-subtitles-into-the-video/

截图

每隔一秒截一张图

ffmpeg -i out.mp4 -f image2 -vf fps=fps=1 out%d.png

每隔20秒截一张图

ffmpeg -i out.mp4 -f image2 -vf fps=fps=1/20 out%d.png

多张截图合并到一个文件里(2x3)每隔一千帧(秒数=1000/fps25)即40s截一张图

ffmpeg -i out.mp4 -frames 3 -vf "select=not(mod(n\,1000)),scale=320:240,tile=2x3" out.png

从视频中生成GIF图片

ffmpeg -i out.mp4 -t 10 -pix_fmt rgb24 out.gif

转换视频为图片(每帧一张图)

ffmpeg -i out.mp4 out%4d.png

图片转换为视频

ffmpeg -f image2 -i out%4d.png -r 25 video.mp4

添加字幕

ffmpeg -i out.mp4 -vf subtitles=out.srt output.mp4

Java Tutorials

巧妙使用cmd和xls批量重命名文件

巧妙使用cmd和xls批量重命名文件

1. 迅速切换到目标文件目录

C:\Windows\System32>cd /d E:\images

E:\images>

2. 列出目录下的内容重定向到xls文件中

E:\images>dir /b > rename.xls

E:\images>

3. 使用Excel打开rename.xls文件

其内容为:

x1s - 副本 (2).png
x1s - 副本 (3).png
x1s - 副本 (4).png
x1s - 副本 (5).png
x1s - 副本 (6).png
x1s - 副本.png
x1s.png

4. 整理数据

A B
x1s - 副本 (2).png x1s-1.png
x1s - 副本 (3).png x1s-2.png
x1s - 副本 (4).png x1s-3.png
x1s - 副本 (5).png x1s-4.png
x1s - 副本 (6).png x1s-5.png
x1s - 副本.png x1s-6.png
x1s.png x1s-7.png

注:

A列数据为从命令行导出的文件名
B为重命名后的文件名

5. 编写公式,生成重命名批处理脚本

="rename "&""""&A1&""""&" "&B1

="rename "&CHAR(34)&A1CHAR(34)&" "&B1

注:在Excel中单个双引号用四个双引号符号表示即""""或者使用CHAR(34)而单引号则用"'"表示

在C1单元格上输入以上公式,回车后最终生成的数据为

rename "x1s - 副本 (2).png" x1s-1.png

按着C1单元格下拉依次生成,最终将看到下面的内容

A B C
x1s - 副本 (2).png x1s-1.png rename "x1s - 副本 (2).png" x1s-1.png
x1s - 副本 (3).png x1s-2.png rename "x1s - 副本 (3).png" x1s-2.png
x1s - 副本 (4).png x1s-3.png rename "x1s - 副本 (4).png" x1s-3.png
x1s - 副本 (5).png x1s-4.png rename "x1s - 副本 (5).png" x1s-4.png
x1s - 副本 (6).png x1s-5.png rename "x1s - 副本 (6).png" x1s-5.png
x1s - 副本.png x1s-6.png rename "x1s - 副本.png" x1s-6.png
x1s.png x1s-7.png rename "x1s.png" x1s-7.png

其中C列数据则为用公式下拉生成的内容.在xls文件同目录下新建.bat文件,拷贝上面的内容后保存,双击执行即可.

Mac下同时安装多个版本的JDK

Mac自带了的JDK6,安装在目录:/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/下。

JDK7,JDK8则需要自己到Oracle官网下载安装对应的版本。自己安装的JDK默认路径为:/Library/Java/JavaVirtualMachines/jdk1.8.0.jdk

1. 在用户目录下的bash配置文件.bashrc中配置JAVA_HOME的路径:

export JAVA_6_HOME=/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
export JAVA_7_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0.jdk/Contents/Home
export JAVA_8_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0.jdk/Contents/Home
export JAVA_HOME=$JAVA_7_HOME

2. 创建alias命令动态切换JAVA_HOME的配置

alias jdk8='export JAVA_HOME=$JAVA_8_HOME'
alias jdk7='export JAVA_HOME=$JAVA_7_HOME'
alias jdk6='export JAVA_HOME=$JAVA_6_HOME'

3. 验证

CNxnliu:Versions xnliu$ java -version
java version "1.6.0_65"
Java(TM) SE Runtime Environment (build 1.6.0_65-b14-462-11M4609)
Java HotSpot(TM) 64-Bit Server VM (build 20.65-b04-462, mixed mode)
CNxnliu:Versions xnliu$ jdk8
CNxnliu:Versions xnliu$ java -version
java version "1.8.0"
Java(TM) SE Runtime Environment (build 1.8.0-b132)
Java HotSpot(TM) 64-Bit Server VM (build 25.0-b70, mixed mode)
CNxnliu:Versions xnliu$

或使用http://www.jenv.be/

原文链接

Vagrant

配置示例

Vagrant.configure("2") do |config|
  config.vm.define :master do |master|
    master.vm.provider "virtualbox" do |v|
          v.customize ["modifyvm", :id, "--name", "master", "--memory", "1024"]
    end
    master.vm.box = "ubuntu/trusty64"
    master.vm.hostname = "master"
    master.vm.network :public_network, ip: "192.168.9.225"
  end

  config.vm.define :slave do |slave|
    slave.vm.provider "virtualbox" do |v|
          v.customize ["modifyvm", :id, "--name", "slave", "--memory", "1024"]
    end
    slave.vm.box = "ubuntu/trusty64"
    slave.vm.hostname = "slave"
    slave.vm.network :public_network, ip: "192.168.9.226"
  end
end

参考文档

Discover Vagrant Boxes

Vgrant安装配置

使用 Vagrant 打造跨平台开发环境

Vagrant 三种网络配置详解

使用vagrant快速搭建环境

在 Mac/win7 下上使用 Vagrant 打造本地开发环境

解决Windows下使用Vagrant出现的Connection Timeout的问题

vagrant使用及trusty的配置记录

http://kiwenlau.com/2016/07/03/vagrant-vm-cluster/

Hive分组求最小值

准备测试数据

cat /home/hadoop/hemers.txt 
2014050201      新街口
2014050202      新街口
2014050203      新街口
2014050204      鼓楼
2014050205      岔路口
2014050206      岔路口
2014050207      新街口
2014050208      新街口

创建测试数据表

CREATE EXTERNAL TABLE IF NOT EXISTS testdb.hemers(
 date STRING,
 ts STRING
)ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t';

导入数据

LOAD DATA LOCAL INPATH '/home/hadoop/hemers.txt' OVERWRITE INTO TABLE testdb.hemers;

执行SQL

use testdb;

select t1.date,t1.ts from hemers t1
  inner join(
   select dt,count(1)ct1,count(distinct ts)ct2 from (
     select unix_timestamp(date,'yyyyMMddhh') dt,date,ts from hemers a
     union all
     select unix_timestamp(date,'yyyyMMddhh')+3600 dt,date,ts from hemers b
   )c group by dt
   having count(1)=1 or ct2=2
)t2 on unix_timestamp(t1.date,'yyyyMMddhh') =t2.dt;

输出

2014050201      新街口
2014050204      鼓楼
2014050205      岔路口
2014050207      新街口

知乎日报API

Swift Singleton

http://www.synappse.pl/swift-first-steps-singleton/

//
//  Singleton.swift
//  Test
//
//  Created by Sebastian Suchanowski on 6/5/14.
//  Copyright (c) 2014 Synappse. All rights reserved.
//

import Foundation

class Singleton {

    class func sharedInstance() -> Singleton! {
        struct Static {
            static var instance: Singleton? = nil
            static var onceToken: dispatch_once_t = 0
        }

        dispatch_once(&Static.onceToken) {
            Static.instance = self()
        }

        return Static.instance!
    }

    @required init() {

    }

}

HBase Storage and PIG

HBase Storage and PIG

  • Getting Started
export HBASE_HOME=/opt/hbase
export PIG_CLASSPATH="`${HBASE_HOME}/bin/hbase classpath`:$PIG_CLASSPATH"
export HADOOP_CLASSPATH="`${HBASE_HOME}/bin/hbase classpath`:$HADOOP_CLASSPATH"
  • Hello World

write a simple script to load some data from a file and write it out to an HBase table

To begin, use the shell to create your table:

jhoover@jhoover2:~$ hbase shell
HBase Shell; enter ‘help‘ for list of supported commands.
Type “exit” to leave the HBase Shell
Version 0.90.3-cdh3u1, r, Mon Jul 18 08:23:50 PDT 2011

hbase(main):002:0> create ‘sample_names’, ‘info’
0 row(s) in 0.5580 seconds

Next, we’ll put some simple data in a file ‘input.csv’:

1, John, Smith
2, Jane, Doe
3, George, Washington
4, Ben, Franklin

Then we’ll write a simple script to extract this data and write it into fixed columns in HBase:

raw_data = LOAD ‘sample_data.csv’ USING PigStorage( ‘,’ ) AS (
listing_id: chararray,
fname: chararray,
lname: chararray );

STORE raw_data INTO ‘hbase://sample_names’ USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage (
‘info:fname info:lname’);

Then run the pig script locally:

jhoover@jhoover2:~/hbase_sample$ pig -x local hbase_sample.pig
…
Success!

Job Stats (time in seconds):
JobId   Alias   Feature Outputs
job_local_0001  raw_data    MAP_ONLY    hbase://hello_world,

Input(s):
Successfully read records from: “file:///autohome/jhoover/hbase_sample/sample_data.csv”

Output(s):
Successfully stored records in: “hbase://sample_names”

Job DAG:
job_local_0001

You can then see the results of your script in the hbase shell:

hbase(main):001:0> scan ‘hello_world’
ROW COLUMN+CELL
1 column=info:fname, timestamp=1356134399789, value= John
1 column=info:lname, timestamp=1356134399789, value= Smith
2 column=info:fname, timestamp=1356134399789, value= Jane
2 column=info:lname, timestamp=1356134399789, value= Doe
3 column=info:fname, timestamp=1356134399789, value= George
3 column=info:lname, timestamp=1356134399789, value= Washington
4 column=info:fname, timestamp=1356134399789, value= Ben
4 column=info:lname, timestamp=1356134399789, value= Franklin
4 row(s) in 0.4850 seconds
  • Sample Code

You can download the sample code from this blog post here.

  • 文章来源

http://blog.whitepages.com/2011/10/27/hbase-storage-and-pig/

Hive编程指南读书笔记

<<Hive编程指南>>读书笔记

1. 设置hive以本地模式运行(即使当前用户是在分布式模式或伪分布式模式下执行也使用这种模式)

set hive.exec.model.local.auto=true;

若想默认使用这个配置,可以将这个命令添加到$HOME/.hiverc文件中

2. 当频繁使用hadoop dfs命令时,最好为这个命令定义一个别名

alias hdfs="hadoop dfs"

3. hive表数据默认存储位置(基于hadoop的运行模式)

  • hadoop为本地模式:file:///user/hive/warehouse
  • hadoop分布式模式:hdfs://namenode_server/user/hive/warehouse

4. 用户使用下面的命令可以自定义数据仓库的位置

set hive.metastore.warehouse.dir=/user/myname/hive/warehouse;

可以将这条语句写入到$HOME/.hiverc文件中

5. 开启在hive命令行中显示当前数据库名

set hive.cli.print.current.db=true;

6. hive -S 静默输出,在标准输出中不显示其他信息

  • hive -e 后面直接跟HiveQL
  • hive -S -e "select name from person;"

Hive会将输出写到标准输出中,可以使用shell中的输出重定向将输出重定向到本地文件中,而不是hdfs中

7. hive模糊取某个属性名

hive -S -e "set" | grep warehouse 查询数据仓库的位置

8. 从文件中执行Hive查询

hive -f /user/scott/hive.hql

或者在hive shell中使用source命令来执行脚本文件

hive > source /user/scott/hive.hql

9. hiverc文件

hive启动后默认会在当前用户home目录下寻找名为.hiverc的文件,且自动执行文件中的命令

一个典型的.hiverc文件中的内容为

ADD JAR /user/scott/hive-examples.jar;   #增加一个jar文件
set hive.cli.print.current.db=true;     #显示当前所在工作数据库
set hive.exec.model.local.auto=true;    #设置以本地文件模式运行(当hadoop是以分布式模式或者伪分布式模式执行时的话,就在本地执行,可以加快小数据集的查询速度)

注:每行语句结尾的分号不可省略

10. 在hive shell中执行shell命令

用户不需要退出hive就可以执行shell命令,只需要在命令前加上!并且以;结尾就可以,如:

hive > !pwd;  #查看当前目录

注:hive shell中使用shell命令时,不能使用需要用户进行输入交互的命令,且不支持shell的管道功能和文件名自动补全功能。

11. 在hive中使用hadoop命令

hive > dfs -ls /;

12. hive脚本中的注释

-- 用户使用以--开头的字符表示注释。该注释只能放在hiveql脚本中若直接在hive shell中使用的话会报错

13. 让cli显示字段名称

hive > set hive.cli.print.header=true;

同理可以将该语句添加到$home/.hiverc文件中

14. Hive中的数据库

如果用户没有显式指定数据库,默认数据库是default

下面这个例子展示了如何创建数据库

CREATE DATABASE simple;

若数据库simple已经存在的话,将会报错误消息。可以使用下面的语句避免错误信息

CREATE DATABASE IF NOT EXISTS simple;

使用SHOW DATABASES命令查看hive中所包含的数据库

SHOW DATABASES;

如果数据库比较多,可以使用正则表达式匹配筛选所需要的数据库名

SHOW DATABASES LIKE 'sim.*';

Hive为每个数据库创建一个目录,数据库中的表将会以这个数据库目录的子目录形式存储。有一个例外就是default数据库中的表。因为这个数据库本身没有自己的目录。数据库所在的目录位于属性hive.metastore.warehouse.dir所指定的顶层目录之后。假若用户使用的这个配置项的默认配置,也就是/user/hive/warehouse,那么当我们创建数据库simple时,Hive将会对应地创建一个目录为/user/hive/warehouse/simple.db。这里请注意,数据库的文件目录名是以.db结尾的。

用户可以通过如下的命令来修改这个默认的配置

hive > CREATE DATABASE simple LOCATION '/user/myname/directory';

用户还可以为这个数据库增加一个描述信息。这样通过DESCRIBE DATABASE databasename命令就可以查看到该信息

hive > CREATE DATABASE simple COMMENT 'a simple database';

使用DESCRIBE DATABASE databasename查看

hive > DESCRIBE DATABASE simple;
simple a simple database
    hdfs://hostname/user/hive/warehouse/simple.db

从上面的例子可以看出DESCRIBE DATABASE不仅会显示这个数据库的描述信息还会显示这个数据库所在文件的目录位置。若Hadoop是本地模式的话前面的前缀为file:///若是分布式模式则前缀为hdfs://

此外用户还可以为数据库增加一些和其他相关的键-值对属性信息。可以使用DESCRIBE DATABASE EXTENDED databasename语句显示这些信息,如:

hive > CREATE DATABASE simple WITH DBPROERTIES ('creator'='scott','date'='2014-05-05');
hive > DESCRIBE DATABASE simple;
hive > DESCRIBE DATABASE EXTENDED simple;

命令USE用于将某个数据库设置为当前的工作数据库。如:

USE simple;

此时可以用SHOW TABLES显示当前数据库下所有的表。不幸的是,没有那个命令让用户查看当前所在的库。幸运的是在Hive中可以重复使用USE,这是因为在Hive中没有嵌套数据库的概念。

hive > set hive.cli.print.current.db=true;
hive (simple) > USE default;
hive (default) > set hive.cli.print.current.db=false;

最后用户可以删除数据库

hive > DROP DATABASE IF EXISTS simple;

IF EXISTS子句是可选的。可以避免因数据库不存在而抛出警告信息。默认情况下Hive是不允许用户删除一个包含有表的数据仓库。要么用户先删除库中的表,然后再删数据库。要么在删除命令的最后面加上关键字CASCADE,这样Hive先自行删除数据库中的表

hive > DROP DATABASE IF EXISTS simple CASCADE;

若使用的是RESTRICT这个关键字,而不是CASCADE这个关键字的话,那么就和默认情况一样。

如果某个数据库被删除了,那么其对应的目录也同时会被删除。

15. 修改数据库

用户可以使用ALTER DATABASE命令为某个数据库的DBPROPERTIES设置键-值对属性值,来描述数据库的属性信息。数据库的其他元数据信息是不可更改的,包括数据库名和数据库所在的目录位置。

hive > ALTER DATABASE simple SET DBPROPERTIES ('edited-by'='Join');

没有办法删除或者"重置"数据库属性。

16. 创建表

CREATE TABLE IF NOT EXISTS mydb.employees (
name STRING COMMENT 'Employee name',
salary FLOAT COMMENT 'Employee salary',
subordinates ARRAY<STRING> COMMENT 'Names of subordinates',
deductions MAP<STRING, FLOAT>
COMMENT 'Keys are deductions names, values are percentages',
address STRUCT<street:STRING, city:STRING, state:STRING, zip:INT>
COMMENT 'Home address')
COMMENT 'Description of the table'
TBLPROPERTIES ('creator'='me', 'created_at'='2012-01-02 10:00:00')
LOCATION '/user/hive/warehouse/mydb.db/employees';

首先,我们注意到,如果用户当前所处的数据库并非是目标数据库,那么用户是可以在表名前增加一个数据库名来进行指定的,也就是例子中的mydb

如果用户增加上选项IF NOT EXISTS,那么若表已经存在了,Hive就会忽略掉后面的执行语句。且不会有任何提示。用户可以在字段类型后使用COMMENT为每个字段增加一个注释。还可以指定一个或多个表属性。大多数情况下TBLPROPERTIES的主要作用是按键-值对的格式为表增加额外的文档说明。

Hive会自动增加两个表属性:一个是last_modified_by,其保存着最后修改这个表的用户的用户名;另一个是last_modified_time其保存着最后一次修改这个表的新纪元时间秒。

使用SHOW TBLPROPERTIES table_name列举出某个表的TBLPROPERTIES属性信息

最后,可以看到我们根据情况为表中的数据指定一个存储路径。在这个例子中,我们使用Hive默认的路径/user/hive/warehouse/mydb.db/employees,其中,/user/hive/warehouse是默认的数据仓库路径,mydb.db是数据库目录,employees是表目录。

默认情况下。Hive总是将创建的表的目录放置在这个表所属的数据库目录之下。不过,default数据库是个例外,其在/user/hive/warehouse下并没有对应一个数据库目录。因此default数据库中的表目录会直接位于/user/hive/warehouse目录下(用户明确指定除外).

用户还可以拷贝一张已经存在的表的模式(无需拷贝数据):

CREATE TABLE IF NOT EXISTS mydb.employees2
LIKE mydb.employees;

该语句可以接受可选的LOCATION子句,但是注意其他的属性,包括模式都是不可能重新定义的。这些信息直接从原是表获得.

SHOW TABLES命令可以列举出所有的表,如果不增加其他参数,那么只会显示当前工作数据库下的表。假设不在那么数据库下,还是可以列出指定数据库下的表使用SHOW TABLES IN dbname

hive > USE default;
hive > SHOW TABLES IN mydb;
employees
department

如果有很多的表,那么可以使用正则表达式来过滤出所需要的表名.

hive> USE mydb;
hive> SHOW TABLES 'empl.*';
employees

注意:IN databasename和表名使用正则表达式过滤这个两个功能尚不支持同时使用.

我们可以使用DESCRIBE EXTENDED mydb.employees命令来查看这个表的详细结构信息(如果当前所处的工作数据库就是mydb的话,可以不加mydb这个前缀).

hive> DESCRIBE EXTENDED mydb.employees;
name string Employee name
salary float Employee salary
subordinates array<string> Names of subordinates
deductions map<string,float> Keys are deductions names, values are percentages
address struct<street:string,city:string,state:string,zip:int> Home address
Detailed Table Information Table(tableName:employees, dbName:mydb, owner:me,
...
location:hdfs://master-server/user/hive/warehouse/mydb.db/employees,
parameters:{creator=me, created_at='2012-01-02 10:00:00',
last_modified_user=me, last_modified_time=1337544510,
comment:Description of the table, ...}, ...)

使用FORMATTED关键字替代EXTENDED关键字的话,可以提供更加可读的输出信息。在应用中多使用FORMATTED关键字

DESCRIBE FORMATTED employees;

如果用户只想查看某一个列的信息,那么只要在表名后面增加这个字段的名称即可.

hive> DESCRIBE mydb.employees.salary;
salary float Employee salary

注:last_modified_bylast_modified_time两个表属性是自动创建的。如果用户没有定义任何的自定义表属性的话,那么这两个表属性也不会显示在表的详细信息中!

17. 管理表

我们目前所创建的表均属于管理表,有时也被称为内部表.因为这种表,Hive会或多或少的控制着数据项的生命周期.如:Hive默认情况下会将这些表的数据存储在由配置项hive.metastore.warehouse.dir(如/user/hive/warehouse)所定义的目录的子目录下.

当我们删除一个管理表时,Hive也会删除这个表中的数据.管理表不方便和其他工作共享数据。

18. 外部表

CREATE EXTERNAL TABLE IF NOT EXISTS stocks (
exchange STRING,
symbol STRING,
ymd STRING,
price_open FLOAT,
price_high FLOAT,
price_low FLOAT,
price_close FLOAT,
volume INT,
price_adj_close FLOAT)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
LOCATION '/data/stocks';

关键字EXTERNAL告诉Hive这个表是外部的。而后面的LOCATION子句告诉Hive数据位于那个路径下.
因为表是外部的,所以Hive并非认为其完全拥有这份数据,因此,删除该表并不会删除掉这份数据,不过描述表的元数据信息将会被删除掉.

用户可以在DESCRIBE EXTENDED tablename语句的输出中查看到表是管理表还是外部表.在末尾的详细表信息输出中,对于管理表,用户可以看到如下信息:

tableType:MANAGED_TABLE

对于外部表

tableType:EXTERNAL_TABLE

对于管理表,用户可以对一张存在的表进行结构复制(不会复制数据)

CREATE EXTERNAL TABLE IF NOT EXISTS mydb.employees3
LIKE mydb.employees
LOCATION '/path/to/data';

若语句中省略掉EXTERNAL关键字,且源表是外部表的话,那么新生成的表也将是外部表,若语句中省略掉EXTERNAL关键字,且源表是管理表的话,那么新生成的表也将是管理表。
若语句中含有EXTERNAL关键字,且源表是管理表的话,那么生成的新表将是外部表。
即使在这种场景下,LOCATION子句同样是可选的。

19. 分区表、管理表

Hive中有分区表的概念。分区表将数据以一种符合逻辑的方式进行组织。比如分层存储。

CREATE TABLE employees (
name STRING,
salary FLOAT,
subordinates ARRAY<STRING>,
deductions MAP<STRING, FLOAT>,
address STRUCT<street:STRING, city:STRING, state:STRING, zip:INT>
)
PARTITIONED BY (country STRING, state STRING);

如果表中的数据以及分区个数非常大的话,执行一个包含所有分区的查询可能会触发一个巨大的MapReduce任务。建议的安全措施是将Hive设置为“strict”模式,这样对分区表查询WHERE子句没有加分区过滤的话,将会禁止提交这个任务.可以按照下面的语句将属性设置为“nonstrict”模式。

hive> set hive.mapred.mode=strict;

hive> SELECT e.name, e.salary FROM employees e LIMIT 100;
FAILED: Error in semantic analysis: No partition predicate found for
Alias "e" Table "employees"

hive> set hive.mapred.mode=nonstrict;

hive> SELECT e.name, e.salary FROM employees e LIMIT 100;

可以通过使用SHOW PARTITIONS命令查看表中存在的所有分区

hive> SHOW PARTITIONS employees;
...
Country=CA/state=AB
country=CA/state=BC
...
country=US/state=AL
country=US/state=AK

如果表中存在很多的分区,而只想查看是否存储某个特定分区键的分区的话。可以在这个命令上增加一个指定了一个或者多个特定分区字段值的PARTITION子句,进行过滤

hive> SHOW PARTITIONS employees PARTITION(country='US');
country=US/state=AL
country=US/state=AK
...
hive> SHOW PARTITIONS employees PARTITION(country='US', state='AK');
country=US/state=AK

DESCRIBE EXTENDED employees命令也会显示出分区键

hive> DESCRIBE EXTENDED employees;
name string,
salary float,
...
address struct<...>,
country string,
state string
Detailed Table Information...
partitionKeys:[FieldSchema(name:country, type:string, comment:null),
FieldSchema(name:state, type:string, comment:null)],

在管理表中用户可以通过载入数据的方式创建分区。下面的例子将从本地目录()载入数据到表中的时候.将会创建一个US和CA分区.用户需要为每个分区字段指定一个值.注意在HiveQL中是如何引用HOME环境变量的:

LOAD DATA LOCAL INPATH '${env:HOME}/california-employees'
INTO TABLE employees
PARTITION (country = 'US', state = 'CA');

Hive将会创建这个分区对应的目录/employees/country=US/state=CA$HOME/california-employees目录下的文件将会被拷贝到上述分区目录下。

20. 外部分区表

CREATE EXTERNAL TABLE IF NOT EXISTS log_messages (
hms INT,
severity STRING,
server STRING,
process_id INT,
message STRING)
PARTITIONED BY (year INT, month INT, day INT)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t';

分区外部表对LOCATION没有要求,使用ALTER TABLE语句单独进行增加分区.这个语句需要为每一个分区键指定一个值。如:

ALTER TABLE log_messages ADD PARTITION(year = 2012, month = 1, day = 2)
LOCATION 'hdfs://master_server/data/log_messages/2012/01/02';

Hive并不控制这些数据,即使表被删除,数据也不会被删除.

和分区管理表一样.通过SHOW PARTITIONS查看外部表的分区。如:

hive> SHOW PARTITIONS log_messages;
...
year=2011/month=12/day=31
year=2012/month=1/day=1
year=2012/month=1/day=2
...

同样,DESCRIBE EXTENDED log_messages语句会将分区键作为表的模式一部分和partitionKeys列表的内容同时显示

hive> DESCRIBE EXTENDED log_messages;
...
message string,
year int,
month int,
day int
Detailed Table Information...
partitionKeys:[FieldSchema(name:year, type:int, comment:null),
FieldSchema(name:month, type:int, comment:null),
FieldSchema(name:day, type:int, comment:null)],
...

这个输出少了一个非常重要的信息.那就是分区数据实际存在的路径.

通过以下方式查看分区数据所在路径

hive> DESCRIBE EXTENDED log_messages PARTITION (year=2012, month=1, day=2);
...
location:s3n://ourbucket/logs/2011/01/02,
...

通常会使用分区外部表.

21. 自定义表的存储格式

Hive默认的存储格式是文本文件格式.可以通过可选的子句STORED AS TEXTFILE显式指定.同时用户可以在创建表的时指定各种各样的分隔符.

CREATE TABLE employees (
name STRING,
salary FLOAT,
subordinates ARRAY<STRING>,
deductions MAP<STRING, FLOAT>,
address STRUCT<street:STRING, city:STRING, state:STRING, zip:INT>
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\001'
COLLECTION ITEMS TERMINATED BY '\002'
MAP KEYS TERMINATED BY '\003'
LINES TERMINATED BY '\n'
STORED AS TEXTFILE;

注:TEXTFILE意味着所有字段都使用字母、数字、字符编码,包括那么国际字符集.Hive默认是使用不可见字符来作为分隔符的。使用TEXTFILE意味着,每一行被认为是一个单独的记录.可以使用SEQUENCEFILERCFILE两种文件格式来替换TEXTFILE.这两种文件格式都是使用二进制编码和压缩来优化磁盘空间及I/O带宽性能的。

记录编码是通过一个input format对象来控制的。Hive使用了一个名为org.apache.hadoop.mapred.TextInputFormat的java类.

记录的解析是由序列化/反序列化(SerDe)来控制的,对于TEXTFILEHive所使用的SerDeorg.apache.hadoop.hive.serde2.lazy.LazySimpleSerDejava类.

Hive使用一个叫做output format的对象将查询输出写入到文件中或者输出到控制台.对于TEXTFILEHive所使用的输出类为org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat

可以使用第三方的输入输出格式及SerDe,允许用户自定义Hive本身不支持的其他文件格式

CREATE TABLE kst
PARTITIONED BY (ds string)
ROW FORMAT SERDE 'com.linkedin.haivvreo.AvroSerDe'
WITH SERDEPROPERTIES ('schema.url'='http://schema_provider/kst.avsc')
STORED AS
INPUTFORMAT 'com.linkedin.haivvreo.AvroContainerInputFormat'
OUTPUTFORMAT 'com.linkedin.haivvreo.AvroContainerOutputFormat';

ROW FORMAT SERDE …指定了使用的SerDe。Hive提供了WITH SERDEPROPERTIES功能,允许用户传递配置信息给SerDe。每个属性名称和值都应该是带引号的字符串.

STORED AS INPUTFORMAT … OUTPUTFORMAT分别定义了用于输入和输出格式的java类。如果要指定,必须对输入和输出格式都指定.

DESCRIBE EXTENDED table会列出输入和输出格式以及SerDe和SerDe所自带的属性信息。如:

hive> DESCRIBE EXTENDED kst
...
inputFormat:com.linkedin.haivvreo.AvroContainerInputFormat,
outputFormat:com.linkedin.haivvreo.AvroContainerOutputFormat,
...
serdeInfo:SerDeInfo(name:null,
serializationLib:com.linkedin.haivvreo.AvroSerDe,
parameters:{schema.url=http://schema_provider/kst.avsc})
...

22. 删除表

DROP TABLE IF EXISTS employees;

对于管理表,表的元数据信息和表内的数据都被删除

对于外部表,表的元数据信息会被删除,但是表中的数据不会被删除

23. 修改表

大多数的表属性可以通过使用ALTER TABLE语句来修改.该操作会修改元数据,但不会修改数据本身.用于修改表模式中的错误及分区路径.

23.1 表重命名

ALTER TABLE log_messages RENAME TO logmsgs;

23.2 增加、修改和删除表分区

增加表分区

ALTER TABLE log_messages ADD IF NOT EXISTS
PARTITION (year = 2011, month = 1, day = 1) LOCATION '/logs/2011/01/01'
PARTITION (year = 2011, month = 1, day = 2) LOCATION '/logs/2011/01/02'
PARTITION (year = 2011, month = 1, day = 3) LOCATION '/logs/2011/01/03'
...;

修改分区路径

ALTER TABLE log_messages PARTITION(year = 2011, month = 12, day = 2)
SET LOCATION 's3n://ourbucket/logs/2011/01/02';

这个命令不会将数据从旧的路径移走,也不会删除旧的数据

删除分区

ALTER TABLE log_messages DROP IF EXISTS PARTITION(year = 2011, month = 12, day = 2);

对于管理表分区内的数据和元数据会一起被删除。对应外部表,分区内的数据不会被删除

23.3 修改列信息

可以对字段重命名、修改其位置、类型或者注释:

ALTER TABLE log_messages
CHANGE COLUMN hms hours_minutes_seconds INT
COMMENT 'The hours, minutes, and seconds part of the timestamp'
AFTER severity;

即使字段名和字段类型都没有改变,也需要完全指定旧的字段名。并给出新的字段名以及新的字段类型。若要将字段移动到第一个位置,只需要使用FIRST关键字替代AFTER other_column子句即可.

这个操作,只会修改元数据信息。要注意数据与模式匹配.

23.4 增加列

可以在分区字段前增加新的字段到已有字段之后

ALTER TABLE log_messages ADD COLUMNS (
app_name STRING COMMENT 'Application name',
session_id LONG COMMENT 'The current session id');

23.5 删除或者替换列

ALTER TABLE log_messages REPLACE COLUMNS (
hours_mins_secs INT COMMENT 'hour, minute, seconds from timestamp',
severity STRING COMMENT 'The message severity'
message STRING COMMENT 'The rest of the message');

23.6 修改表属性

ALTER TABLE log_messages SET TBLPROPERTIES (
'notes' = 'The process id is no longer captured; this column is always NULL');

可以增加附加的表属性或者修改已经存在的表属性。但是无法删除属性。

23.7 修改存储属性

ALTER TABLE log_messages
PARTITION(year = 2012, month = 1, day = 1)
SET FILEFORMAT SEQUENCEFILE;

如果是分区表,需要使用PARTITION子句

ALTER TABLE table_using_JSON_storage
SET SERDE 'com.example.JSONSerDe'
WITH SERDEPROPERTIES (
'prop1' = 'value1',
'prop2' = 'value2');

23.8 众多的修改表语句

ALTER TABLE … TOUCH语句用于触发钩子

ALTER TABLE log_messages TOUCH
PARTITION(year = 2012, month = 1, day = 1);

ALTER TABLE … ARCHIVE PARTITION将分区内的文件打成一个Hadoop压缩包(HAR)文件.仅仅降低文件系统中的文件数和NameNode的压力,不会减少存储空间

ALTER TABLE log_messages ARCHIVE
PARTITION(year = 2012, month = 1, day = 1);

最后Hive提供了保护,下面的语句防止分区被删除和被查询

ALTER TABLE log_messages
PARTITION(year = 2012, month = 1, day = 1) ENABLE NO_DROP;
ALTER TABLE log_messages
PARTITION(year = 2012, month = 1, day = 1) ENABLE OFFLINE;

vim强制保存root用户下的readonly文件

结合sudo和tee两个命令来实现: w !sudo tee %

:w !sudo tee %

解释如下:

:w – Write a file.

!sudo – Call shell sudo command.

tee – The output of write (vim :w) command redirected using tee.

% – is nothing but current file name

Ubuntu14.0.4+Hadoop2.5.2+jdk1.7源码编译

  • 缺少 libXtst.so.6
Caused by: java.lang.UnsatisfiedLinkError: /opt/jdk1.7/jre/lib/amd64/xawt/libmawt.so: libXtst.so.6: cannot open shared object file: No such file or directory

安装 libxtst6

$ sudo apt-get update
$ sudo apt-get install libxtst6

http://stackoverflow.com/questions/17355863/cant-find-install-libxtst-so-6

之后再执行如下的命令

$ sudo updatedb
$ locate libXtst
/usr/lib/x86_64-linux-gnu/libXtst.so.6
/usr/lib/x86_64-linux-gnu/libXtst.so.6.1.0

Could not find com.google.android.gms:play-services:4.4.52

Gradle构建的时候报以下错误:

Could not find com.google.android.gms:play-services:4.4.52.
Required by:
    google-io-2014:app:unspecified

解决方法:

打开SDK Manager,安装Google RepositoryGoogle Support Repository

iOS: The singleton pattern

person.h

#import <Foundation/Foundation.h>

@interface Person : NSObject 

+ (id)sharedInstance;

@end

person.m

+ (id)sharedInstance {
    static Person *theSharedInstance = nil;
    static dispatch_once_t onceToken;
    dispatch_once(&onceToken, ^{
        theSharedInstance = [[self alloc] init];
    });
    return theSharedInstance;
}

Red Hat Nginx 启动 Script

Should work on RHEL, Fedora, CentOS. Tested on CentOS 5.

/etc/init.d/nginx

#!/bin/sh
#
# nginx - this script starts and stops the nginx daemon
#
# chkconfig:   - 85 15 
# description:  Nginx is an HTTP(S) server, HTTP(S) reverse \
#               proxy and IMAP/POP3 proxy server
# processname: nginx
# config:      /etc/nginx/nginx.conf
# config:      /etc/sysconfig/nginx
# pidfile:     /var/run/nginx.pid

# Source function library.
. /etc/rc.d/init.d/functions

# Source networking configuration.
. /etc/sysconfig/network

# Check that networking is up.
[ "$NETWORKING" = "no" ] && exit 0

nginx="/usr/sbin/nginx"
prog=$(basename $nginx)

NGINX_CONF_FILE="/etc/nginx/nginx.conf"

[ -f /etc/sysconfig/nginx ] && . /etc/sysconfig/nginx

lockfile=/var/lock/subsys/nginx

make_dirs() {
   # make required directories
   user=`$nginx -V 2>&1 | grep "configure arguments:" | sed 's/[^*]*--user=\([^ ]*\).*/\1/g' -`
   if [ -z "`grep $user /etc/passwd`" ]; then
       useradd -M -s /bin/nologin $user
   fi
   options=`$nginx -V 2>&1 | grep 'configure arguments:'`
   for opt in $options; do
       if [ `echo $opt | grep '.*-temp-path'` ]; then
           value=`echo $opt | cut -d "=" -f 2`
           if [ ! -d "$value" ]; then
               # echo "creating" $value
               mkdir -p $value && chown -R $user $value
           fi
       fi
   done
}

start() {
    [ -x $nginx ] || exit 5
    [ -f $NGINX_CONF_FILE ] || exit 6
    make_dirs
    echo -n $"Starting $prog: "
    daemon $nginx -c $NGINX_CONF_FILE
    retval=$?
    echo
    [ $retval -eq 0 ] && touch $lockfile
    return $retval
}

stop() {
    echo -n $"Stopping $prog: "
    killproc $prog -QUIT
    retval=$?
    echo
    [ $retval -eq 0 ] && rm -f $lockfile
    return $retval
}

restart() {
    configtest || return $?
    stop
    sleep 1
    start
}

reload() {
    configtest || return $?
    echo -n $"Reloading $prog: "
    killproc $nginx -HUP
    RETVAL=$?
    echo
}

force_reload() {
    restart
}

configtest() {
  $nginx -t -c $NGINX_CONF_FILE
}

rh_status() {
    status $prog
}

rh_status_q() {
    rh_status >/dev/null 2>&1
}

case "$1" in
    start)
        rh_status_q && exit 0
        $1
        ;;
    stop)
        rh_status_q || exit 0
        $1
        ;;
    restart|configtest)
        $1
        ;;
    reload)
        rh_status_q || exit 7
        $1
        ;;
    force-reload)
        force_reload
        ;;
    status)
        rh_status
        ;;
    condrestart|try-restart)
        rh_status_q || exit 0
            ;;
    *)
        echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}"
        exit 2
esac

参考来源

Install Linux, Apache, MySQL, PHP (LAMP) stack on Ubuntu 14.04

具体操作

  • Step One — Install Apache
sudo apt-get update
sudo apt-get install apache2
  • Step Two — Install MySQL
sudo apt-get install mysql-server php5-mysql
sudo mysql_install_db
sudo mysql_secure_installation
sudo apt-get install php5 libapache2-mod-php5 php5-mcrypt
sudo service apache2 restart
  • Step Three — Install PHP
sudo apt-get install php5-cli

参考地址

https://www.digitalocean.com/community/tutorials/how-to-install-linux-apache-mysql-php-lamp-stack-on-ubuntu-14-04

mongo-hadoop(使用hadoop mapreduce将数据在hdfs和mongodb中导入导出)

ExportToMongoDBFromHDFS(将hdfs中的文件导出到mongo中)

import java.io.*;

import org.apache.commons.logging.*;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.*;
import org.bson.*;
import org.bson.types.ObjectId;


import com.mongodb.hadoop.*;
import com.mongodb.hadoop.util.*;

public class ExportToMongoDBFromHDFS {

    private static final Log log = LogFactory.getLog(ExportToMongoDBFromHDFS.class);

    public static class ReadWeblogs extends Mapper<LongWritable, Text, ObjectId, BSONObject>{

        public void map(Text key, Text value, Context context) throws IOException, InterruptedException{

            System.out.println("Key: " + key);
            System.out.println("Value: " + value);

            String[] fields = value.toString().split("\t");

            String md5 = fields[0];
            String url = fields[1];
            String date = fields[2];
            String time = fields[3];
            String ip = fields[4];

            BSONObject b = new BasicBSONObject();
            b.put("md5", md5);
            b.put("url", url);
            b.put("date", date);
            b.put("time", time);
            b.put("ip", ip);

            context.write( new ObjectId(), b);
        }
    }

    public static void main(String[] args) throws Exception{

        final Configuration conf = new Configuration();
        MongoConfigUtil.setOutputURI(conf,"mongodb://<HOST>:<PORT>/test.weblogs");

        //MongoConfigUtil.setCreateInputSplits(conf, false);
        System.out.println("Configuration: " + conf);

        final Job job = new Job(conf, "Export to Mongo");

        Path in = new Path("/data/weblogs/weblog_entries.txt");
        FileInputFormat.setInputPaths(job, in);

        job.setJarByClass(ExportToMongoDBFromHDFS.class);
        job.setMapperClass(ReadWeblogs.class);

        job.setOutputKeyClass(ObjectId.class);
        job.setOutputValueClass(BSONObject.class);

        job.setInputFormatClass(TextInputFormat.class);
        job.setOutputFormatClass(MongoOutputFormat.class);

        job.setNumReduceTasks(0);

        System.exit(job.waitForCompletion(true) ? 0 : 1 );

    }

将hdfs中的文件导出到mongo中

Ganglia监控Hadoop及Hbase集群性能(安装配置)

Ganglia监控Hadoop及Hbase集群性能(安装配置)

1. 在主节点上安装ganglia-webfrontend和ganglia-monitor

sudo apt-get install ganglia-webfrontend ganglia-monitor

在主节点上安装ganglia-webfrontend和ganglia-monitor。在其他监视节点上,只需要安装ganglia-monitor即可

将ganglia的文件链接到apache的默认目录下

sudo ln -s /usr/share/ganglia-webfrontend /var/www/ganglia

2. 安装ganglia-monitor

在其他监视节点上,只需要安装ganglia-monitor

sudo apt-get install ganglia-monitor

3. Ganglia配置

  • gmond.conf

在每个节点上都需要配置/etc/ganglia/gmond.conf,配置相同如下所示

sudo vim /etc/ganglia/gmond.conf

修改后的/etc/ganglia/gmond.conf

globals {                    
  daemonize = yes  ##以后台的方式运行            
  setuid = yes             
  user = ganglia     #运行Ganglia的用户              
  debug_level = 0               
  max_udp_msg_len = 1472        
  mute = no             
  deaf = no             
  host_dmax = 0 /*secs */ 
  cleanup_threshold = 300 /*secs */ 
  gexec = no             
  send_metadata_interval = 10     #发送数据的时间间隔
} 

/* If a cluster attribute is specified, then all gmond hosts are wrapped inside 
 * of a <CLUSTER> tag.  If you do not specify a cluster tag, then all <HOSTS> will 
 * NOT be wrapped inside of a <CLUSTER> tag. */ 
cluster { 
  name = "hadoop-cluster"         #集群名称
  owner = "ganglia"               #运行Ganglia的用户
  latlong = "unspecified" 
  url = "unspecified" 
} 

/* The host section describes attributes of the host, like the location */ 
host { 
  location = "unspecified" 
} 

/* Feel free to specify as many udp_send_channels as you like.  Gmond 
   used to only support having a single channel */ 
udp_send_channel { 
  #mcast_join = 239.2.11.71     #注释掉组播
  host = master                 #发送给安装gmetad的机器
  port = 8649                   #监听端口
  ttl = 1 
} 

/* You can specify as many udp_recv_channels as you like as well. */ 
udp_recv_channel { 
  #mcast_join = 239.2.11.71     #注释掉组播
  port = 8649 
  #bind = 239.2.11.71 
} 

/* You can specify as many tcp_accept_channels as you like to share 
   an xml description of the state of the cluster */ 
tcp_accept_channel { 
  port = 8649 
} 
  • gmetad.conf

在主节点上还需要配置/etc/ganglia/gmetad.conf,这里面的名字hadoop-cluster和上面gmond.conf中name应该一致。 

/etc/ganglia/gmetad.conf

sudo vim /etc/ganglia/gmetad.conf

修改为以下内容

data_source "hadoop-cluster" 10 master:8649 slave:8649
setuid_username "nobody"
rrd_rootdir "/var/lib/ganglia/rrds"
gridname "hadoop-cluster"

注:master:8649 slave:8649为要监听的主机和端口,data_sourcehadoop-clustergmond.conf中name一致

4. Hadoop配置

在所有hadoop所在的节点,均需要配置hadoop-metrics2.properties,配置如下:

#
#   Licensed to the Apache Software Foundation (ASF) under one or more
#   contributor license agreements.  See the NOTICE file distributed with
#   this work for additional information regarding copyright ownership.
#   The ASF licenses this file to You under the Apache License, Version 2.0
#   (the "License"); you may not use this file except in compliance with
#   the License.  You may obtain a copy of the License at
#
#       http://www.apache.org/licenses/LICENSE-2.0
#
#   Unless required by applicable law or agreed to in writing, software
#   distributed under the License is distributed on an "AS IS" BASIS,
#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#   See the License for the specific language governing permissions and
#   limitations under the License.
#

# syntax: [prefix].[source|sink].[instance].[options]
# See javadoc of package-info.java for org.apache.hadoop.metrics2 for details

#注释掉以前原有配置

#*.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink
# default sampling period, in seconds
#*.period=10

# The namenode-metrics.out will contain metrics from all context
#namenode.sink.file.filename=namenode-metrics.out
# Specifying a special sampling period for namenode:
#namenode.sink.*.period=8

#datanode.sink.file.filename=datanode-metrics.out

# the following example split metrics of different
# context to different sinks (in this case files)
#jobtracker.sink.file_jvm.context=jvm
#jobtracker.sink.file_jvm.filename=jobtracker-jvm-metrics.out
#jobtracker.sink.file_mapred.context=mapred
#jobtracker.sink.file_mapred.filename=jobtracker-mapred-metrics.out

#tasktracker.sink.file.filename=tasktracker-metrics.out

#maptask.sink.file.filename=maptask-metrics.out

#reducetask.sink.file.filename=reducetask-metrics.out

*.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31  
*.sink.ganglia.period=10

*.sink.ganglia.slope=jvm.metrics.gcCount=zero,jvm.metrics.memHeapUsedM=both  
*.sink.ganglia.dmax=jvm.metrics.threadsBlocked=70,jvm.metrics.memHeapUsedM=40  

namenode.sink.ganglia.servers=master:8649  
resourcemanager.sink.ganglia.servers=master:8649  

datanode.sink.ganglia.servers=master:8649    
nodemanager.sink.ganglia.servers=master:8649    


maptask.sink.ganglia.servers=master:8649    
reducetask.sink.ganglia.servers=master:8649

5. Hbase配置

在所有的hbase节点中均配置hadoop-metrics2-hbase.properties,配置如下:

# syntax: [prefix].[source|sink].[instance].[options]
# See javadoc of package-info.java for org.apache.hadoop.metrics2 for details

#*.sink.file*.class=org.apache.hadoop.metrics2.sink.FileSink
# default sampling period
#*.period=10

# Below are some examples of sinks that could be used
# to monitor different hbase daemons.

# hbase.sink.file-all.class=org.apache.hadoop.metrics2.sink.FileSink
# hbase.sink.file-all.filename=all.metrics

# hbase.sink.file0.class=org.apache.hadoop.metrics2.sink.FileSink
# hbase.sink.file0.context=hmaster
# hbase.sink.file0.filename=master.metrics

# hbase.sink.file1.class=org.apache.hadoop.metrics2.sink.FileSink
# hbase.sink.file1.context=thrift-one
# hbase.sink.file1.filename=thrift-one.metrics

# hbase.sink.file2.class=org.apache.hadoop.metrics2.sink.FileSink
# hbase.sink.file2.context=thrift-two
# hbase.sink.file2.filename=thrift-one.metrics

# hbase.sink.file3.class=org.apache.hadoop.metrics2.sink.FileSink
# hbase.sink.file3.context=rest
# hbase.sink.file3.filename=rest.metrics


*.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31  
*.sink.ganglia.period=10  

hbase.sink.ganglia.period=10  
hbase.sink.ganglia.servers=master:8649

6. 启动hadoop、hbase集群

start-dfs.sh
start-yarn.sh
start-hbase.sh

7. 启动Ganglia

先需要重启hadoop和hbase 。在各个节点上启动gmond服务,主节点还需要启动gmetad服务。

使用apt-get方式安装的Ganglia,可以直接用service方式启动。

  • sudo service ganglia-monitor start(每台机器都需要启动)
  • sudo service gmetad start(在安装了ganglia-webfrontend的机器上启动)

8. 检验

登录浏览器查看:http://master/ganglia,如果Hosts up为9即表示安装成功。

若安装不成功,有几个很有用的调试命令:

  • 以调试模式启动gmetad:gmetad -d 9
  • 查看gmetad收集到的XML文件:telnet master 8649

9. 截图



参考文章

#pragma mark in Swift

  • // MARK:
class Connection {
    // MARK:  - NSURLConnectionDataDelegate
    func connectionDidFinishLoading(_: NSURLConnection!) {

    }
}
  • // TODO:
class Connection {
    // MARK:  - NSURLConnectionDataDelegate
    func connectionDidFinishLoading(_: NSURLConnection!) {
    // TODO: process connection
    }
}
  • // FIXME:
class Connection {
    // MARK:  - NSURLConnectionDataDelegate
    func connectionDidFinishLoading(_: NSURLConnection!) {
        // FIXME: do something
        doSomething()
    }
}

Beginning iOS 8 Programming with Swift 读书笔记

1. 设置图片圆角

thumbnailImageView.layer.cornerRadius = thumbnailImageView.frame.size.width / 2
thumbnailImageView.clipsToBounds = true

2. UIAlertController

// Create an option menu as an action sheet
let optionMenu = UIAlertController(title: nil, message: "What do you want to do?",preferredStyle: .ActionSheet)
// Add actions to the menu
let cancelAction = UIAlertAction(title: "Cancel", style: .Cancel, handler: nil)
optionMenu.addAction(cancelAction)
// Display the menu
self.presentViewController(optionMenu, animated: true, completion: nil)

声明并创建闭包,填充UIAlertAction的handler

let callActionHandler = { (action:UIAlertAction!) -> Void in
    let alertMessage = UIAlertController(title: "Service Unavailable", message: "Sorry,the call feature is not available yet. Please retry later.", preferredStyle: .Alert)
    alertMessage.addAction(UIAlertAction(title: "OK", style: .Default, handler: nil))
    self.presentViewController(alertMessage, animated: true, completion: nil)
}
let callAction = UIAlertAction(title: "Call " + "123-000-\(indexPath.row)", style: UIAlertActionStyle.Default, handler: callActionHandler)
optionMenu.addAction(callAction)

直接使用闭包,填充UIAlertAction的handler

let isVisitedAction = UIAlertAction(title: "I've been here", style: .Default, handler: {
    (action:UIAlertAction!) -> Void in
    let cell = tableView.cellForRowAtIndexPath(indexPath)
    cell?.accessoryType = .Checkmark
})
optionMenu.addAction(isVisitedAction)

3. 批量初始化一个数组

var restaurantIsVisited = [Bool](count: 21, repeatedValue: false)

4. 隐藏状态栏

override func prefersStatusBarHidden() -> Bool {
return true
}

info.plist文件中,View controller-based status bar appearance项设为YES,则View controller对status bar的设置优先级高于application的设置。为NO则以application的设置为准,view controller的prefersStatusBarHidden方法无效,是根本不会被调用的。

根据以上描述分以下两种情形:

一.View controller-based status bar appearance设为YES。

这时 view controller中对status bar的设置优先级高于application的设置,用下面的方式隐藏status bar。

分两步实现:

第一步:在view controller中调用setNeedsStatusBarAppearanceUpdate,更新status bar的显示

- (void)viewDidAppear:(BOOL)animated
{
    [super viewDidAppear:animated];

    if ([self respondsToSelector:@selector(setNeedsStatusBarAppearanceUpdate)]) {
        [self prefersStatusBarHidden];
        [self performSelector:@selector(setNeedsStatusBarAppearanceUpdate)];
    }
}

第二步:覆盖view controller的prefersStatusBarHidden的实现,返会YES。

- (BOOL)prefersStatusBarHidden
{
    return YES;
}

二.View controller-based status bar appearance设为NO

这时application的设置优先级最高,用下面的方式隐藏status bar:

[[UIApplication sharedApplication] setStatusBarHidden:YES withAnimation:NO];

结论

  • 如果View controller-based status bar appearance 设为NO,iOS6和iOS7都是用下面的方法隐藏status bar。
[[UIApplication sharedApplication] setStatusBarHidden:YES withAnimation:NO];
  • 如果View controller-based status bar appearance 设为YES,则需要判断当前是iOS6还是iOS7。

如果是iOS6,则还通过sharedApplication隐藏。

如果是iOS7,则用setNeedsStatusBarAppearanceUpdate加prefersStatusBarHidden的方式来隐藏 status bar。

取info.plist中 View controller-based status bar appearance中的设置

NSNumber *isVCBasedStatusBarAppearanceNum = [[NSBundle mainBundle]objectForInfoDictionaryKey:@"UIViewControllerBasedStatusBarAppearance"];
if (isVCBasedStatusBarAppearanceNum) {
    _isVCBasedStatusBarAppearance = isVCBasedStatusBarAppearanceNum.boolValue;
} else {
    _isVCBasedStatusBarAppearance = YES; // default
}

参考链接:http://www.cnblogs.com/machenglong/p/3795876.html

5. UITableView

  • UITableViewDataSource

tableView(_:numberOfRowsInSection:) 控制tableView中section中对应的行数(一个section有多少行)

override func tableView(tableView: UITableView, numberOfRowsInSection section: Int) -> Int
{
// Return the number of rows in the section.
}

tableView(_:cellForRowAtIndexPath:) 定制tableView单元格样式,及填充数据

override func tableView(tableView: UITableView, cellForRowAtIndexPath indexPath: NSIndexPath) -> UITableViewCell {

}

tableView(_:commitEditingStyle:forRowAtIndexPath:) 设置tableView可编辑

override func tableView(tableView: UITableView!, commitEditingStyle editingStyle: UITableViewCellEditingStyle, forRowAtIndexPath indexPath: NSIndexPath!) {

}

控制tableView的section,默认为0

override func numberOfSectionsInTableView(tableView: UITableView) -> Int {
// Return the number of sections.
return 1
}
  • UITableViewDataSource

6. UITableView Delete Row

override func tableView(tableView: UITableView, commitEditingStyle editingStyle: UITableViewCellEditingStyle, forRowAtIndexPath indexPath: NSIndexPath) {
    if editingStyle == .Delete {

        // Delete the row from the data source

        // self.restaurantNames.removeAtIndex(indexPath.row)
        // self.restaurantLocations.removeAtIndex(indexPath.row)
        // self.restaurantTypes.removeAtIndex(indexPath.row)
        // self.restaurantIsVisited.removeAtIndex(indexPath.row)
        // self.restaurantImages.removeAtIndex(indexPath.row)

        //self.tableView.reloadData() 更新tableView,推荐使用后者的代码,具有动画效果

        self.tableView.deleteRowsAtIndexPaths([indexPath], withRowAnimation: .Fade)

    }
}

7. UITableViewRowAction (iOS8新特性)

覆盖tableView(_:editActionsForRowAtIndexPath:)方法

override func tableView(tableView: UITableView, editActionsForRowAtIndexPath indexPath: NSIndexPath) -> [AnyObject] {

        var shareAction = UITableViewRowAction(style: UITableViewRowActionStyle.Default, title:"Share", handler: { (action:UITableViewRowAction!, indexPath:NSIndexPath!) -> Void in

        let shareMenu = UIAlertController(title: nil, message: "Share using",preferredStyle: .ActionSheet)

        let twitterAction = UIAlertAction(title: "Twitter", style:UIAlertActionStyle.Default, handler: nil)

        let facebookAction = UIAlertAction(title: "Facebook", style:UIAlertActionStyle.Default, handler: nil)

        let emailAction = UIAlertAction(title: "Email", style: UIAlertActionStyle.Default,handler: nil)

        let cancelAction = UIAlertAction(title: "Cancel", style: UIAlertActionStyle.Cancel,handler: nil)

        shareMenu.addAction(twitterAction)
        shareMenu.addAction(facebookAction)
        shareMenu.addAction(emailAction)
        shareMenu.addAction(cancelAction)

        self.presentViewController(shareMenu, animated: true, completion: nil)
    })

    var deleteAction = UITableViewRowAction(style: UITableViewRowActionStyle.Default,title: "Delete",handler: { (action:UITableViewRowAction!, indexPath:NSIndexPath!) -> Void in

        // Delete the row from the data source
        self.restaurantNames.removeAtIndex(indexPath.row)
        self.restaurantLocations.removeAtIndex(indexPath.row)
        self.restaurantTypes.removeAtIndex(indexPath.row)
        self.restaurantIsVisited.removeAtIndex(indexPath.row)
        self.restaurantImages.removeAtIndex(indexPath.row)
        self.tableView.deleteRowsAtIndexPaths([indexPath], withRowAnimation: .Fade)

    })

    shareAction.backgroundColor = UIColor(red: 255.0/255.0, green: 166.0/255.0, blue:51.0/255.0, alpha: 1.0)

    deleteAction.backgroundColor = UIColor(red: 51.0/255.0, green: 51.0/255.0, blue:51.0/255.0, alpha: 1.0)

    return [deleteAction, shareAction]
}

8. prepareForSegue

override func prepareForSegue(segue: UIStoryboardSegue, sender: AnyObject!) {
    if segue.identifier == "showRestaurantDetail" {
        if let indexPath = self.tableView.indexPathForSelectedRow() {
            let destinationController = segue.destinationViewController as DetailViewController
            destinationController.restaurantImage = self.restaurantImages[indexPath.row]
        }
    }
}

9. Customizing the Table View Appearance

  • 修改tableView背景色
self.tableView.backgroundColor = UIColor(red: 240.0/255.0, green: 240.0/255.0, blue: 240.0/255.0, alpha: 0.2)
  • 设置tableView单元格透明

tableView(_:cellForRowAtIndexPath:)中添加如下代码(设置单元格透明,使tableView背景色可见):

cell.backgroundColor = UIColor.clearColor()
  • 移除tableView多余的分割线

_在viewDidLoad_方法中设置*

self.tableView.tableFooterView = UIView(frame: CGRectZero)
  • 修改tableView分割线颜色

_在viewDidLoad_方法中设置*

self.tableView.separatorColor = UIColor(red: 240.0/255.0, green: 240.0/255.0, blue: 240.0/255.0,alpha: 0.8)

10. Customizing the Appearance of NavigationBar

  • 修改导航栏背景色
UINavigationBar.appearance().barTintColor = UIColor(red: 231.0/255.0, green: 95.0/255.0, blue: 53.0/255.0, alpha: 0.3)
  • 修改导航栏标题字体大小及颜色
UINavigationBar.appearance().titleTextAttributes = [NSForegroundColorAttributeName:UIColor.whiteColor(),NSFontAttributeName: UIFont(name: "AvenirNextCondensed-DemiBold",size: 22.0)]

iOS Font Name http://iosfonts.com/

  • 修改导航栏返回按钮颜色
UINavigationBar.appearance().tintColor = UIColor.whiteColor()
  • 修改导航栏返回按钮标题
override func viewDidLoad() {
    super.viewDidLoad()
    // Empty back button title
    self.navigationItem.backBarButtonItem = UIBarButtonItem(title: "", style: .Plain, target: nil, action: nil)
}

完整的代码大致如下:

// 设置导航栏背景色
UINavigationBar.appearance().barTintColor = UIColor(red: 231.0/255.0, green: 95.0/255.0, blue: 53.0/255.0, alpha: 0.3)

// 设置导航栏按钮文字颜色
UINavigationBar.appearance().tintColor = UIColor.whiteColor()

// 设置导航栏标题字体大小及颜色
UINavigationBar.appearance().titleTextAttributes =
[NSForegroundColorAttributeName:UIColor.whiteColor(), NSFontAttributeName:UIFont(name:
"AvenirNextCondensed-DemiBold", size: 22.0)]

将以上代码添加到application(_:didFinishLaunchingWithOptions:)方法中

  • 修改导航栏标题

viewDidLoad方法中添加如下代码

title = self.restaurant.name

11. 收缩导航栏 (iOS8新特性)

在视图A中添加如下代码:

override func viewWillAppear(animated: Bool) {
    super.viewWillAppear(animated)
    // 设置隐藏导航栏
    self.navigationController?.hidesBarsOnSwipe = true
}

在视图B中添加如下代码:

override func viewWillAppear(animated: Bool) {
    super.viewWillAppear(animated)
    // 设置不隐藏导航栏
    self.navigationController?.hidesBarsOnSwipe = false
    self.navigationController?.setNavigationBarHidden(false, animated: true)
}

viewDidLoad方法在视图可见或移除时调用。当视图可见时会调用viewWillAppearviewDidAppear方法。viewWillAppear方法在视图将要显示时调用,viewDidAppear在视图已经显示可见后调用。viewWillAppear方法在每次视图可见的时候都会调用。

12. Change the Style of Status Bar(修改状态栏样式)

  • 方法一: 在每个视图中添加如下代码
override func preferredStatusBarStyle() -> UIStatusBarStyle {
    return .LightContent
}
  • 方法二: 基于配置文件和编码

选择项目,在项目属性的info选项中添加新的属性。key为View controller-based status bar appearancevalue为NO。这将影响整个项目

AppDelegate中的application(_:didFinishLaunchingWithOptions:)方法中添加如下代码:

UIApplication.sharedApplication().statusBarStyle = .LightContent

参考链接:http://stackoverflow.com/questions/17678881/how-to-change-status-bar-text-color-in-ios-7

13. Self Sizing Cells (iOS8新特性)

viewDidLoad中添加如下代码:

tableView.estimatedRowHeight = 36.0;//与tableView的rowHeight相等
tableView.rowHeight = UITableViewAutomaticDimension;

注意:同时记得设置Cell中label的lines属性的值为0,默认为1

14. 连线Storyboard退出

@IBAction func close(segue:UIStoryboardSegue) {
}

将Storyboard中视图控制器上的Exit图标与上面的代码关联就好。注意检测类型为unwind segue

15. 设置背景模糊

这里是通过给ViewController添加一个UIImageView控件,然后为UIImageView设置毛玻璃效果

在ViewController创建UIImageView的一个属性引用

@IBOutlet weak var backgroundImageView:UIImageView!

在ViewController的viewDidLoad方法中添加如下代码

var blurEffect = UIBlurEffect(style: UIBlurEffectStyle.Dark)//模糊的样式
var blurEffectView = UIVisualEffectView(effect: blurEffect)//创建UIVisualEffectView
blurEffectView.frame = view.bounds//获取当前ViewController的view.bounds
backgroundImageView.addSubview(blurEffectView)//为backgroundImageView添加蒙板

16. Creating Round Buttons in Interface Builder

在Interface Builder中为UIButton设置圆角(选中UIButton,在属性面板的User Defined Runtime Attributes添加如下的配置)

layer.cornerRadius Number 30

Key Path为layer.cornerRadius

Type为Number

Value为30

17. 设置UIBarButtonItem的颜色及UIToolbar的背景色

AppDelegateapplication(_:willFinishLaunchingWithOptions:)方法中添加如下代码:

// 设置UIBarButtonItem的颜色
UIBarButtonItem.appearance().tintColor = UIColor(red: 235.0/255.0, green: 73.0/255.0, blue: 27.0/255.0, alpha: 1.0)
// 设置UIToolbar的背景色
UIToolbar.appearance().barTintColor = UIColor(red: 237.0/255.0, green: 240.0/255.0, blue: 243.0/255.0, alpha: 0.5)

18. Basic Animations Using UIView

为UIView中添加动画,主要是设置控件的transform属性。

  • CGAffineTransformMakeScale 缩放动画

首先在viewDidLoad中为目标控件设置动画初始值,代码如下:

dialogView.transform = CGAffineTransformMakeScale(0.0, 0.0)

以上代码设置dialogView的transform为CGAffineTransformMakeScale(0.0, 0.0)

接着在viewDidAppear方法中设置该控件动画的结束值

override func viewDidAppear(animated: Bool) {
    UIView.animateWithDuration(0.7, delay: 0.0, options: nil, animations: {
        self.dialogView.transform = CGAffineTransformMakeScale(1, 1)
    }, completion: nil)
}

以上代码表达的意思是设置dialogView等比放大一倍,整个动画持续(或耗时)0.7秒,不延时。

  • Spring Animation (iOS 7)

上述的动画效果用Spring Animation的代码如下(同样在viewDidAppear方法中):

UIView.animateWithDuration(0.7, delay: 0.0, usingSpringWithDamping: 0.5, initialSpringVelocity: 0.5, options: nil, animations: {
    self.dialogView.transform = CGAffineTransformMakeScale(1, 1)
}, completion: nil)
  • Slide Up Animation (CGAffineTransformMakeTranslation(x, y)位移动画)

CGAffineTransformMakeTranslation(x, y)该类动画主要是通过修改控件x,y的坐标值,来达到动画效果

同样首先在viewDidLoad方法中为目标控件设置一个动画状态值(同样是控件的transform属性)

dialogView.transform = CGAffineTransformMakeTranslation(0, 500)

接着在viewDidAppear方法中设置该控件动画的结束值

override func viewDidAppear(animated: Bool) {
    // Spring animation
    UIView.animateWithDuration(0.7, delay: 0.0, usingSpringWithDamping: 0.6, initialSpringVelocity: 0.5, options: nil, animations: {
        self.dialogView.transform = CGAffineTransformMakeTranslation(0, 0)
    }, completion: nil)
}

以上代码的意思是,移动dialogView到(0,0)点,整个动画过程耗时0.7秒,不延迟。在viewDidLoad方法中为dialogView的transform属性设置为CGAffineTransformMakeTranslation(0, 500),紧接着在viewDidAppear方法中为dialogView的transform属性设置为CGAffineTransformMakeTranslation(0, 0),由于viewDidLoad方法在viewDidAppear方法之前调用,一开始dialogView位于(0, 500),随后位移到(0, 0)坐标点,x轴不变,y轴由500缩小到0(垂直方向缩小),由此观察到dialogView是一个Slide Up的动画效果。

  • Combining Two Transforms (动画合并)

顾名思义,就是为一个视图控件,同时绑定多个动画效果,主要通过使用CGAffineTransformConcat(transform1, transform2)来实现。

CGAffineTransformConcat(transform1, transform2)

首先在viewDidLoad方法中定义一个等比缩放动画的初始值及一个位移动画的初始值,代码如下:

let scale = CGAffineTransformMakeScale(0.0, 0.0)
let translate = CGAffineTransformMakeTranslation(0, 500)
dialogView.transform = CGAffineTransformConcat(scale, translate)

接着在viewDidAppear方法中同样定义一个等比缩放动画的结束值及一个位移动画的结束值

UIView.animateWithDuration(0.7, delay: 0.0, usingSpringWithDamping: 0.6, initialSpringVelocity: 0.5, options: nil, animations: {
    let scale = CGAffineTransformMakeScale(1, 1)
    let translate = CGAffineTransformMakeTranslation(0, 0)
    self.dialogView.transform = CGAffineTransformConcat(scale, translate)
}, completion: nil)

上述代码表达的意思是:dialogView等比放大一倍,同时向上移动(y坐标从0改变到500),整个动画过程耗时0.7秒,不延时。通俗了讲就是:等比放大一倍,y坐标从 0 Slide Up 到 500。同理Slide Down为y坐标减小(比如:y坐标从0减小到-500)

在动画这里,需要了解视图的生命周期及与之对应的每一个方法:

ViewController的生命周期中各方法执行流程如下:

init—>loadView—>viewDidLoad—>viewWillApper—>viewDidApper—>viewWillDisapper—>viewDidDisapper—>viewWillUnload->viewDidUnload—>dealloc

注:loadView和viewDidLoad的区别就是,loadView时view还没有生成,viewDidLoad时,view已经生成了,loadView只会被调用一次,而viewDidLoad可能会被调用多次(View可能会被多次加载),当view被添加到其他view中之前,会调用viewWillAppear,之后会调用viewDidAppear。当view从其他view中移除之前,调用viewWillDisAppear,移除之后会调用viewDidDisappear。当view不再使用时,受到内存警告时,ViewController会将view释放并将其指向为nil。

19. MapView

首先添加MapKit framework。选中项目的target,在capabilities选项卡下,开启MapsON即可。

  • 在地图上添加标注
override func viewDidLoad() {
    super.viewDidLoad()
    // Convert address to coordinate and annotate it on map
    let geoCoder = CLGeocoder()
    geoCoder.geocodeAddressString(restaurant.location, completionHandler: { placemarks,
        error in
        if error != nil {
            println(error)
            return
        }
        if placemarks != nil && placemarks.count > 0 {
            let placemark = placemarks[0] as CLPlacemark
            // Add Annotation
            let annotation = MKPointAnnotation()
            annotation.title = self.restaurant.name
            annotation.subtitle = self.restaurant.type
            annotation.coordinate = placemark.location.coordinate
            self.mapView.showAnnotations([annotation], animated: true)
            self.mapView.selectAnnotation(annotation, animated: true)
        }
    })
}
  • 为标注添加图片

首先实现MKMapViewDelegate协议,接着重写mapView(_:viewForAnnotation:)方法。记着在viewDidLoad方法中为mapView设置代理mapView.delegate = self;

func mapView(mapView: MKMapView!, viewForAnnotation annotation: MKAnnotation!) -> MKAnnotationView! {
    let identifier = "MyPin"
    if annotation.isKindOfClass(MKUserLocation) {
        return nil
    }

    // Reuse the annotation if possible
    var annotationView = mapView.dequeueReusableAnnotationViewWithIdentifier(identifier)
    if annotationView == nil {
        annotationView = MKAnnotationView(annotation: annotation, reuseIdentifier: identifier)
        annotationView.canShowCallout = true
    }
    let leftIconView = UIImageView(frame: CGRectMake(0, 0, 47, 47))
    leftIconView.image = UIImage(named: restaurant.image)
    annotationView.leftCalloutAccessoryView = leftIconView
    return annotationView
}

20. Static Table View and UIImagePickerController

  • Static Table View

首先拖一个UITableViewController,然后在tableView的属性栏中修改tableViewContent属性为Static Cells。默认会创建三个静态空白的Cell。

  • Displaying Photo Library Using UIImagePickerController
if UIImagePickerController.isSourceTypeAvailable(.PhotoLibrary) {
    let imagePicker = UIImagePickerController()
    imagePicker.allowsEditing = false
    imagePicker.sourceType = .PhotoLibrary
    self.presentViewController(imagePicker, animated: true, completion: nil)
}

注:若指定imagePicker的sourceType为.Camera,则为照相模式。

获取用户选择的照片,需要实现UIImagePickerControllerDelegate协议,需要实现imagePickerController(_:didFinishPickingMediaWithInfo:)方法。具体代码如下:

@IBOutlet weak var imageView:UIImageView!

func imagePickerController(picker: UIImagePickerController!, didFinishPickingImage image: UIImage!, editingInfo: [NSObject : AnyObject]!) {
    imageView.image = image
    imageView.contentMode = UIViewContentMode.ScaleAspectFill
    imageView.clipsToBounds = true
    dismissViewControllerAnimated(true, completion: nil)//dismiss image picker
}

注:记得在viewDidLoad方法中为UIImagePickerController设置imagePicker.delegate = self

注意:此处有一个bug,之前设置了状态栏的文字及背景色会失效,此处需要修复,需要实现UINavigationControllerDelegate协议,具体代码如下:

func navigationController(navigationController: UINavigationController!, willShowViewController viewController: UIViewController!, animated: Bool) {
    UIApplication.sharedApplication().setStatusBarStyle(.LightContent, animated: false)
}

注:记得为navigationController设置代理

21. Core Data

1. 首先创建Data Model。Core Data -> Data Model -> FoodPin.xcdatamodeld
2. Add Entity(创建EntityEntity与实体名称对应,比如说该示例中的Restaurant)
3. Add attributes(添加属性)
name        String
type        String
location    String
image       Binary Data
isVisited   Boolean

注:以上二三步是创建Data Model,紧接着,第四部创建Data Object

4. Create Data Object
import Foundation
import CoreData

class Restaurant:NSManagedObject {
    @NSManaged var name:String!
    @NSManaged var type:String!
    @NSManaged var location:String!
    @NSManaged var image:NSData!
    @NSManaged var isVisited:NSNumber!
}

注:在上面的Data Object定义中Binary Data使用NSData类型,Boolean使用NSNumber来定义。当使用NSNumber来表达Boolean类型时,非零的值表示为true,零为false

5. Working with Managed Objects
5.1. Get the managed object context from AppDelegate
let managedObjectContext = (UIApplication.sharedApplication().delegate as AppDelegate).managedObjectContext

注:获取managedObjectContext

5.2. Create a managed object for the Restaurant entity
NSEntityDescription.insertNewObjectForEntityForName("Restaurant", inManagedObjectContext: managedObjectContext) as Restaurant

注:此处的RestaurantEntity

5.3. Use the context to save the new object into database
managedObjectContext.save(&e)

具体代码如下:

import CoreData

if let managedObjectContext = (UIApplication.sharedApplication().delegate as AppDelegate).managedObjectContext {

    restaurant = NSEntityDescription.insertNewObjectForEntityForName("Restaurant", inManagedObjectContext: managedObjectContext) as Restaurant

    restaurant.name = nameTextField.text
    restaurant.type = typeTextField.text
    restaurant.location = locationTextField.text
    restaurant.image = UIImagePNGRepresentation(imageView.image)
    restaurant.isVisited = isVisited.boolValue//此处与书中有出入

    var e: NSError?

    if managedObjectContext.save(&e) != true {
        println("insert error: \(e!.localizedDescription)")
        return
    }
}

注:UIImagePNGRepresentation,将Image转化为NSData

5.4. Fetching Data Using Core Data
  • 简便的方法(在viewWillAppear方法中添加如下代码:)
if let managedObjectContext = (UIApplication.sharedApplication().delegate as AppDelegate).managedObjectContext {
    let fetchRequest = NSFetchRequest(entityName: "Restaurant")
    var e: NSError?
    restaurants = managedObjectContext.executeFetchRequest(fetchRequest, error: &e) as [Restaurant]
    if e != nil {
        println("Failed to retrieve record: \(e!.localizedDescription)")
    }
}
  • 使用NSFetchedResultsController (实现NSFetchedResultsControllerDelegate协议)

具体代码如下:

import CoreData

class RestaurantTableViewController:UITableViewController, NSFetchedResultsControllerDelegate {

    var fetchResultController:NSFetchedResultsController!

    override func viewDidLoad() {
        super.viewDidLoad()

        var fetchRequest = NSFetchRequest(entityName: "Restaurant")
        let sortDescriptor = NSSortDescriptor(key: "name", ascending: true)
        fetchRequest.sortDescriptors = [sortDescriptor]

        if let managedObjectContext = (UIApplication.sharedApplication().delegate as AppDelegate).managedObjectContext {

            fetchResultController = NSFetchedResultsController(fetchRequest: fetchRequest, managedObjectContext: managedObjectContext, sectionNameKeyPath: nil, cacheName: nil)

            fetchResultController.delegate = self
            var e: NSError?

            var result = fetchResultController.performFetch(&e)
            restaurants = fetchResultController.fetchedObjects as [Restaurant]

            if result != true {
                println(e?.localizedDescription)
            }
        }
    }
}

使用NSFetchedResultsController,若内容发生改变时,将自动调用NSFetchedResultsControllerDelegate的以下几个方法:

  • controllerWillChangeContent(_:)
  • controller(_:didChangeObject:atIndexPath:forChangeType:newIndexPath:)
  • controllerDidChangeContent(_:)

其对应的调用顺序以上文的顺序自上而下依次调用。

controllerWillChangeContent(_:)

func controllerWillChangeContent(controller: NSFetchedResultsController!) {
    tableView.beginUpdates()
}

controller(_:didChangeObject:atIndexPath:forChangeType:newIndexPath:)

func controller(controller: NSFetchedResultsController!, didChangeObject anObject: AnyObject!, atIndexPath indexPath: NSIndexPath!, forChangeType type: NSFetchedResultsChangeType, newIndexPath: NSIndexPath!) {

    switch type {
        case .Insert:
            tableView.insertRowsAtIndexPaths([newIndexPath], withRowAnimation: .Fade)//注意:第一个参数为[newIndexPath]
        case .Delete:
            tableView.deleteRowsAtIndexPaths([indexPath], withRowAnimation: .Fade)
        case .Update:
            tableView.reloadRowsAtIndexPaths([indexPath], withRowAnimation: .Fade)
        default:
            tableView.reloadData()
    }
    restaurants = controller.fetchedObjects as [Restaurant]
}

controllerDidChangeContent(_:)

func controllerDidChangeContent(controller: NSFetchedResultsController!) {
    tableView.endUpdates()
}
5.5. Deleting Data Using Core Data
managedObjectContext.deleteObject(restaurantToDelete)

具体代码如下:

var deleteAction = UITableViewRowAction(style: UITableViewRowActionStyle.Default, title: "Delete",handler: {

    (action:UITableViewRowAction!, indexPath:NSIndexPath!) -> Void in

    // Delete the row from the data source
    if let managedObjectContext = (UIApplication.sharedApplication().delegate as AppDelegate).managedObjectContext {

        let restaurantToDelete = self.fetchResultController.objectAtIndexPath(indexPath) as Restaurant

        managedObjectContext.deleteObject(restaurantToDelete)

        var e: NSError?

        if managedObjectContext.save(&e) != true {
            println("delete error: \(e!.localizedDescription)")
        }

    }
})
6. Viewing the Raw SQL Statement

选择Stop按钮右边的项目名称,选择Edit Scheme,选择Arguments选项卡,在Argument Passed on Launch选项下添加如下参数:

-com.apple.CoreData.SQLDebug 1

点击OK后,再次运行即可在控制台看到真实的SQL输出。

22. Search Bar (UISearchController iOS 8新特性)

在iOS 8中使用UISearchController替换 UISearchDisplayController

  • Using UISearchController

使用UISearchController的代码大致如下:

searchController = UISearchController(searchResultsController: nil)
searchController.searchResultsUpdater = self//需要实现UISearchResultsUpdating协议
tableView.tableHeaderView = searchController.searchBar
definesPresentationContext = true
  • Adding Search Bar

具体代码如下:

searchController = UISearchController(searchResultsController: nil)
searchController.searchBar.sizeToFit()
searchController.searchResultsUpdater = self//需要实现UISearchResultsUpdating协议
tableView.tableHeaderView = searchController.searchBar
definesPresentationContext = true

注:上述代码是为tableView添加searchBar(充当tableHeaderView)

  • Filtering Content
var searchResults:[Restaurant] = []

func filterContentForSearchText(searchText: String) {

    searchResults = restaurants.filter({ ( restaurant: Restaurant) -> Bool in
        let nameMatch = restaurant.name.rangeOfString(searchText, options: NSStringCompareOptions.CaseInsensitiveSearch)
        let locationMatch = restaurant.location.rangeOfString(searchText, options: NSStringCompareOptions.CaseInsensitiveSearch)
        return nameMatch != nil || locationMatch != nil
    })

}

注:以上代码是通过使用数组的filter方法来实现过滤

  • Updating Search Results

实现UISearchResultsUpdating协议。

viewDidLoad方法中添加以下代码:

searchController.searchResultsUpdater = self
searchController.dimsBackgroundDuringPresentation = false

重写updateSearchResultsForSearchController方法,代码大致如下:

func updateSearchResultsForSearchController(searchController: UISearchController) {
    let searchText = searchController.searchBar.text//获取检索字符
    filterContentForSearchText(searchText)//内容过滤
    tableView.reloadData()//更新tableView
}
  • Customizing the Appearance of Search Bar(个性化定制Search Bar外观)
searchController.searchbar.tintColor //searchbar文字颜色
searchController.searchbar.placeholder //searchbar占位提示字符内容
searchController.searchbar.prompt //位于searchbar上方的文字内容
searchController.searchbar.barTintColor //searchbar背景色

23. UIPageViewController

UIPageViewController提供verticalhorizontal两种样式,其过渡样式又分为Page CurlScroll两种,默认为Page Curl

首先在Storyboard中拖入一个PageViewController。(并为其指定Storyboard IDPageViewController),事实上PageViewController为一个容器,用来控制和显示具体的PageView。一般将这个视图称为PageContentViewController,在该视图上设计要显示的内容。同样在在Storyboard中拖入一个View Controller设置其Storyboard ID为PageContentViewController。拖入两个label和一个imageview填充PageContentViewController,作为PageViewController要控制显示的视图。

  • 创建PageContentViewControllerclass继承UIViewController,并将UI与代码关联。其代码大致如下:
@IBOutlet weak var headingLabel:UILabel!//大标题
@IBOutlet weak var subHeadingLabel:UILabel!//二级标题
@IBOutlet weak var contentImageView:UIImageView!//图片

var index : Int = 0 //索引,标识当前PageContentViewController的索引
var heading : String = ""
var imageFile : String = ""
var subHeading : String = ""

override func viewDidLoad() {
    super.viewDidLoad()
    headingLabel.text = heading
    subHeadingLabel.text = subHeading
    contentImageView.image = UIImage(named: imageFile)
}
  • 创建PageViewControllerclass继承UIPageViewController,并且实现UIPageViewControllerDataSource协议。其代码大致如下:

PageViewController继承UIPageViewController并且实现UIPageViewControllerDataSource协议,该类主要控制及显示具体的PageContentView(PageContentViewController),通过UIPageViewControllerDataSource协议中的两个方法来控制其显示。

pageViewController(_:viewControllerBeforeViewController:)//上一个PageContentView

pageViewController(_:viewControllerAfterViewController:)//下一个PageContentView

class PageViewController: UIPageViewController, UIPageViewControllerDataSource{

    var pageHeadings = ["Personalize", "Locate", "Discover"]//大标题
    var pageImages = ["homei", "mapintro", "fiveleaves"]//图片名称
    var pageSubHeadings = ["Pin your favourite restaurants and create your own food guide", "Search and locate your favourite restaurant on Maps", "Find restaurants pinned by your friends and other foodies around the world"]//二级标题


    override func viewDidLoad() {
        super.viewDidLoad()
        // Set the data source to itself
        dataSource = self
        // Create the first walkthrough screen 创建第一个PageContentView
        if let startingViewController = self.viewControllerAtIndex(0) {
            setViewControllers([startingViewController], direction: .Forward, animated: true, completion: nil)
        }
    }


    //下一个PageContentView
    func pageViewController(pageViewController: UIPageViewController, viewControllerAfterViewController viewController: UIViewController) -> UIViewController? {
        var index = (viewController as PageContentViewController).index
        index++
        return self.viewControllerAtIndex(index)
    }

    //上一个PageContentView
    func pageViewController(pageViewController: UIPageViewController, viewControllerBeforeViewController viewController: UIViewController) -> UIViewController? {
        var index = (viewController as PageContentViewController).index
        index--
        return self.viewControllerAtIndex(index)
    }

    //控制PageContentView的轮询切换
    func viewControllerAtIndex(index: Int) -> PageContentViewController? {
        if index == NSNotFound || index < 0 || index >= self.pageHeadings.count {
            return nil
        }
        // Create a new view controller and pass suitable data.
        if let pageContentViewController =
            storyboard?.instantiateViewControllerWithIdentifier("PageContentViewController") as? PageContentViewController {
            pageContentViewController.imageFile = pageImages[index]
            pageContentViewController.heading = pageHeadings[index]
            pageContentViewController.subHeading = pageSubHeadings[index]
            pageContentViewController.index = index
            return pageContentViewController
        }
        return nil
    }

    /*************************************默认的Page Indicator*********************************************/
    func presentationCountForPageViewController(pageViewController: UIPageViewController) -> Int {
        return pageHeadings.count
    }

    func presentationIndexForPageViewController(pageViewController: UIPageViewController) -> Int {
        if let pageContentViewController = storyboard?.instantiateViewControllerWithIdentifier("PageContentViewController") as? PageContentViewController {
            return pageContentViewController.index
        }
        return 0
    }

}

注:类PageViewController创建和控制PageContentViewstoryboard?.instantiateViewControllerWithIdentifier("PageContentViewController") as? PageContentViewController,使用storyboard通过在storyboard中为ViewController设置的Storyboard ID获取ViewController(PageContentViewController)

  • Display(使用)
if let pageViewController = storyboard?.instantiateViewControllerWithIdentifier("PageViewController") as? PageViewController {
    self.presentViewController(pageViewController, animated: true, completion: nil)
}

通过storyboard?.instantiateViewControllerWithIdentifier("PageViewController") as? PageViewController获取PageViewController(PageView控制器),最终使用presentViewController显示。

  • 添加默认的Page Indicator

通过在PageViewController中实现UIPageViewControllerDataSource中的以下两个方法实现:

presentationCountForPageViewController PageContentView总个数

presentationIndexForPageViewController 当前选中的PageContentView的索引

其代码大致如下:

func presentationCountForPageViewController(pageViewController: UIPageViewController) -> Int {
    return pageHeadings.count
}

func presentationIndexForPageViewController(pageViewController: UIPageViewController) -> Int {
    if let pageContentViewController = storyboard?.instantiateViewControllerWithIdentifier("PageContentViewController") as? PageContentViewController {
        return pageContentViewController.index
    }
    return 0
}

注:dismissViewControllerAnimated(true, completion: nil)关闭或销毁当前ViewController

  • Custom Page Indicator(自定义Page Indicator)

通过UIPageControl控件实现。(略)

  • NSUserDefaults

使用NSUserDefaults.standardUserDefaults()获取NSUserDefaults对象。

通过下面的方法检索值。

arrayForKey(_:)
boolForKey(_:)
dataForKey(_:)
dictionaryForKey(_:)
floatForKey(_:)
integerForKey(_:)
objectForKey(_:)
stringArrayForKey(_:)
stringForKey(_:)
doubleForKey(_:)
URLForKey(_:)

大致代码如下:

let defaults = NSUserDefaults.standardUserDefaults()
defaults.setBool(true, forKey: "hasViewedWalkthrough")//存放一个Boolean值,其值为true,键为hasViewedWalkthrough

24. Tab Bar

select the Navigation Controller(Initial View Controller) -> select Editor > Embed in > Tab Bar Controller.

  • Hide Tab Bar When Pushed

在使用Navigation Controller push后的ViewController中隐藏Tab Bar有以下两种方法:

第一种方法:

StoryBoard中选择目标ViewController在Attribute Inspector选项中勾选Hide Bottom Bar on Push

第二种方法:

prepareForSegue方法中设置destinationControllerhideBottomBarWhenPushed属性为true

destinationController.hideBottomBarWhenPushed = true
  • Customizing the Appearance of Tab Bar

tintColor 文字颜色

UITabBar.appearance().tintColor = UIColor(red: 235.0/255.0, green: 75.0/255.0, blue: 27.0/255.0, alpha: 1.0)

barTintColor 背景色

UITabBar.appearance().barTintColor = UIColor.blackColor()

Tab Bar Item Image

修改TabBar选项卡的图片,选中该选项,在Attribute Inspector选项中修改system item选项为Custom然后设置TitleImage属性。

Selection Indicator Image 设置选中后的图片

UITabBar.appearance().selectionIndicatorImage = UIImage(named: "tabitem_selected")

25. WebView and Email

  • Loading Web Content Using UIWebView
let url = NSURL(string: "http://www.appcoda.com")
//let url = NSURL(fileURLWithPath: "about.html")
let request = NSURLRequest(URL: url)
webView.loadRequest(request)
  • MFMailComposeViewController

使用MFMailComposeViewController发送邮件,实现MFMailComposeViewControllerDelegate协议中的mailComposeController(_:didFinishWithResult:error:)方法

import MessageUI

class AboutViewController: UIViewController, MFMailComposeViewControllerDelegate, UINavigationControllerDelegate {

    // 点击后触发写邮件界面
    @IBAction func sendEmail (sender: AnyObject) {
        if MFMailComposeViewController.canSendMail() {
            var composer = MFMailComposeViewController()

            composer.mailComposeDelegate = self
            composer.setToRecipients(["[email protected]"])
            composer.navigationBar.tintColor = UIColor.whiteColor()

            presentViewController(composer, animated: true, completion: {
                UIApplication.sharedApplication().setStatusBarStyle(.LightContent, animated: false)
            })
        }
    }

    func mailComposeController(controller: MFMailComposeViewController!, didFinishWithResult result: MFMailComposeResult, error: NSError!) {

        switch result.value {
            case MFMailComposeResultCancelled.value:
                println("Mail cancelled")
            case MFMailComposeResultSaved.value:
                println("Mail saved")
            case MFMailComposeResultSent.value:
                println("Mail sent")
            case MFMailComposeResultFailed.value:
                println("Failed to send mail: \(error.localizedDescription)")
            default:
                break
        }
        // Dismiss the Mail interface
        dismissViewControllerAnimated(true, completion: nil)
    }
}

26. CloudKit

  • Enabling CloudKit in Your App(启用CloudKit)

Targets -> Capabilities -> iCloud -> ON -> CloudKit

选择Targets,切换到Capabilities选项卡,在iCloud选项上选择ON,并且选择iCloud选项下方的Services属性为CloudKit

  • Managing Your Record in CloudKit Dashboard

使用CloudKit Dashboard来管理和创建(具体与CoreData的用法类似)

在左侧的面板区域,选择Record Types,点右边的+创建一个Record Type(如:Restaurant),接着定义attribute。CloudKit支持String,Data/Time,Double,Location,Asset(存放图片)等类型。

本书例子中的属性定义对应如下:

name String
type String
location String
image Asset

定义好属性后,可以使用面板上的+添加数据。

  • Fetching Data from Public Database Using Convenience API

大致代码如下所示:

let cloudContainer = CKContainer.defaultContainer()
let publicDatabase = CKContainer.defaultContainer().publicCloudDatabase
let predicate = NSPredicate(value: true)
let query = CKQuery(recordType: "Restaurant", predicate: predicate) //Restaurant为在iCloud中创建的Record Type
publicDatabase.performQuery(query, inZoneWithID: nil, completionHandler: { results, error in
// Process the records
})

具体代码片段:

import CloudKit

var restaurants:[CKRecord] = []
self.getRecordsFromCloud()

func getRecordsFromCloud() {

    // Fetch data using Convenience API
    let cloudContainer = CKContainer.defaultContainer()
    let publicDatabase = CKContainer.defaultContainer().publicCloudDatabase
    let predicate = NSPredicate(value: true)
    let query = CKQuery(recordType: "Restaurant", predicate: predicate) //Restaurant为在iCloud中创建的Record Type
    publicDatabase.performQuery(query, inZoneWithID: nil, completionHandler: { results, error in
        if error == nil {
            println("Completed the download of Restaurant data")
            self.restaurants = results as [CKRecord] //将结果转化为[CKRecord]
            //self.tableView.reloadData() //更新tableView数据源

            // 使用dispatch_async优化代码。在主线程中异步更新tableView数据源
            dispatch_async(dispatch_get_main_queue(), {
                self.tableView.reloadData() //更新tableView数据源
            })

        } else {
            println(error)
        }
    })

}

override func numberOfSectionsInTableView(tableView: UITableView) -> Int {
    // Return the number of sections.
    return 1
}

override func tableView(tableView: UITableView, numberOfRowsInSection section: Int) -> Int {
    // Return the number of rows in the section.
    return restaurants.count
}

override func tableView(tableView: UITableView, cellForRowAtIndexPath indexPath: NSIndexPath) -> UITableViewCell {
    let cell = tableView.dequeueReusableCellWithIdentifier("Cell", forIndexPath: indexPath) as UITableViewCell
    // Configure the cell...
    let restaurant = restaurants[indexPath.row]
    cell.textLabel.text = restaurant.objectForKey("name") as? String //获取iCloud中创建的属性为name值

    if (restaurant.objectForKey("image") != nil) {
        let imageAsset = restaurant.objectForKey("image") as CKAsset //获取iCloud中创建的属性为image值并转化为CKAsset
        cell.imageView?.image = UIImage(data: NSData(contentsOfURL: imageAsset.fileURL))
    }
    return cell
}
  • Fetching Data from Public Database Using Operational API

替换getRecordsFromCloud方法:

func getRecordsFromCloud() {

    // Initialize an empty restaurants array
    restaurants = []

    // Get the Public iCloud Database
    let cloudContainer = CKContainer.defaultContainer()
    let publicDatabase = CKContainer.defaultContainer().publicCloudDatabase

    // Prepare the query
    let predicate = NSPredicate(value: true)
    let query = CKQuery(recordType: "Restaurant", predicate: predicate)

    // Create the query operation with the query
    let queryOperation = CKQueryOperation(query: query)
    queryOperation.desiredKeys = ["name", "image"]
    queryOperation.queuePriority = .VeryHigh
    queryOperation.resultsLimit = 50

    queryOperation.recordFetchedBlock = { (record:CKRecord!) -> Void in
        if let restaurantRecord = record {
            self.restaurants.append(restaurantRecord)
        }
    }

    queryOperation.queryCompletionBlock = { (cursor:CKQueryCursor!, error:NSError!) -> Void in
        if (error != nil) {
            println("Failed to get data from iCloud - \(error.localizedDescription)")
        } else {
            println("Successfully retrieve the data from iCloud")

            dispatch_async(dispatch_get_main_queue(), {
                self.tableView.reloadData()
            })
        }
    }

    // Execute the query
    publicDatabase.addOperation(queryOperation)
}
  • Activity Indicator (UIActivityIndicatorView)

显示UIActivityIndicatorView

var spinner:UIActivityIndicatorView = UIActivityIndicatorView()
spinner.activityIndicatorViewStyle = .Gray //设置样式为Gray
spinner.center = self.view.center //局中显示
spinner.hidesWhenStopped = true //设置停止的时候可隐藏
self.parentViewController?.view.addSubview(spinner) //添加到父视图控制器中
spinner.startAnimating() //显示UIActivityIndicatorView

UIActivityIndicatorView有三种样式:GrayWhite (default)WhiteLarge

隐藏或关闭UIActivityIndicatorView

dispatch_async(dispatch_get_main_queue(), {
    self.spinner.stopAnimating() //在主线程中调用stopAnimating()隐藏或关闭UIActivityIndicatorView
})
  • Lazy Loading Images(Image懒加载)

修改上文中的getRecordsFromCloud方法

queryOperation.desiredKeys = ["name", "image"]
//修改为
queryOperation.desiredKeys = ["name"]

修改上文中的tableView(_:cellForRowAtIndexPath:)方法

override func tableView(tableView: UITableView, cellForRowAtIndexPath indexPath: NSIndexPath) -> UITableViewCell {
    let cell = tableView.dequeueReusableCellWithIdentifier("Cell", forIndexPath: indexPath) as UITableViewCell
    if restaurants.isEmpty {
        return cell
    }
    // Configure the cell...
    let restaurant = restaurants[indexPath.row]
    cell.textLabel.text = restaurant.objectForKey("name") as? String

    // Set default image
    cell.imageView.image = UIImage(named: "camera")

    // Fetch Image from Cloud in background
    let publicDatabase = CKContainer.defaultContainer().publicCloudDatabase
    let fetchRecordsImageOperation = CKFetchRecordsOperation(recordIDs: [restaurant.recordID])
    fetchRecordsImageOperation.desiredKeys = ["image"]
    fetchRecordsImageOperation.queuePriority = .VeryHigh
    fetchRecordsImageOperation.perRecordCompletionBlock = {(record:CKRecord!, recordID:CKRecordID!, error:NSError!) ->  Void in
        if (error != nil) {
            println("Failed to get restaurant image: \(error.localizedDescription)")
        } else {
            if let restaurantRecord = record {
                dispatch_async(dispatch_get_main_queue(), { //后台异步加载image
                    let imageAsset = restaurantRecord.objectForKey("image") as CKAsset
                    cell.imageView.image = UIImage(data: NSData(contentsOfURL: imageAsset.fileURL)!)
                })
            }
        }
    }
    publicDatabase.addOperation(fetchRecordsImageOperation)
    return cell
}
  • Caching Images Using NSCache(使用缓存缓存Image)
var imageCache:NSCache = NSCache()

override func tableView(tableView: UITableView, cellForRowAtIndexPath indexPath: NSIndexPath) -> UITableViewCell {

    let cell = tableView.dequeueReusableCellWithIdentifier("Cell", forIndexPath: indexPath) as UITableViewCell
    if restaurants.isEmpty {
        return cell
    }

    // Configure the cell...
    let restaurant = restaurants[indexPath.row]
    cell.textLabel.text = restaurant.objectForKey("name") as? String

    // Set default image
    cell.imageView.image = UIImage(named: "camera")

    // See if we can get the image from cache 检测Cache中是否存在image
    if let imageFileURL = imageCache.objectForKey(restaurant.recordID) as? NSURL {
        println("Get image from cache")
        cell.imageView.image = UIImage(data: NSData(contentsOfURL: imageFileURL)!)
    } else {
        // Fetch Image from Cloud in background
        let publicDatabase = CKContainer.defaultContainer().publicCloudDatabase
        let fetchRecordsImageOperation = CKFetchRecordsOperation(recordIDs: [restaurant.recordID])
        fetchRecordsImageOperation.desiredKeys = ["image"]
        fetchRecordsImageOperation.queuePriority = .VeryHigh
        fetchRecordsImageOperation.perRecordCompletionBlock = {(record:CKRecord!, recordID:CKRecordID!, error:NSError!) -> Void in
            if (error != nil) {
                println("Failed to get restaurant image: \(error.localizedDescription)")
            } else {
                if let restaurantRecord = record {
                    dispatch_async(dispatch_get_main_queue(), {
                        let imageAsset = restaurantRecord.objectForKey("image") as CKAsset
                        self.imageCache.setObject(imageAsset.fileURL, forKey: restaurant.recordID)
                        cell.imageView.image = UIImage(data: NSData(contentsOfURL: imageAsset.fileURL)!)
                    })
                }
            }
        }
        publicDatabase.addOperation(fetchRecordsImageOperation)
    }
    return cell
}
  • Pull to Refresh(下拉刷新)

TableViewControllerviewDidLoad方法中添加如下代码:

// Pull To Refresh Control
refreshControl = UIRefreshControl()
refreshControl?.backgroundColor = UIColor.whiteColor()
refreshControl?.tintColor = UIColor.grayColor()
refreshControl?.addTarget(self, action: "getRecordsFromCloud", forControlEvents: UIControlEvents.ValueChanged)

使用下面的代码隐藏refresh control

// Hide the refresh control
self.refreshControl?.endRefreshing()
  • Saving Data Using CloudKit(使用CloudKit保存数据到iCloud中)
func saveRecord(_ record: CKRecord!, completionHandler completionHandler: ((CKRecord!, NSError!) -> Void)!)
func saveRecordToCloud(restaurant:Restaurant!) -> Void {

    // Prepare the record to save
    var record = CKRecord(recordType: "Restaurant")
    record.setValue(restaurant.name, forKey: "name")
    record.setValue(restaurant.type, forKey: "type")
    record.setValue(restaurant.location, forKey: "location")

    // Resize the image
    var originalImage = UIImage(data: restaurant.image)
    var scalingFactor = (originalImage!.size.width > 1024) ? 1024 / originalImage!.size.width : 1.0
    var scaledImage = UIImage(data: restaurant.image, scale: scalingFactor)

    // Write the image to local file for temporary use
    let imageFilePath = NSTemporaryDirectory() + restaurant.name
    UIImageJPEGRepresentation(scaledImage, 0.8).writeToFile(imageFilePath, atomically: true)

    // Create image asset for upload
    let imageFileURL = NSURL(fileURLWithPath: imageFilePath)
    var imageAsset = CKAsset(fileURL: imageFileURL)
    record.setValue(imageAsset, forKey: "image")

    // Get the Public iCloud Database
    let cloudContainer = CKContainer.defaultContainer()
    let publicDatabase = CKContainer.defaultContainer().publicCloudDatabase

    // Save the record to iCloud
    public Database.saveRecord(record, completionHandler: { (record:CKRecord!, error:NSError!) -> Void in
        // Remove temp file
        NSFileManager.defaultManager().removeItemAtPath(imageFilePath, error: nil)

        if (error != nil) {
            println("Failed to save record to the cloud: \(error.description)")
        }
    })
}
  • Sorting the Result by Creation Date

CloudKit dashboard中,在meta index,选择Fields选项,为Date Created meta data 选项勾选Sort

getRecordsFromCloud方法中添加如下代码:

// Prepare the query
let predicate = NSPredicate(value: true)
let query = CKQuery(recordType: "Restaurant", predicate: predicate)
query.sortDescriptors = [NSSortDescriptor(key: "creationDate", ascending: false)]

27. Localization(国际化)

通过使用NSLocalizedString宏来实现国际化,Xcode将国际化资源存储在Localizable.strings文件中。

NSLocalizedString("Share using", comment: "For social sharing")
  • Export for Localization

select Editor > Export For Localization(最终导出为XLIFF文件,XLIFF是一种标准格式的xml文件)

28. APPENDIX

28.1. Swift Basics

Objective-C

const int count = 10;
double price = 23.55;
NSString *myMessage = @"Objective-C is not dead yet!";

NSString *firstMessage = @"Swift is awesome. ";
NSString *secondMessage = @"What do you think?";
NSString *message = [NSString stringWithFormat:@"%@%@", firstMessage, secondMessage];
NSLog(@"%@", message);

Swift

let count = 10
var price = 23.55

//var myMessage = "Swift is the future!"
var myMessage : String = "Swift is the future!"

let dontModifyMe = "You cannot modify this string"
var modifyMe = "You can modify this string"

let firstMessage = "Swift is awesome. "
let secondMessage= "What do you think?"
var message = firstMessage + secondMessage
println(message)


var string1 = "Hello"
var string2 = "Hello"
if string1 == string2 {
    println("Both are the same")
}
28.2. Arrays

Objective-C:

NSArray *recipes = @[@"Egg Benedict", @"Mushroom Risotto", @"Full Breakfast", @"Hamburger", @"Ham and Egg Sandwich"];

Swift:

//var recipes = ["Egg Benedict", "Mushroom Risotto", "Full Breakfast", "Hamburger", "Ham and Egg Sandwich"]

var recipes : String[] = ["Egg Benedict", "Mushroom Risotto", "Full Breakfast", "Hamburger","Ham and Egg Sandwich"]

var numberOfItems = recipes.count
recipes += "Thai Shrimp Cake"
recipes += ["Creme Brelee", "White Chocolate Donut", "Ham and Cheese Panini"]

var recipeItem = recipes[0]
recipes[1] = "Cupcake"

recipes[1...3] = ["Cheese Cake", "Greek Salad", "Braised Beef Cheeks"]
28.3. Dictionaries

Objective-C:

NSDictionary *companies = @{@"AAPL" : @"Apple Inc", @"GOOG" : @"Google Inc", @"AMZN" : @"Amazon.com, Inc", @"FB" : @"Facebook Inc"};

Swift:

var companies = ["AAPL" : "Apple Inc", "GOOG" : "Google Inc", "AMZN" : "Amazon.com, Inc", "FB" : "Facebook Inc"]

//var companies: Dictionary<String, String> = ["AAPL" : "Apple Inc", "GOOG" : "Google Inc", "AMZN" : "Amazon.com, Inc", "FB" : "Facebook Inc"]

for (stockCode, name) in companies {
    println("\(stockCode) = \(name)")
}

for stockCode in companies.keys {
    println("Stock code = \(stockCode)")
}

for name in companies.values {
    println("Company name = \(name)")
}

companies["TWTR"] = "Twitter Inc"
28.4. Classes
class Recipe {
    var name: String = ""
    var duration: Int = 10
    var ingredients: String[] = ["egg"]
}
class Recipe {
    var name: String?
    var duration: Int = 10
    var ingredients: String[]?
}
var recipeItem = Recipe()
recipeItem.name = "Mushroom Risotto"
recipeItem.duration = 30
recipeItem.ingredients = ["1 tbsp dried porcini mushrooms", "2 tbsp olive oil", "1 onion, chopped", "2 garlic cloves", "350g/12oz arborio rice", "1.2 litres/2 pints hot vegetable stock", "salt and pepper", "25g/1oz butter"]

Objective-C:

@interface SimpleTableViewController : UIViewController <UITableViewDelegate, UITableViewDataSource>

Swift:

class SimpleTableViewController : UIViewController, UITableViewDelegate, UITableViewDataSource
28.5. Methods
class TodoManager {

    func printWelcomeMessage() {
        println("Welcome to My ToDo List")
    }

    func printWelcomeMessage(name:String) -> Int {
        println("Welcome to \(name)'s ToDo List")
        return 10
    }

}

方法调用:

Objective-C:

TodoManager todoManager = [[TodoManager alloc] init]
[todoManager printWelcomeMessage];

Swift:

var todoManager = TodoManager()
todoManager.printWelcomeMessage()
let numberOfTodoItem = todoManager.printWelcomeMessage("Simon")
println(numberOfTodoItem)
28.6. Control Flow
  • for loops
for i in 0..<5 {
    println("index = \(i)")
}

输出结果:

index = 0
index = 1
index = 2
index = 3
index = 4
for i in 0...<5 {
    println("index = \(i)")
}

输出结果:

index = 0
index = 1
index = 2
index = 3
index = 4
index = 5

注:..不包含后者,...包含后者。

for var i = 0; i < 5; i++ {
    println("index = \(i)")
}
  • if-else statement
var bookPrice = 1000;
if bookPrice >= 999 {
    println("Hey, the book is expensive")
} else {
    println("Okay, I can affort it")
}
  • switch statement
switch recipeName {
    case "Egg Benedict":
        println("Let's cook!")
    case "Mushroom Risotto":
        println("Hmm... let me think about it")
    case "Hamburger":
        println("Love it!")
    default:
        println("Anything else")
}
var speed = 50
switch speed {
    case 0:
        println("stop")
    case 0...40:
        println("slow")
    case 41...70:
        println("normal")
    case 71..<101:
        println("fast")
    default:
        println("not classified yet")
}
28.7. Tuples
let company = ("AAPL", "Apple Inc", 93.5)

let (stockCode, companyName, stockPrice) = company
println("stock code = \(stockCode)")
println("company name = \(companyName)")
println("stock price = \(stockPrice)")

let product = (id: "AP234", name: "iPhone 6", price: 599)
println("id = \(product.id)")
println("name = \(product.name)")
println("price = USD\(product.price)")
class Store {
    func getProduct(number: Int) -> (id: String, name: String, price: Int) {
        var id = "IP435", name = "iMac", price = 1399

        switch number {
            case 1:
                id = "AP234"
                name = "iPhone 6"
                price = 599
            case 2:
                id = "PE645"
                name = "iPad Air"
                price = 499
            default:
                break
        }
        return (id, name, price)
    }
}
let store = Store()
let product = store.getProduct(2)
println("id = \(product.id)")
println("name = \(product.name)")
println("price = USD\(product.price)")
28.8. Optionals
var message: String = "Swift is awesome!" // OK
message = nil // compile-time error

class Messenger {
    var message1: String = "Swift is awesome!" // OK
    var message2: String // compile-time error
}

class Messenger {
    var message1: String = "Swift is awesome!" // OK
    var message2: String? // OK
}
func findStockCode(company: String) -> String? {
    if (company == "Apple") {
        return "AAPL"
    } else if (company == "Google") {
        return "GOOG"
    }
    return nil
}

var stockCode:String? = findStockCode("Facebook")
let text = "Stock Code - "
let message = text + stockCode // compile-time error
println(message)
28.9. Unwrapping Optionals
var stockCode:String? = findStockCode("Facebook")
let text = "Stock Code - "
if stockCode != nil {
    let message = text + stockCode!
    println(message)
}


var stockCode:String? = findStockCode("Facebook")
let text = "Stock Code - "
let message = text + stockCode! // runtime error
28.10. Optional Binding
var stockCode:String? = findStockCode("Facebook")

let text = "Stock Code - "
if let tempStockCode = stockCode {
    let message = text + tempStockCode
    println(message)
}

let text = "Stock Code - "
if var stockCode = findStockCode("Apple") {
    let message = text + stockCode
    println(message)
}
28.11. Optional Chaining
class Stock {
    var code: String?
    var price: Double?
}

func findStockCode(company: String) -> Stock? {
    if (company == "Apple") {
        let aapl: Stock = Stock()
        aapl.code = "AAPL"
        aapl.price = 90.32
        return aapl
    } else if (company == "Google") {
        let goog: Stock = Stock()
        goog.code = "GOOG"
        goog.price = 556.36
        return goog
    }
    return nil
}
if let stock = findStockCode("Apple") {
    if let sharePrice = stock.price {
        let totalCost = sharePrice * 100
        println(totalCost)
    }
}
if let sharePrice = findStockCode("Apple")?.price {
    let totalCost = sharePrice * 100
    println(totalCost)
}

elasticsearch articles

状态栏显示网络加载进度

[UIApplication sharedApplication].networkActivityIndicatorVisible = YES;

或者使用宏,定义如下:

#define ShowNetworkActivityIndicator() [UIApplication sharedApplication].networkActivityIndicatorVisible = YES
#define HideNetworkActivityIndicator() [UIApplication sharedApplication].networkActivityIndicatorVisible = NO

使用下面的代码来调用

ShowNetworkActivityIndicator(); //加载进度可见
HideNetworkActivityIndicator();//加载进度不可见

基于hadoop 2.5.2源码编译hadoop-eclipse-plugin

环境:Ubuntu 14.04.1 + jdk1.7 + hadoop 2.5.2 + eclipse-jee-luna-SR1-linux-gtk-x86_64

  • 首先clone一份源码:
git clone https://github.com/winghc/hadoop2x-eclipse-plugin.git
  • 环境设置

修改libraries.properties文件中的jackson.version值为1.9.13,共两处:

hadoop2x-eclipse-plugin/ivy
hadoop2x-eclipse-plugin/src/ivy

  • 编译

进入hadoop2x-eclipse-plugin/src/contrib/eclipse-plugin目录,执行ant jar -Dversion=2.5.2 -Dhadoop.version=2.5.2 -Declipse.home=/vagrant/eclipse -Dhadoop.home=/opt/hadoop

hadoop2x-eclipse-plugin/src/contrib/eclipse-plugin$  ant jar -Dversion=2.5.2 -Dhadoop.version=2.5.2 -Declipse.home=/vagrant/eclipse -Dhadoop.home=/opt/hadoop
  • eclipse中的配置

Map/Reduce V2 Master中的配置:host为namenode节点的主机名称,port为yarn-site.xml中yarn.resourcemanager.address的端口,默认为8032

DFS Master中的配置:host为在core-site.xml中fs.defaultFS配置的主机名,port为在core-site.xml中fs.defaultFS配置的端口

Hadoop权威指南气象数据

Hadoop权威指南气象数据

  • 新版气象数据

下载地址:

ftp://ftp3.ncdc.noaa.gov/pub/data/noaa/isd-lite/

ftp://ftp.ncdc.noaa.gov/pub/data/noaa/isd-lite/

数据格式:

2014 01 01 00    33   -23 -9999     0 -9999 -9999 -9999     0
2014 01 01 06    41    -9 -9999     0 -9999 -9999 -9999 -9999
2014 01 01 12    32   -21 -9999     0 -9999 -9999 -9999     0
2014 01 01 18    19   -39 -9999     0 -9999 -9999 -9999 -9999
2014 01 02 12    13   -41 -9999     0 -9999 -9999 -9999 -9999
2014 01 02 18    25   -40 -9999     0 -9999 -9999 -9999 -9999
2014 01 03 00    29   -16 -9999     0 -9999 -9999 -9999     4
2014 01 03 06    30   -35 -9999     0 -9999 -9999 -9999 -9999
2014 01 03 12    41   -27 -9999     0 -9999 -9999 -9999     0
2014 01 03 18    22    -8 -9999     0 -9999 -9999 -9999 -9999
2014 01 04 00    20   -38 -9999     0 -9999 -9999 -9999     2
2014 01 04 06    26   -41 -9999     0 -9999 -9999 -9999 -9999
2014 01 04 12    12   -11 -9999     0 -9999 -9999 -9999     1
2014 01 04 18    16     0 -9999     0 -9999 -9999 -9999 -9999
2014 01 05 00    11   -18 -9999     0 -9999 -9999 -9999     0
2014 01 05 06    36   -18 -9999     0 -9999 -9999 -9999 -9999
2014 01 05 12    42   -29 -9999     0 -9999 -9999 -9999     4
2014 01 05 18    35   -23 -9999     0 -9999 -9999 -9999 -9999
2014 01 06 00    24   -33 -9999     0 -9999 -9999 -9999     0
2014 01 06 06    17   -52 -9999     0 -9999 -9999 -9999 -9999
  • 旧版气象数据

下载地址:

ftp://ftp.ncdc.noaa.gov/pub/data/noaa/

数据格式:

0083014220999992014010100004+59167+006033FM-12+002599999V0200601N003019999999N999999999+00791+00111999999ADDAA106000091OD139901201999REMSYN04601422 16/// /0603 10079 20011 60001 333 91112=
0039014220999992014010101004+59167+006033FM-12+002599999V0200701N003019999999N999999999+00801+00121999999REMSYN03001422 46/// /0703 10080 20012=
0039014220999992014010102004+59167+006033FM-12+002599999V0200401N003019999999N999999999+00781+00191999999REMSYN03001422 46/// /0403 10078 20019=
0039014220999992014010103004+59167+006033FM-12+002599999V0200701N002019999999N999999999+00741+00171999999REMSYN03001422 46/// /0702 10074 20017=
0039014220999992014010104004+59167+006033FM-12+002599999V0200501N002019999999N999999999+00731+00201999999REMSYN03001422 46/// /0502 10073 20020=
0039014220999992014010105004+59167+006033FM-12+002599999V0200701N002019999999N999999999+00701+00191999999REMSYN03001422 46/// /0702 10070 20019=
0119014220999992014010106004+59167+006033FM-12+002599999V0200601N002019999999N999999999+00671+00201999999ADDAA112000091AA224001831KA1240N+00671OD139901001999REMSYN05801422 16/// /0602 10067 20020 60002 333 20067 70018 91110=
0039014220999992014010107004+59167+006033FM-12+002599999V0200601N002019999999N999999999+00621+00211999999REMSYN03001422 46/// /0602 10062 20021=
0039014220999992014010108004+59167+006033FM-12+002599999V0201101N002019999999N999999999+00571+00241999999REMSYN03001422 46/// /1102 10057 20024=
0039014220999992014010109004+59167+006033FM-12+002599999V0201401N001019999999N999999999+00481+00301999999REMSYN03001422 46/// /1401 10048 20030=
0039014220999992014010110004+59167+006033FM-12+002599999V0201701N002019999999N999999999+00451+00291999999REMSYN03001422 46/// /1702 10045 20029=
0039014220999992014010111004+59167+006033FM-12+002599999V0201301N003019999999N999999999+00491+00231999999REMSYN03001422 46/// /1303 10049 20023=
0083014220999992014010112004+59167+006033FM-12+002599999V0201101N004019999999N999999999+00541+00211999999ADDAA106003031OD139900801999REMSYN04601422 16/// /1104 10054 20021 60031 333 91108=

Sqoop - Data transfer tool from RDBMS to Hadoop Box

TO IMPORT & EXPORT DATA FROM RDBMS (MYSQL,ORACLE, etc) INTO HDFS / HIVE / HBASE

Pre-requisite

Apache Hadoop

Apache Sqoop (compatible with Hadoop version)

Apache Hive (optional)

Apache HBase (optional)

Apache HCatalog (optional)

JDBC/ODBC connector

For all RDBMS, Connection URL changes and remaining all command line arguments remains same. You need to download specific JDBC/ODBC connector JAR and copy it to $SQOOP_HOME/lib

MySQL

Download mysql-connector-java.jar and place in $SQOOP_HOME/lib folder

cp mysql-connector-java-5.1.18-bin.jar /usr/local/hadoop/sqoop-1.4.3-cdh4.4.0/lib/

Expecting you have data in MySQL tables.

Retrieving list of Databases available in MySQL from SQOOP

sqoop list-databases --connect jdbc:mysql://localhost:3306/  --username root -P
  • MySQL to HDFS Import

Have Primary key:

sqoop import -connect jdbc:mysql://localhost:3306/db1 -username root -password password --table tableName --target-dir /path/to/directoryName

No Primary key:

sqoop import -connect jdbc:mysql://localhost:3306/db1 -username root -password password --table tableName --target-dir /path/to/directoryName  -m 1
  • MySQL to Hive Import

Have Primary key:

sqoop-import  --connect jdbc:mysql://localhost:3306/db1 -username root -password password --table tableName  --hive-table tableName --create-hive-table --hive-import --hive-home path/to/hive_home

No Primary key:

sqoop-import  --connect jdbc:mysql://localhost:3306/db1 -username root -password password --table tableName  --hive-table tableName --create-hive-table --hive-import --hive-home  path/to/hive_home -m 1
  • MySQL to HBase Import

Have Import All columns:

sqoop import --connect jdbc:mysql://localhost:3306/db1 --username root --password root --table tableName --hbase-table hbase_tableName  --column-family hbase_table_col1 --hbase-create-table

HBase import few columns

sqoop import --connect jdbc:mysql://localhost:3306/db1 --username root --password root --table tableName --hbase-table hbase_tableName --columns column1,column2 --column-family hbase_table_col1 --hbase-create-table

To HBase with Primary key:

sqoop import --connect jdbc:mysql://localhost:3306/db1 --username root --password root --table tableName --hbase-table hbase_tableName --column-family hbase_table_col1 --hbase-row-key column1 –hbase-create-table

To Hbase with no primary key:

sqoop import --connect jdbc:mysql://localhost:3306/db1 --username root --password root --table tableName --hbase-table hbase_tableName --columns column1,column2 --column-family hbase_table_col --hbase-row-key column1 --hbase-create-table
  • Export from HDFS to MySQL:

Same for all Hive/HBase/HDFS: Because Hive tables are nothing but directories in HDFS. So you're just exporting a directory to MySQL

sqoop export --connect jdbc:mysql://localhost:3306/test_db --table tableName  --export-dir /user/hive/warehouse/tableName --username root --password password -m 1 --input-fields-terminated-by '\001'

SQL Server

Connection URL:

sqoop import --connect 'jdbc:sqlserver://<IP(or)hostname>;username=dbuser;password=dbpasswd;database=<DB>' --table <table> --target-dir /path/to/hdfs/dir --split-by <KEY> -m 1

Download Connector from Microsoft website

http://www.microsoft.com/en-us/download/confirmation.aspx?id=11774
Place it in $SQOOP_HOME/lib

Oracle

Connection URL:

sqoop import --connect "jdbc:oracle:thin:@(description=(address=(protocol=tcp)(host=myhost)(port=1521))(connect_data=(service_name=myservice)))" \
--username USER --table SCHEMA.TABLE_NAME --hive-import --hive-table SCHEMA.TABLE_NAME \
--num-mappers 1 --verbose -P \

IBM DB2

Download the DB2Driver and place it in $SQOOP_HOME/lib

sqoop import --driver com.ibm.db2.jcc.DB2Driver --connect jdbc:db2://db2.my.com:50000/testdb --username db2user --db2pwd --table db2tbl --split-by tbl_primarykey --target-dir sqoopimports
sqoop export --driver com.ibm.db2.jcc.DB2Driver --connect jdbc:db2://db2.my.com:50000/myDB --username db2user --password db2pwd --table db2tbl --export-dir /sqoop/dataFile.csv

Different Connection Strings for Different RDBMS

Database version --direct support? connect string matches

HSQLDB 1.8.0+ No jdbc:hsqldb:*//

MySQL 5.0+ Yes jdbc:mysql://

Oracle 10.2.0+ No jdbc:oracle:*//

PostgreSQL 8.3+ Yes (import only) jdbc:postgresql://

https://coderwall.com/p/kgrwwq

linux 统计文件行数 字节数 单词个数

Linux统计文件行数 字节数 单词个数

wc - lcw file1 file2

输出结果

4 33 file1
7 52 file2
11 11 85 total

该命令各选项含义如下:

- c 统计字节数。
- l 统计行数。
- w 统计字数。

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.