Comments (10)
@soniazhai 请问你启动了几个 pd-server?
from docs-cn.
三个
from docs-cn.
please give us the whole pd log.
from docs-cn.
邮件发还是都贴出来?
from docs-cn.
可以 attach 到 github 上
from docs-cn.
# 我又重启了所有的机器,重新试了一次,就是在运行tidb时,pd server leader 开始报错
2016/12/29 18:15:42 conn.go:75: [error] read request message err EOF
2016/12/29 18:15:42 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 0
2016/12/29 18:15:43 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 1
2016/12/29 18:15:43 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 2
2016/12/29 18:15:43 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 3
2016/12/29 18:15:43 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 4
2016/12/29 18:15:44 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 5
2016/12/29 18:15:44 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 6
2016/12/29 18:15:44 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 7
2016/12/29 18:15:44 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 8
2016/12/29 18:15:44 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 9
2016/12/29 18:15:45 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 10
2016/12/29 18:15:45 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 11
2016/12/29 18:15:45 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 12
2016/12/29 18:15:45 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 13
2016/12/29 18:15:45 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 14
2016/12/29 18:15:46 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 15
2016/12/29 18:15:46 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 16
2016/12/29 18:15:46 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 17
2016/12/29 18:15:46 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 18
2016/12/29 18:15:46 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 19
2016/12/29 18:15:47 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 20
2016/12/29 18:15:47 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 21
2016/12/29 18:15:47 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 22
2016/12/29 18:15:47 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 23
2016/12/29 18:15:47 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 24
2016/12/29 18:15:48 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 25
2016/12/29 18:15:48 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 26
2016/12/29 18:15:48 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 27
2016/12/29 18:15:48 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 28
2016/12/29 18:15:48 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 29
2016/12/29 18:15:49 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 30
2016/12/29 18:15:49 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 31
2016/12/29 18:15:49 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 32
2016/12/29 18:15:49 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 33
2016/12/29 18:15:49 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 34
2016/12/29 18:15:50 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 35
2016/12/29 18:15:50 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 36
2016/12/29 18:15:50 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 37
2016/12/29 18:15:50 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 38
2016/12/29 18:15:50 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 39
2016/12/29 18:15:51 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 40
2016/12/29 18:15:51 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 41
2016/12/29 18:15:51 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 42
2016/12/29 18:15:51 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 43
2016/12/29 18:15:51 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 44
2016/12/29 18:15:52 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 45
2016/12/29 18:15:52 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 46
2016/12/29 18:15:52 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 47
2016/12/29 18:15:52 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 48
2016/12/29 18:15:52 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 49
2016/12/29 18:15:53 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 50
2016/12/29 18:15:53 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 51
2016/12/29 18:15:53 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 52
2016/12/29 18:15:53 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 53
2016/12/29 18:15:53 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 54
2016/12/29 18:15:54 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 55
2016/12/29 18:15:54 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 56
2016/12/29 18:15:54 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 57
2016/12/29 18:15:54 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 58
2016/12/29 18:15:54 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 59
2016/12/29 18:15:55 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 60
2016/12/29 18:15:55 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 61
2016/12/29 18:15:55 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 62
2016/12/29 18:15:55 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 63
2016/12/29 18:15:55 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 64
2016/12/29 18:15:56 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 65
2016/12/29 18:15:56 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 66
2016/12/29 18:15:56 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 67
2016/12/29 18:15:56 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 68
2016/12/29 18:15:56 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 69
2016/12/29 18:15:57 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 70
2016/12/29 18:15:57 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 71
2016/12/29 18:15:57 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 72
2016/12/29 18:15:57 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 73
2016/12/29 18:15:57 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 74
2016/12/29 18:15:58 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 75
2016/12/29 18:15:58 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 76
2016/12/29 18:15:58 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 77
2016/12/29 18:15:58 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 78
2016/12/29 18:15:58 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 79
2016/12/29 18:15:59 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 80
2016/12/29 18:15:59 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 81
2016/12/29 18:15:59 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 82
2016/12/29 18:15:59 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 83
2016/12/29 18:15:59 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 84
2016/12/29 18:16:00 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 85
2016/12/29 18:16:00 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 86
2016/12/29 18:16:00 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 87
2016/12/29 18:16:00 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 88
2016/12/29 18:16:00 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 89
2016/12/29 18:16:01 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 90
2016/12/29 18:16:01 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 91
2016/12/29 18:16:01 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 92
2016/12/29 18:16:01 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 93
2016/12/29 18:16:01 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 94
2016/12/29 18:16:02 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 95
2016/12/29 18:16:02 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 96
2016/12/29 18:16:02 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 97
2016/12/29 18:16:02 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 98
2016/12/29 18:16:02 tso.go:155: [error] we haven't synced timestamp ok, wait and retry, retry count 99
2016/12/29 18:16:03 conn.go:102: [error] handle request header:<uuid:"\227\231\222\034R\214B\370\220\223\220\256\274L\221\330" cluster_id:6369440677371556373 > cmd_type:Tso tso:<count:1 > err /home/jenkins/workspace/BUILD_PD_WORKFLOW/go/src/github.com/pingcap/pd/server/tso.go:169: can not get timestamp
/home/jenkins/workspace/BUILD_PD_WORKFLOW/go/src/github.com/pingcap/pd/server/command.go:33:
2016/12/29 18:16:03 conn.go:75: [error] read request message err EOF
同时,tidb-server也没运行成功
2016/12/29 18:15:43 printer.go:31: [info] Welcome to TiDB.
2016/12/29 18:15:43 printer.go:32: [info] Version:
2016/12/29 18:15:43 printer.go:33: [info] Git Commit Hash: f1aaeafade7c78f1ca6ee7edec395bc3306282df
2016/12/29 18:15:43 printer.go:34: [info] UTC Build Time: 2016-12-13 06:50:46
2016/12/29 18:15:43 client.go:52: [info] [pd] create pd client with endpoints [10.5.77.11:22379 10.5.77.12:22379 10.5.77.13:22379]
2016/12/29 18:15:43 rpc_worker.go:82: [info] [pd] init cluster id 6369440677371556373
2016/12/29 18:15:43 rpc_worker.go:113: [info] [pd] connect to pd server [http://10.5.77.11:22379 http://10.5.77.12:22379 http://10.5.77.13:22379]
2016/12/29 18:15:43 rpc_worker.go:118: [info] [pd] connected to 10.5.77.12:22379
2016/12/29 18:15:44 rpc_worker.go:196: [error] [pd] rpc failed: read tcp 10.5.77.11:37702->10.5.77.12:22379: i/o timeout
2016/12/29 18:15:44 rpc_worker.go:113: [info] [pd] connect to pd server [http://10.5.77.11:22379 http://10.5.77.12:22379 http://10.5.77.13:22379]
2016/12/29 18:15:44 main.go:156: [fatal] github.com/pingcap/pd/pd-client/rpc_worker.go:371: [pd] rpc failed: read tcp 10.5.77.11:37702->10.5.77.12:22379: i/o timeout
github.com/pingcap/pd/pd-client/rpc_worker.go:279:
/home/jenkins/workspace/BUILD_TIDB_WORKFLOW/go/src/github.com/pingcap/tidb/store/tikv/oracle/oracles/pd.go:86:
/home/jenkins/workspace/BUILD_TIDB_WORKFLOW/go/src/github.com/pingcap/tidb/store/tikv/oracle/oracles/pd.go:75:
/home/jenkins/workspace/BUILD_TIDB_WORKFLOW/go/src/github.com/pingcap/tidb/store/tikv/oracle/oracles/pd.go:57:
/home/jenkins/workspace/BUILD_TIDB_WORKFLOW/go/src/github.com/pingcap/tidb/store/tikv/kv.go:95:
/home/jenkins/workspace/BUILD_TIDB_WORKFLOW/go/src/github.com/pingcap/tidb/store/tikv/kv.go:73:
/home/jenkins/workspace/BUILD_TIDB_WORKFLOW/go/src/github.com/pingcap/tidb/tidb.go:248:
from docs-cn.
请把三个 PD 的完整日志发到 [email protected]
from docs-cn.
@soniazhai 从 log 来看,我们猜测是两种可能,1. PD 向磁盘写东西写不进去,比如磁盘慢了, 2. PD 之间的通讯有问题,比如某个端口被防火墙block了。请帮忙把三个pd完整的log贴在这里,我们确认一下问题。
多谢!
from docs-cn.
不好意思,放假几天,没及时跟进。问题已经解决了,还是时钟同步有问题。重新调用了ntp服务,已经正常了。
from docs-cn.
好的,欢迎继续试用,有问题随时反馈,多谢!
from docs-cn.
Related Issues (20)
- 建议补充 tikv-cdc 的相关内容
- tiup-component-cluster-template 文档待更新 HOT 1
- tiup-component-management 文档待更新 HOT 1
- 生成所有分区表的更新统计信息的SQL是否有误? HOT 1
- TiDB 版本号格式的问题 HOT 1
- 开头疑似乱码 HOT 2
- 6.5 tikv_gc_can_not_work 指标缺失 HOT 2
- tiup拓扑配置例子中tikv_servers中一个节点上配置双实例有误
- 多处出现错别字。“登陆”应改为“登录”,请全局搜索并修改。 HOT 1
- 文档勘误
- index复数表示错误
- 文档中自相矛盾 HOT 3
- 笔误
- 存储成本太高怎么解释?
- Include `systemd-timesyncd` into the list of NTP services
- UOS V20
- typo: tidb-lightning-physical-import-mode-usage HOT 2
- Blog Bug HOT 1
- Remove unsupport FLOAT4/FLOAT8 HOT 2
- The system table is incomplete.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from docs-cn.