细节性能优化 about mdrill HOT 6 CLOSED

2014-07-24 13:24:42 MakeUnivertedFieldBySigment [INFO] makeInit startprice1 ,this.baseAdvanceDocs=226
2014-07-24 13:24:43 DocValuesReader [INFO] 44>DocValuesReader [tispos=914726124, tisposTxt=3641104, tisposVal=334278448, maxsize=13938676, maxtm=10882, useBits=14, lst=86, sizeOfStrings=1892]
2014-07-24 13:24:45 RamDocValueFill [INFO] file timetaken:0,2209,189
2014-07-24 13:24:45 SolrCore [INFO] field value cache lru removed key org.apache.solr.search.SolrIndexSearcher$QueryResultKeyCache@b6f20ca
2014-07-24 13:24:45 SolrCore [INFO] field value cache lru removed key org.apache.solr.search.SolrIndexSearcher$QueryKey@f5ef277a
2014-07-24 13:24:45 SolrCore [INFO] field value cache lru removed key org.apache.solr.search.SolrIndexSearcher$QueryKey@2db0d42
2014-07-24 13:24:45 SolrCore [INFO] field value cache lru removed key org.apache.solr.search.SolrIndexSearcher$QueryKey@dffc39f1
2014-07-24 13:24:45 SolrCore [INFO] field value cache lru removed key seg@true@thedate@12584976@[email protected]/disk1/taobao/mdrill/higo/rpt_hitfake_auctionall_d/shard_0_42/201407_32_455019181_3846575312_1406157327000_20140724071527@_6
2014-07-24 13:24:45 SolrCore [INFO] field value cache lru removed key seg@true@e_alipay_direct_amt@12584976@[email protected]/disk1/taobao/mdrill/higo/rpt_hitfake_auctionall_d/shard_0_42/201407_32_455019181_3846575312_1406157327000_20140724071527@6
2014-07-24 13:24:45 SolrCore [INFO] ####fieldvaluecache####1240@1290mb,size=4105,mem=28mb,key seg@true@price1@13938677@[email protected]/disk7/taobao/mdrill/higo/rpt_p4padhoc_product/shard_0_42/201403_32_3174774810_2928937847_1404019067000_20140629131747@6
2014-07-24 13:24:45 UnInvertedField [INFO] ####timetaken####:2790,{field=price1,memSize=28.78921890258789mb,nullTermNum=10883,indexshort=13959168@13938679,,termValueDouble=65536@10884,bits.size=226}
2014-07-24 13:24:45 ShardGroupByGroupbyRowCompare [INFO] ####SortType [sortFieldNum=-1, typeNum=8, typeEnum=column]
2014-07-24 13:24:45 ShardGroupByGroupbyRowCompare [INFO] ##cmp string##
2014-07-24 13:24:45 ShardGroupByTermNumCompare [INFO] ####SortType [sortFieldNum=-1, typeNum=8, typeEnum=column]
2014-07-24 13:24:45 MdrillParseGroupby [INFO] ##baseDocs.size## 226@226
2014-07-24 13:24:45 MergerGroupByGroupbyRowCompare [INFO] ####SortType [sortFieldNum=-1, typeNum=8, typeEnum=column]
2014-07-24 13:24:45 MergerGroupByGroupbyRowCompare [INFO] ##cmp string##
2014-07-24 13:24:45 MdrillUtils [INFO] fieldValueCache.size=102018,size=1
2014-07-24 13:24:45 MdrillGroupBy [INFO] ##FacetCross## time taken ,total:2828,init:2826,makeGroups:1,transGroupValue:1,groups.size:31
2014-07-24 13:24:45 FacetComponent [INFO] ####task####2828,0
2014-07-24 13:24:45 SolrCore [INFO] [rpt_p4padhoc_product] status=0 QTime=2828 webapp=/solr path=/select params={{echoParams=explicit,rows=0,facet.cross.join=@,facet.cross=true,mdrill.crc.key.set=4c0e8a93-fc5d-4281-9ccf-6eef16e117d2,facet=true,facet.cross.offset=0,facet.cross.sort.fl=thedate,indexpartion=201403,version=2,facet.cross.sort.desc=true,showlog=1,facet.cross.limit=10010,facet.field=thedate,fq=[Ljava.lang.String;@c018d9b,facet.cross.sort.cp=string,higo_ms_depth=2,wt=javabin,rows=0,mlogtime=20140724132442,start=0,facet.sort=index,q=:,facet.cross.sort.tp=column,maxshards=64,facet.cross.fl=price1,mdrill.isRestat=false,mergeservers=10.246.14.55:51270/solr/rpt_p4padhoc_product,172.24.195.155:51271/solr/rpt_p4padhoc_product,10.246.45.21:51272/solr/rpt_p4padhoc_product,10.246.45.22:51273/solr/rpt_p4padhoc_product,10.246.45.23:51274/solr/rpt_p4padhoc_product,10.246.45.24:51275/solr/rpt_p4padhoc_product,10.246.45.41:51276/solr/rpt_p4padhoc_product,10.246.45.42:51277/solr/rpt_p4padhoc_product,10.246.45.43:51278/solr/rpt_p4padhoc_product,10.246.45.44:51279/solr/rpt_p4padhoc_product,172.24.195.154:51280/solr/rpt_p4padhoc_product,,isShard=true}}
^C

from mdrill.

2014-07-24 13:44:49 RamDocValueFill [INFO] file timetaken:0,4452,198

from mdrill.

double类型的数据更慢

from mdrill.

2014-07-24 13:44:49 RamDocValueFill [INFO] file timetaken:0,4452,198
首次这里慢的原因为 首次这里的size的大小为doc文档的个数，应该修改为
for (int i = 0; i <=docValues.maxtm; i++)

from mdrill.

另一个慢的原因在这里
Arrays.fill(data, v); 这个data比较大，耗费时间

from mdrill.

最终结果，效果提升明显

2014-07-24 18:29:18 GrobalCache [INFO] ####fieldvaluecache####675@1105mb,size=1505,mem=1mb,key org.apache.solr.search.SolrIndexSearcher$QueryKey@e6fde45e
2014-07-24 18:29:18 UnInvertedField [INFO] ajustBase 16,baseAdvanceDocs=320@3051832@12207331,org.apache.lucene.index.ReadOnlySegmentReader
2014-07-24 18:29:18 MakeUnivertedFieldBySigment [INFO] makeInit start:price0 ,this.baseAdvanceDocs=320
2014-07-24 18:29:18 DocValuesReader [INFO] 39>DocValuesReader [tispos=773640043, tisposTxt=3210104, tisposVal=294199504, maxsize=12207330, maxtm=42062, useBits=16, lst=329, sizeOfStrings=7238]
2014-07-24 18:29:19 RamDocValueFill [INFO] file timetaken:0,25,288
2014-07-24 18:29:19 MakeUnivertedFieldBySigment [INFO] ####makeInit end####:price0,1@314@1@7@3
2014-07-24 18:29:19 GrobalCache [INFO] ####fieldvaluecache####676@1105mb,size=1507,mem=48mb,key seg@true@price0@12207331@[email protected]/disk1/taobao/mdrill/higo/rpt_p4padhoc_product/shard_0_0/201406_32_4283647007_2574064039_1404693655000_20140707084055@6
2014-07-24 18:29:19 UnInvertedField [INFO] ####timetaken####:327@1,{field=price0,memSize=48.38665580749512mb,nullTermNum=42063,index=12210176@12207333,,termValueDouble=45056@42064,bits.size=320}
2014-07-24 18:29:19 ShardGroupByGroupbyRowCompare [INFO] ####SortType [sortFieldNum=-1, typeNum=0, typeEnum=index]
2014-07-24 18:29:19 ShardGroupByTermNumCompare [INFO] ####SortType [sortFieldNum=-1, typeNum=0, typeEnum=index]
2014-07-24 18:29:19 MdrillParseGroupby [INFO] ##baseDocs.size## 320@320
2014-07-24 18:29:19 MergerGroupByGroupbyRowCompare [INFO] ####SortType [sortFieldNum=-1, typeNum=0, typeEnum=index]
2014-07-24 18:29:19 MdrillUtils [INFO] fieldValueCache.size=1,size=1
2014-07-24 18:29:19 MdrillGroupBy [INFO] ##FacetCross## time taken ,total:337,init:336,makeGroups:0,transGroupValue:1,groups.size:1
2014-07-24 18:29:19 FacetComponent [INFO] ####task####337,0
2014-07-24 18:29:19 SolrCore [INFO] [rpt_p4padhoc_product] status=0 QTime=1027 webapp=/solr path=/select params={{echoParams=explicit,rows=0,facet.cross.join=@,facet.cross=true,mdrill.crc.key.set=880b2e6d-e98b-41d9-b853-8a247fc4a135,facet=true,facet.cross.offset=0,facet.cross.sort.fl=higoempty_sort_s,indexpartion=201406,version=2,facet.cross.sort.desc=true,showlog=1,facet.cross.limit=10010,facet.field=higoempty_groupby_l,fq=[Ljava.lang.String;@627f29ad,facet.cross.sort.cp=string,higo_ms_depth=2,wt=javabin,rows=0,mlogtime=20140724182918,start=0,facet.sort=index,q=:_,facet.cross.sort.tp=index,maxshards=64,facet.cross.fl=price0,mdrill.isRestat=false,mergeservers=10.246.14.55:51270/solr/rpt_p4padhoc_product,172.24.195.155:51271/solr/rpt_p4padhoc_product,10.246.45.21:51272/solr/rpt_p4padhoc_product,10.246.45.22:51273/solr/rpt_p4padhoc_product,10.246.45.23:51274/solr/rpt_p4padhoc_product,10.246.45.24:51275/solr/rpt_p4padhoc_product,10.246.45.41:51276/solr/rpt_p4padhoc_product,10.246.45.42:51277/solr/rpt_p4padhoc_product,10.246.45.43:51278/solr/rpt_p4padhoc_product,10.246.45.44:51279/solr/rpt_p4padhoc_product,172.24.195.154:51280/solr/rpt_p4padhoc_product,,isShard=true}}
2014-07-24 18:29:20 GrobalCache [INFO] ####fieldvaluecache####725@1105mb,size=1508,mem=1mb,key org.apache.solr.search.SolrIndexSearcher$QueryKey@da110e56

from mdrill.