hbase SessionExpiredException: KeeperErrorCode = Session expired

hbase master 和 regionserver session过期,具体情况如下:
2017-10-13 05:21:56,854 INFO  [main-EventThread] regionserver.HRegionServer: STOPPED: regionserver:60020-0x15f141db117000b, quorum=hahd1:2181,hahm2:2181,hahm1:2181, baseZNode=/hbase regionserver:60020-0x15f141db117000b received expired from ZooKeeper, aborting
2017-10-13 05:21:56,854 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2017-10-13 05:21:56,854 INFO [regionserver/hahd1/10.211.18.39:60020] regionserver.SplitLogWorker: Sending interrupt to stop the worker thread
2017-10-13 05:21:56,855 INFO [regionserver/hahd1/10.211.18.39:60020] regionserver.HRegionServer: Stopping infoServer
2017-10-13 05:21:56,856 INFO [SplitLogWorker-hahd1:60020] regionserver.SplitLogWorker: SplitLogWorker interrupted. Exiting.
2017-10-13 05:21:56,856 INFO [SplitLogWorker-hahd1:60020] regionserver.SplitLogWorker: SplitLogWorker hahd1,60020,1507871462810 exiting
2017-10-13 05:21:56,860 INFO [regionserver/hahd1/10.211.18.39:60020] mortbay.log: Stopped SelectChannelConnector@0.0.0.0:60030
2017-10-13 05:21:56,961 INFO [regionserver/hahd1/10.211.18.39:60020] regionserver.HeapMemoryManager: Stoping HeapMemoryTuner chore.
2017-10-13 05:21:56,962 INFO [hahd1,60020,1507871462810-HeapMemoryTunerChore] regionserver.HeapMemoryManager$HeapMemoryTunerChore: hahd1,60020,1507871462810-HeapMemoryTunerChore exiting
2017-10-13 05:21:56,962 INFO [regionserver/hahd1/10.211.18.39:60020.compactionChecker] regionserver.HRegionServer$CompactionChecker: regionserver/hahd1/10.211.18.39:60020.compactionChecker exiting
2017-10-13 05:21:56,962 INFO [regionserver/hahd1/10.211.18.39:60020.nonceCleaner] regionserver.ServerNonceManager$1: regionserver/hahd1/10.211.18.39:60020.nonceCleaner exiting
2017-10-13 05:21:56,962 INFO [regionserver/hahd1/10.211.18.39:60020] snapshot.RegionServerSnapshotManager: Stopping RegionServerSnapshotManager abruptly.
2017-10-13 05:21:56,962 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher: MemStoreFlusher.0 exiting
2017-10-13 05:21:56,963 INFO [regionserver/hahd1/10.211.18.39:60020] flush.RegionServerFlushTableProcedureManager: Stopping region server flush procedure manager abruptly.
2017-10-13 05:21:56,963 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher: MemStoreFlusher.1 exiting
2017-10-13 05:21:56,973 INFO [StoreCloserThread-ha_cut_mobile,,1500866422681.e6c3c4f3bb252e0382f4997040b0b971.-1] regionserver.HStore: Closed cf
2017-10-13 05:21:56,973 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed ha_cut_mobile,,1500866422681.e6c3c4f3bb252e0382f4997040b0b971.
2017-10-13 05:21:56,974 INFO [regionserver/hahd1/10.211.18.39:60020] regionserver.HRegionServer: aborting server hahd1,60020,1507871462810
2017-10-13 05:21:56,974 INFO [regionserver/hahd1/10.211.18.39:60020] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x15f141db117000c
2017-10-13 05:21:56,975 INFO [StoreCloserThread-trackTreeTable2479w,4_955,1499429207444.b62da80466c7541c1e2344bbca7d5b34.-1] regionserver.HStore: Closed code
2017-10-13 05:21:56,975 INFO [StoreCloserThread-trackTreeTable2481w,4_984,1500590391057.8e7774d367e551c22cf33251726c061b.-1] regionserver.HStore: Closed code
2017-10-13 05:21:56,975 INFO [StoreCloserThread-ha_cut_rum,7,1496371215377.dc1c30c525c2837336cca02091ffc33b.-1] regionserver.HStore: Closed cf
2017-10-13 05:21:56,976 INFO [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Closed trackTreeTable2479w,4_955,1499429207444.b62da80466c7541c1e2344bbca7d5b34.
2017-10-13 05:21:56,976 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed trackTreeTable2481w,4_984,1500590391057.8e7774d367e551c22cf33251726c061b.
2017-10-13 05:21:56,977 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed ha_cut_rum,7,1496371215377.dc1c30c525c2837336cca02091ffc33b.
2017-10-13 05:21:56,983 INFO [StoreCloserThread-trackTreeTable2482w,7_49782,1501230821564.473e5769b1a04447a9cff56de5b171c2.-1] regionserver.HStore: Closed code
2017-10-13 05:21:56,983 INFO [StoreCloserThread-trackTreeTable2484w,7_550607,1502762506624.6ea8cc54078ce04e27e86fa8e2f886b4.-1] regionserver.HStore: Closed code
2017-10-13 05:21:56,984 INFO [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Closed trackTreeTable2482w,7_49782,1501230821564.473e5769b1a04447a9cff56de5b171c2.
2017-10-13 05:21:56,984 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed trackTreeTable2484w,7_550607,1502762506624.6ea8cc54078ce04e27e86fa8e2f886b4.
2017-10-13 05:21:57,001 INFO [StoreCloserThread-trackTreeTable2486w,5_105,1503999665523.de62c1006cfd9d5ae5cf29a16ea5a71c.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,004 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed trackTreeTable2486w,5_105,1503999665523.de62c1006cfd9d5ae5cf29a16ea5a71c.
2017-10-13 05:21:57,009 INFO [StoreCloserThread-trackTreeTable2488w,4_937,1504911529272.cdfb410fad55b2b075523e388fe40574.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,009 INFO [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Closed trackTreeTable2488w,4_937,1504911529272.cdfb410fad55b2b075523e388fe40574.
2017-10-13 05:21:57,010 INFO [StoreCloserThread-trackTreeTable2487w,2_46085,1504257297010.e8f4e33a4f424f2b4a6c5f3e68d5555f.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,012 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed trackTreeTable2487w,2_46085,1504257297010.e8f4e33a4f424f2b4a6c5f3e68d5555f.
2017-10-13 05:21:57,025 INFO [StoreCloserThread-mobileTable2480w,,1499904931169.3f6ee1d4d8cb217543d9144c9a09c4de.-1] regionserver.HStore: Closed mobileMassage
2017-10-13 05:21:57,026 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed mobileTable2480w,,1499904931169.3f6ee1d4d8cb217543d9144c9a09c4de.
2017-10-13 05:21:57,029 INFO [StoreCloserThread-trackTreeTable2493w,,1507871482977.229e564fa548aec999353d39852180d8.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,029 ERROR [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Memstore size is 18517104
2017-10-13 05:21:57,029 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed trackTreeTable2493w,,1507871482977.229e564fa548aec999353d39852180d8.
2017-10-13 05:21:57,030 INFO [StoreCloserThread-trackTreeTable2483w,2_463335,1502235267495.3f9ee54673cd761c65e2d4d8d393ac5c.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,030 INFO [StoreCloserThread-trackTreeTable2479w,,1499830310451.4c14fab3d6fc15c4000227b609de7b2d.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,030 INFO [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Closed trackTreeTable2483w,2_463335,1502235267495.3f9ee54673cd761c65e2d4d8d393ac5c.
2017-10-13 05:21:57,030 INFO [StoreCloserThread-trackTreeTable2484w,2_553332,1502892195452.e163793d591e7f347af9b4dfdfa28a1f.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,030 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed trackTreeTable2479w,,1499830310451.4c14fab3d6fc15c4000227b609de7b2d.
2017-10-13 05:21:57,031 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed trackTreeTable2484w,2_553332,1502892195452.e163793d591e7f347af9b4dfdfa28a1f.
2017-10-13 05:21:57,034 INFO [StoreCloserThread-trackTreeTable2480w,7_19883,1500084161334.2fc71624c6e84b2376f56f3bb2db737c.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,034 INFO [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Closed trackTreeTable2480w,7_19883,1500084161334.2fc71624c6e84b2376f56f3bb2db737c.
2017-10-13 05:21:57,035 INFO [StoreCloserThread-trackTreeTable2480w,5_105,1500084161334.e2b840ef2d18e04363caa51fc6180d44.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,035 INFO [StoreCloserThread-mobileTable2483w,,1501718550915.8f56141c042afcfefd16bbc5a703f8eb.-1] regionserver.HStore: Closed mobileMassage
2017-10-13 05:21:57,035 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed trackTreeTable2480w,5_105,1500084161334.e2b840ef2d18e04363caa51fc6180d44.
2017-10-13 05:21:57,035 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed mobileTable2483w,,1501718550915.8f56141c042afcfefd16bbc5a703f8eb.
2017-10-13 05:21:57,037 INFO [StoreCloserThread-ha_cut_mobile,1,1500866422681.b659f3906865ef150b8943dd36d57cc4.-1] regionserver.HStore: Closed cf
2017-10-13 05:21:57,037 INFO [StoreCloserThread-trackTreeTable2486w,7_532602,1503999665523.32b8007ad39e6d9054c8fbb2ce50abf9.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,038 INFO [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Closed trackTreeTable2486w,7_532602,1503999665523.32b8007ad39e6d9054c8fbb2ce50abf9.
2017-10-13 05:21:57,037 ERROR [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Memstore size is 448
2017-10-13 05:21:57,038 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed ha_cut_mobile,1,1500866422681.b659f3906865ef150b8943dd36d57cc4.
2017-10-13 05:21:57,039 INFO [StoreCloserThread-ha_cut_mobile,5,1500866422681.ac8b0b57d9b03dd3b0f4d6b81397f841.-1] regionserver.HStore: Closed cf
2017-10-13 05:21:57,040 INFO [StoreCloserThread-trackTreeTable2483w,,1502235267495.ea67dda4d784efd6cb51b54d38e80e1e.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,040 INFO [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Closed ha_cut_mobile,5,1500866422681.ac8b0b57d9b03dd3b0f4d6b81397f841.
2017-10-13 05:21:57,040 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed trackTreeTable2483w,,1502235267495.ea67dda4d784efd6cb51b54d38e80e1e.
2017-10-13 05:21:57,041 INFO [StoreCloserThread-ha_cut_app,8,1496371225239.a46a411fa5ab591559c5b17171ef5740.-1] regionserver.HStore: Closed cf
2017-10-13 05:21:57,042 INFO [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Closed ha_cut_app,8,1496371225239.a46a411fa5ab591559c5b17171ef5740.
2017-10-13 05:21:57,042 INFO [StoreCloserThread-ha_cut_rum,3,1496371215377.4cf30a787097237505f2a083d5992b3c.-1] regionserver.HStore: Closed cf
2017-10-13 05:21:57,042 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed ha_cut_rum,3,1496371215377.4cf30a787097237505f2a083d5992b3c.
2017-10-13 05:21:57,045 INFO [StoreCloserThread-mobileTable2139w,,1505288530519.f649daee69651687c580af7dd141dafc.-1] regionserver.HStore: Closed mobileMassage
2017-10-13 05:21:57,046 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed mobileTable2139w,,1505288530519.f649daee69651687c580af7dd141dafc.
2017-10-13 05:21:57,045 INFO [StoreCloserThread-mobileTable2489w,,1505347352451.b3fae5da6ad415e3f186cbc633594b1b.-1] regionserver.HStore: Closed mobileMassage
2017-10-13 05:21:57,047 INFO [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Closed mobileTable2489w,,1505347352451.b3fae5da6ad415e3f186cbc633594b1b.
2017-10-13 05:21:57,047 INFO [StoreCloserThread-trackTreeTable2482w,4_978,1501230821564.3fba02142f6cd444c5443704d93fb858.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,047 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed trackTreeTable2482w,4_978,1501230821564.3fba02142f6cd444c5443704d93fb858.
2017-10-13 05:21:57,048 INFO [StoreCloserThread-trackTreeTable2489w,7_4323,1505465918965.2ab45f4ae818b74a239e30dfe27b7d2e.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,048 INFO [StoreCloserThread-ha_cut_app,1,1496371225239.2f425e06e10aa9130bcc5094dab33657.-1] regionserver.HStore: Closed cf
2017-10-13 05:21:57,048 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed trackTreeTable2489w,7_4323,1505465918965.2ab45f4ae818b74a239e30dfe27b7d2e.
2017-10-13 05:21:57,048 ERROR [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Memstore size is 1600984
2017-10-13 05:21:57,049 INFO [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Closed ha_cut_app,1,1496371225239.2f425e06e10aa9130bcc5094dab33657.
2017-10-13 05:21:57,050 INFO [StoreCloserThread-ha_cut_mobile,3,1500866422681.5360f21848c8c7a5c4467db3ac8adf61.-1] regionserver.HStore: Closed cf
2017-10-13 05:21:57,050 INFO [StoreCloserThread-trackTreeTable2480w,2_201744,1500009309073.da3388e7dbfc7f4d911479a067f751c4.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,050 INFO [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Closed ha_cut_mobile,3,1500866422681.5360f21848c8c7a5c4467db3ac8adf61.
2017-10-13 05:21:57,050 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed trackTreeTable2480w,2_201744,1500009309073.da3388e7dbfc7f4d911479a067f751c4.
2017-10-13 05:21:57,051 INFO [StoreCloserThread-ha_cut_app,3,1496371225239.e6fe7ebe80974797a10018455cb18206.-1] regionserver.HStore: Closed cf
2017-10-13 05:21:57,051 INFO [StoreCloserThread-trackTreeTable2491w,7_49883,1506743354950.26a9e258ea6fb3d33fd3a5fe8709c7f2.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,052 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed trackTreeTable2491w,7_49883,1506743354950.26a9e258ea6fb3d33fd3a5fe8709c7f2.
2017-10-13 05:21:57,052 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed ha_cut_app,3,1496371225239.e6fe7ebe80974797a10018455cb18206.
2017-10-13 05:21:57,057 INFO [StoreCloserThread-hbase:namespace,,1496371213362.5ad2cc2b932c4ff70da907f858736148.-1] regionserver.HStore: Closed info
2017-10-13 05:21:57,058 INFO [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Closed hbase:namespace,,1496371213362.5ad2cc2b932c4ff70da907f858736148.
2017-10-13 05:21:57,058 INFO [StoreCloserThread-mobileTable2486w,,1503458777516.0be6738e8416f4a2f3a77666e1d09985.-1] regionserver.HStore: Closed mobileMassage
2017-10-13 05:21:57,058 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed mobileTable2486w,,1503458777516.0be6738e8416f4a2f3a77666e1d09985.
2017-10-13 05:21:57,062 INFO [StoreCloserThread-trackTreeTable2485w,7_44849,1502983793818.664fcb7d50b102bf68ae0a656b003cc5.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,062 INFO [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Closed trackTreeTable2485w,7_44849,1502983793818.664fcb7d50b102bf68ae0a656b003cc5.
2017-10-13 05:21:57,063 INFO [StoreCloserThread-trackTreeTable2487w,4_9617,1504618231759.051b769ab9022b7ffea09f5fb8264c1e.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,064 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed trackTreeTable2487w,4_9617,1504618231759.051b769ab9022b7ffea09f5fb8264c1e.
2017-10-13 05:21:57,064 INFO [StoreCloserThread-trackTreeTable2491w,4_9842,1506743354950.12044c6e5ffb579682543ff158643478.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,064 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed trackTreeTable2491w,4_9842,1506743354950.12044c6e5ffb579682543ff158643478.
2017-10-13 05:21:57,064 INFO [StoreCloserThread-ha_cut_rum,5,1496371215377.294da0b7c17e56e90e5c303929ad1300.-1] regionserver.HStore: Closed cf
2017-10-13 05:21:57,065 INFO [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Closed ha_cut_rum,5,1496371215377.294da0b7c17e56e90e5c303929ad1300.
2017-10-13 05:21:57,065 INFO [StoreCloserThread-ha_cut_app,,1496371225239.c15c61e71af14fae604525a3b4f4fb50.-1] regionserver.HStore: Closed cf
2017-10-13 05:21:57,066 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed ha_cut_app,,1496371225239.c15c61e71af14fae604525a3b4f4fb50.
2017-10-13 05:21:57,066 INFO [StoreCloserThread-trackTreeTable2479w,7_1936275,1499429207444.387542d8ba5ce812f0bb7cbaf62dcdab.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,067 INFO [StoreCloserThread-trackTreeTable2485w,4_9153,1502983793818.cbf941b8645b4c1a96f0ff793a0193d4.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,067 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed trackTreeTable2479w,7_1936275,1499429207444.387542d8ba5ce812f0bb7cbaf62dcdab.
2017-10-13 05:21:57,067 INFO [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Closed trackTreeTable2485w,4_9153,1502983793818.cbf941b8645b4c1a96f0ff793a0193d4.
2017-10-13 05:21:57,068 INFO [StoreCloserThread-mobileTable2481w,,1500508917344.527b1618e350314bd976cf4e92d6d183.-1] regionserver.HStore: Closed mobileMassage
2017-10-13 05:21:57,069 INFO [StoreCloserThread-trackTreeTable2488w,,1504752305180.c2827a0faab7291a25664c72a1f45fb5.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,069 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed mobileTable2481w,,1500508917344.527b1618e350314bd976cf4e92d6d183.
2017-10-13 05:21:57,069 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed trackTreeTable2488w,,1504752305180.c2827a0faab7291a25664c72a1f45fb5.
2017-10-13 05:21:57,072 INFO [StoreCloserThread-mobileTable2488w,,1504742413506.71749f3723c1ffdb29d5d123ba26e9ee.-1] regionserver.HStore: Closed mobileMassage
2017-10-13 05:21:57,073 INFO [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Closed mobileTable2488w,,1504742413506.71749f3723c1ffdb29d5d123ba26e9ee.
2017-10-13 05:21:57,073 INFO [StoreCloserThread-ha_cut_mobile,8,1500866422681.b3c4d2e5da4a3719daa70ed28be759be.-1] regionserver.HStore: Closed cf
2017-10-13 05:21:57,073 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed ha_cut_mobile,8,1500866422681.b3c4d2e5da4a3719daa70ed28be759be.
2017-10-13 05:21:57,075 INFO [StoreCloserThread-trackTreeTable2488w,7_4762,1504911529272.3fe8daf2285624da99c5c6726bd569b3.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,075 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed trackTreeTable2488w,7_4762,1504911529272.3fe8daf2285624da99c5c6726bd569b3.
2017-10-13 05:21:57,075 INFO [StoreCloserThread-trackTreeTable2493w,4_9353,1507871482977.fc849c2fffb2662fdbcf4943a343a113.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,075 ERROR [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Memstore size is 19181944
2017-10-13 05:21:57,075 INFO [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Closed trackTreeTable2493w,4_9353,1507871482977.fc849c2fffb2662fdbcf4943a343a113.
2017-10-13 05:21:57,076 INFO [StoreCloserThread-deviceInfo,,1496371626913.1c0e20556986a77ca25bb6b4a9a8c474.-1] regionserver.HStore: Closed infos
2017-10-13 05:21:57,077 INFO [StoreCloserThread-trackTreeTable2485w,2_4529,1503286569849.5de28e2afe7c23629638c5bc2967a376.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,077 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed deviceInfo,,1496371626913.1c0e20556986a77ca25bb6b4a9a8c474.
2017-10-13 05:21:57,077 INFO [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Closed trackTreeTable2485w,2_4529,1503286569849.5de28e2afe7c23629638c5bc2967a376.
2017-10-13 05:21:57,078 INFO [StoreCloserThread-ha_cut_mobile,9,1500866422681.098ff00b80acc2939e53187faad5c58a.-1] regionserver.HStore: Closed cf
2017-10-13 05:21:57,078 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed ha_cut_mobile,9,1500866422681.098ff00b80acc2939e53187faad5c58a.
2017-10-13 05:21:57,078 INFO [StoreCloserThread-trackTreeTable2483w,7_477,1501855313509.3754e9eae26669b0c9cae5a637f18a28.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,079 INFO [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Closed trackTreeTable2483w,7_477,1501855313509.3754e9eae26669b0c9cae5a637f18a28.
2017-10-13 05:21:57,079 INFO [StoreCloserThread-ha_cut_app,2,1496371225239.bfda7a345a6f706682364e8313a92641.-1] regionserver.HStore: Closed cf
2017-10-13 05:21:57,079 ERROR [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Memstore size is 22504536
2017-10-13 05:21:57,079 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed ha_cut_app,2,1496371225239.bfda7a345a6f706682364e8313a92641.
2017-10-13 05:21:57,080 INFO [StoreCloserThread-trackTreeTable2480w,,1500009309073.7e416bc5cd815ed61847db702c0d2780.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,080 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed trackTreeTable2480w,,1500009309073.7e416bc5cd815ed61847db702c0d2780.
2017-10-13 05:21:57,081 INFO [StoreCloserThread-mobileTable2487w,,1504082949934.d95d8eefeb9af758bc51e6f340d33f47.-1] regionserver.HStore: Closed mobileMassage
2017-10-13 05:21:57,081 INFO [StoreCloserThread-mobileTable2493w,,1507684033802.6cccc3ea4503884eebc9dfff38ec7193.-1] regionserver.HStore: Closed mobileMassage
2017-10-13 05:21:57,081 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed mobileTable2487w,,1504082949934.d95d8eefeb9af758bc51e6f340d33f47.
2017-10-13 05:21:57,081 INFO [StoreCloserThread-ha_cut_app,5,1496371225239.d11a895a59ad0cc4d4e57cccf81beb97.-1] regionserver.HStore: Closed cf
2017-10-13 05:21:57,081 ERROR [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Memstore size is 321920
2017-10-13 05:21:57,082 INFO [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Closed mobileTable2493w,,1507684033802.6cccc3ea4503884eebc9dfff38ec7193.
2017-10-13 05:21:57,082 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed ha_cut_app,5,1496371225239.d11a895a59ad0cc4d4e57cccf81beb97.
2017-10-13 05:21:57,083 INFO [StoreCloserThread-ha_cut_app,9,1496371225239.9d5b4f08c6dd58ecb51fe01ec7087b1d.-1] regionserver.HStore: Closed cf
2017-10-13 05:21:57,083 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed ha_cut_app,9,1496371225239.9d5b4f08c6dd58ecb51fe01ec7087b1d.
2017-10-13 05:21:57,085 INFO [StoreCloserThread-trackTreeTable2492w,,1507183313550.4e69bab41ed8d388b1bcfcd78d5ca587.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,085 INFO [StoreCloserThread-mobileTable2485w,,1502928017181.4c21bddea069bc45b29a5c49cad59d4c.-1] regionserver.HStore: Closed mobileMassage
2017-10-13 05:21:57,085 INFO [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Closed trackTreeTable2492w,,1507183313550.4e69bab41ed8d388b1bcfcd78d5ca587.
2017-10-13 05:21:57,085 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed mobileTable2485w,,1502928017181.4c21bddea069bc45b29a5c49cad59d4c.
2017-10-13 05:21:57,087 INFO [StoreCloserThread-mobileTable2492w,,1507161810879.b5920ef1827bc9d07d326fb69ef5916f.-1] regionserver.HStore: Closed mobileMassage
2017-10-13 05:21:57,087 ERROR [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Memstore size is 10920
2017-10-13 05:21:57,087 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed mobileTable2492w,,1507161810879.b5920ef1827bc9d07d326fb69ef5916f.
2017-10-13 05:21:57,088 INFO [StoreCloserThread-trackTreeTable2482w,2_496509,1501665050015.6f7a356f5cdfc980ad082aa2b6321b5d.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,088 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed trackTreeTable2482w,2_496509,1501665050015.6f7a356f5cdfc980ad082aa2b6321b5d.
2017-10-13 05:21:57,089 INFO [StoreCloserThread-trackTreeTable2487w,,1504257297010.283ebe3d4e70711b7528f5bb3cb14142.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,089 INFO [StoreCloserThread-ha_cut_app,7,1496371225239.bb8ded4b65638bd65d94c4e54ff7ccc6.-1] regionserver.HStore: Closed cf
2017-10-13 05:21:57,089 INFO [StoreCloserThread-ha_cut_rum,2,1496371215377.4ef645821758f03d9632678684e90985.-1] regionserver.HStore: Closed cf
2017-10-13 05:21:57,089 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed ha_cut_app,7,1496371225239.bb8ded4b65638bd65d94c4e54ff7ccc6.
2017-10-13 05:21:57,089 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed trackTreeTable2487w,,1504257297010.283ebe3d4e70711b7528f5bb3cb14142.
2017-10-13 05:21:57,089 ERROR [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Memstore size is 5224
2017-10-13 05:21:57,090 INFO [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Closed ha_cut_rum,2,1496371215377.4ef645821758f03d9632678684e90985.
2017-10-13 05:21:57,091 INFO [StoreCloserThread-trackTreeTable2490w,5_10786,1505961945042.7189ad8103ee0ad51352811ad593b467.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,091 INFO [StoreCloserThread-mobileTable2491w,,1506498657842.ffb07c8bb53048d735413ea6433bc3db.-1] regionserver.HStore: Closed mobileMassage
2017-10-13 05:21:57,091 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed trackTreeTable2490w,5_10786,1505961945042.7189ad8103ee0ad51352811ad593b467.
2017-10-13 05:21:57,091 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed mobileTable2491w,,1506498657842.ffb07c8bb53048d735413ea6433bc3db.
2017-10-13 05:21:57,091 INFO [StoreCloserThread-ha_cut_rum,,1496371215377.fd944e2ad8cf591626d0e1aa47b3ca5d.-1] regionserver.HStore: Closed cf
2017-10-13 05:21:57,092 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed ha_cut_rum,,1496371215377.fd944e2ad8cf591626d0e1aa47b3ca5d.
2017-10-13 05:21:57,092 INFO [StoreCloserThread-trackTreeTable2486w,2_528002,1503641361425.5e50db42c70a72a48673d39bbf3d3061.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,092 INFO [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Closed trackTreeTable2486w,2_528002,1503641361425.5e50db42c70a72a48673d39bbf3d3061.
2017-10-13 05:21:57,093 INFO [StoreCloserThread-mobileTable2482w,,1501113672623.1dec8fcba7931725e4a1c2a3cc358670.-1] regionserver.HStore: Closed mobileMassage
2017-10-13 05:21:57,093 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed mobileTable2482w,,1501113672623.1dec8fcba7931725e4a1c2a3cc358670.
2017-10-13 05:21:57,094 INFO [StoreCloserThread-mobileTable2484w,,1502323468755.f9d347975bd88315b8d6a1447e5e6d49.-1] regionserver.HStore: Closed mobileMassage
2017-10-13 05:21:57,095 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed mobileTable2484w,,1502323468755.f9d347975bd88315b8d6a1447e5e6d49.
2017-10-13 05:21:57,095 INFO [StoreCloserThread-trackTreeTable2481w,,1500863925601.76aa05afb46486aa215d9204cbf75806.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,095 INFO [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Closed trackTreeTable2481w,,1500863925601.76aa05afb46486aa215d9204cbf75806.
2017-10-13 05:21:57,095 INFO [StoreCloserThread-ha_cut_rum,1,1496371215377.3bf9b0846aab1c0fe846513ed3bf8559.-1] regionserver.HStore: Closed cf
2017-10-13 05:21:57,095 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed ha_cut_rum,1,1496371215377.3bf9b0846aab1c0fe846513ed3bf8559.
2017-10-13 05:21:57,096 INFO [StoreCloserThread-ha_cut_mobile,6,1500866422681.5ac7bc299f3c3aead8d1fe604718a5b3.-1] regionserver.HStore: Closed cf
2017-10-13 05:21:57,096 INFO [StoreCloserThread-trackTreeTable2491w,,1506566053131.cb4d9dc34ea2fcb46fdbb90483620fb8.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,096 INFO [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Closed ha_cut_mobile,6,1500866422681.5ac7bc299f3c3aead8d1fe604718a5b3.
2017-10-13 05:21:57,096 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed trackTreeTable2491w,,1506566053131.cb4d9dc34ea2fcb46fdbb90483620fb8.
2017-10-13 05:21:57,097 INFO [StoreCloserThread-trackTreeTable2485w,,1503286569849.7c7d310b2d9c837a2ffd142f80a99abc.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,097 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed trackTreeTable2485w,,1503286569849.7c7d310b2d9c837a2ffd142f80a99abc.
2017-10-13 05:21:57,097 INFO [StoreCloserThread-ha_cut_rum,9,1496371215377.35d1e6311322fd4702158083104ca39b.-1] regionserver.HStore: Closed cf
2017-10-13 05:21:57,097 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed ha_cut_rum,9,1496371215377.35d1e6311322fd4702158083104ca39b.
2017-10-13 05:21:57,098 INFO [StoreCloserThread-ha_cut_app,4,1496371225239.04f75774d8c64d6b64bc23e7e136a580.-1] regionserver.HStore: Closed cf
2017-10-13 05:21:57,098 INFO [StoreCloserThread-ha_cut_rum,6,1496371215377.041d30379d6434cfa064f4c600c6dc63.-1] regionserver.HStore: Closed cf
2017-10-13 05:21:57,098 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed ha_cut_rum,6,1496371215377.041d30379d6434cfa064f4c600c6dc63.
2017-10-13 05:21:57,098 INFO [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Closed ha_cut_app,4,1496371225239.04f75774d8c64d6b64bc23e7e136a580.
2017-10-13 05:21:57,099 INFO [StoreCloserThread-trackTreeTable2489w,,1505703582107.b115f7b04b0ecc6bcbe2829ef2d1039d.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,099 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed trackTreeTable2489w,,1505703582107.b115f7b04b0ecc6bcbe2829ef2d1039d.
2017-10-13 05:21:57,100 INFO [StoreCloserThread-mobileTable2479w,,1499299240091.f7323ce66c7575df7df22f7619fd8e5c.-1] regionserver.HStore: Closed mobileMassage
2017-10-13 05:21:57,100 INFO [StoreCloserThread-trackTreeTable2490w,2_54585,1506132030839.bcb15a9bfcef09e53a731ef7c8d251e6.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,100 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed mobileTable2479w,,1499299240091.f7323ce66c7575df7df22f7619fd8e5c.
2017-10-13 05:21:57,100 INFO [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Closed trackTreeTable2490w,2_54585,1506132030839.bcb15a9bfcef09e53a731ef7c8d251e6.
2017-10-13 05:21:57,100 INFO [StoreCloserThread-trackTreeTable2486w,,1503641361425.4a4fb7a5f1c25283d93be3742db53cb3.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,100 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed trackTreeTable2486w,,1503641361425.4a4fb7a5f1c25283d93be3742db53cb3.
2017-10-13 05:21:57,101 INFO [StoreCloserThread-ha_cut_mobile,7,1500866422681.8c6b0d22ba519da2eec59313858268fe.-1] regionserver.HStore: Closed cf
2017-10-13 05:21:57,101 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed ha_cut_mobile,7,1500866422681.8c6b0d22ba519da2eec59313858268fe.
2017-10-13 05:21:57,101 INFO [StoreCloserThread-trackTreeTable2489w,2_463014,1505703582107.43fc1c0d488d7bf169bdd04fd12e3478.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,102 INFO [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Closed trackTreeTable2489w,2_463014,1505703582107.43fc1c0d488d7bf169bdd04fd12e3478.
2017-10-13 05:21:57,102 INFO [StoreCloserThread-trackTreeTable2484w,,1502894536820.ee5a08db52695c13e2b945f2df8c88bc.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,103 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed trackTreeTable2484w,,1502894536820.ee5a08db52695c13e2b945f2df8c88bc.
2017-10-13 05:21:57,103 INFO [StoreCloserThread-trackTreeTable2482w,,1501665050015.7901de7ac34e63935d4af8f20352ebb6.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,103 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed trackTreeTable2482w,,1501665050015.7901de7ac34e63935d4af8f20352ebb6.
2017-10-13 05:21:57,104 INFO [StoreCloserThread-trackTreeTable2483w,4_9574,1501855313509.463f7429f78fa69a68e8416a44d7e7c5.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,104 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed trackTreeTable2483w,4_9574,1501855313509.463f7429f78fa69a68e8416a44d7e7c5.
2017-10-13 05:21:57,104 INFO [StoreCloserThread-trackTreeTable2492w,7_41145,1507476471082.b64e0dee7d56a413d6e896c9eae0e4ba.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,105 INFO [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Closed trackTreeTable2492w,7_41145,1507476471082.b64e0dee7d56a413d6e896c9eae0e4ba.
2017-10-13 05:21:57,105 INFO [StoreCloserThread-mobileTable2490w,,1505951214411.0383152024e5d2828392c37c57422e63.-1] regionserver.HStore: Closed mobileMassage
2017-10-13 05:21:57,105 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed mobileTable2490w,,1505951214411.0383152024e5d2828392c37c57422e63.
2017-10-13 05:21:57,106 INFO [StoreCloserThread-ha_cut_mobile,2,1500866422681.d612ac81be610089dbc8860fe6e581f5.-1] regionserver.HStore: Closed cf
2017-10-13 05:21:57,106 INFO [StoreCloserThread-trackTreeTable2489w,4_904,1505465918965.7dad963037885a10f408bd32cfdf2c9f.-1] regionserver.HStore: Closed code
2017-10-13 05:21:57,106 ERROR [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Memstore size is 216224
2017-10-13 05:21:57,106 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed ha_cut_mobile,2,1500866422681.d612ac81be610089dbc8860fe6e581f5.
2017-10-13 05:21:57,107 INFO [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Closed trackTreeTable2489w,4_904,1505465918965.7dad963037885a10f408bd32cfdf2c9f.
2017-10-13 05:15:04,325 INFO [StoreCloserThread-trackTreeTable2484w,1_256008,1502894536820.ed7240bd47e7d40bce876fd2b113d956.-1] regionserver.HStore: Closed code
2017-10-13 05:15:04,325 INFO [StoreCloserThread-trackTreeTable2479w,2_195531,1499830310451.a5972f046388ebf00e28470d1847ae0b.-1] regionserver.HStore: Closed code
2017-10-13 05:15:04,325 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed trackTreeTable2484w,1_256008,1502894536820.ed7240bd47e7d40bce876fd2b113d956.
2017-10-13 05:15:04,325 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed trackTreeTable2479w,2_195531,1499830310451.a5972f046388ebf00e28470d1847ae0b.
2017-10-13 05:15:04,326 INFO [StoreCloserThread-trackTreeTable2481w,2_195962,1500863925601.608ffa9f42ca8a50254d8009af3ff36e.-1] regionserver.HStore: Closed code
2017-10-13 05:15:04,326 INFO [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Closed trackTreeTable2481w,2_195962,1500863925601.608ffa9f42ca8a50254d8009af3ff36e.
2017-10-13 05:15:04,327 INFO [StoreCloserThread-trackTreeTable2487w,7_487104,1504618231759.a1d7621ad29dce96ab3c4bcfe6a7874d.-1] regionserver.HStore: Closed code
2017-10-13 05:15:04,327 INFO [StoreCloserThread-trackTreeTable2490w,,1506132030839.edbc5e77ab144d4727e4a0a50cd16130.-1] regionserver.HStore: Closed code
2017-10-13 05:15:04,327 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed trackTreeTable2487w,7_487104,1504618231759.a1d7621ad29dce96ab3c4bcfe6a7874d.
2017-10-13 05:15:04,327 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed trackTreeTable2490w,,1506132030839.edbc5e77ab144d4727e4a0a50cd16130.
2017-10-13 05:15:04,327 INFO [StoreCloserThread-trackTreeTable2484w,3_81844332,1502892195452.163af29e4c95f490149b002405c9a064.-1] regionserver.HStore: Closed code
2017-10-13 05:15:04,327 INFO [StoreCloserThread-ha_cut_rum,4,1496371215377.c5bf6349864b88ca38d7dfadb6f549ac.-1] regionserver.HStore: Closed cf
2017-10-13 05:15:04,327 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed ha_cut_rum,4,1496371215377.c5bf6349864b88ca38d7dfadb6f549ac.
2017-10-13 05:15:04,328 INFO [StoreCloserThread-ha_cut_rum,8,1496371215377.b9aa19806d917b5d47ddb86d556ece98.-1] regionserver.HStore: Closed cf
2017-10-13 05:15:04,328 INFO [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Closed trackTreeTable2484w,3_81844332,1502892195452.163af29e4c95f490149b002405c9a064.
2017-10-13 05:15:04,328 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed ha_cut_rum,8,1496371215377.b9aa19806d917b5d47ddb86d556ece98.
2017-10-13 05:15:04,328 INFO [StoreCloserThread-ha_cut_app,6,1496371225239.a60cacd3ed576ca4cae73975bc75a8be.-1] regionserver.HStore: Closed cf
2017-10-13 05:15:04,328 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed ha_cut_app,6,1496371225239.a60cacd3ed576ca4cae73975bc75a8be.
2017-10-13 05:15:04,329 INFO [StoreCloserThread-trackTreeTable2481w,7_195332,1500590391057.5bc4105c1360b501cbd07ccd59965e40.-1] regionserver.HStore: Closed code
2017-10-13 05:15:04,329 INFO [StoreCloserThread-trackTreeTable2492w,4_8607,1507476471082.63cfea9b2fdecaf527e2e2738bf95baa.-1] regionserver.HStore: Closed code
2017-10-13 05:15:04,330 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed trackTreeTable2481w,7_195332,1500590391057.5bc4105c1360b501cbd07ccd59965e40.
2017-10-13 05:15:04,330 INFO [RS_CLOSE_REGION-hahd1:60020-1] regionserver.HRegion: Closed trackTreeTable2492w,4_8607,1507476471082.63cfea9b2fdecaf527e2e2738bf95baa.
2017-10-13 05:15:04,330 INFO [StoreCloserThread-ha_cut_mobile,4,1500866422681.51bf3b75d86a1d78fee391cfe53168eb.-1] regionserver.HStore: Closed cf
2017-10-13 05:15:04,330 INFO [StoreCloserThread-trackTreeTable2484w,5_10815,1502762506624.7c2b61b63e574475501ce6389239c13f.-1] regionserver.HStore: Closed code
2017-10-13 05:15:04,331 INFO [RS_CLOSE_REGION-hahd1:60020-2] regionserver.HRegion: Closed ha_cut_mobile,4,1500866422681.51bf3b75d86a1d78fee391cfe53168eb.
2017-10-13 05:15:04,331 INFO [RS_CLOSE_REGION-hahd1:60020-0] regionserver.HRegion: Closed trackTreeTable2484w,5_10815,1502762506624.7c2b61b63e574475501ce6389239c13f.
2017-10-13 05:21:59,227 INFO [regionserver/hahd1/10.211.18.39:60020-SendThread(hahm2:2181)] zookeeper.ClientCnxn: Opening socket connection to server hahm2/10.211.18.48:2181. Will not attempt to authenticate using SASL (unknown error)
2017-10-13 05:21:59,230 INFO [regionserver/hahd1/10.211.18.39:60020-SendThread(hahm2:2181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.211.18.39:45682, server: hahm2/10.211.18.48:2181
2017-10-13 05:21:59,230 INFO [regionserver/hahd1/10.211.18.39:60020.leaseChecker] regionserver.Leases: regionserver/hahd1/10.211.18.39:60020.leaseChecker closing leases
2017-10-13 05:21:59,230 INFO [regionserver/hahd1/10.211.18.39:60020.leaseChecker] regionserver.Leases: regionserver/hahd1/10.211.18.39:60020.leaseChecker closed leases
2017-10-13 05:21:59,232 INFO [regionserver/hahd1/10.211.18.39:60020-SendThread(hahm2:2181)] zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x35f141db127000e has expired, closing socket connection
2017-10-13 05:21:59,233 WARN [regionserver/hahd1/10.211.18.39:60020-EventThread] client.ConnectionManager$HConnectionImplementation: This client just lost it's session with ZooKeeper, closing it. It will be recreated next time someone needs it
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:525)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:436)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
2017-10-13 05:21:59,235 INFO [regionserver/hahd1/10.211.18.39:60020-EventThread] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x35f141db127000e
2017-10-13 05:21:59,235 INFO [regionserver/hahd1/10.211.18.39:60020-EventThread] zookeeper.ClientCnxn: EventThread shut down
2017-10-13 05:21:59,231 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2051ms
No GCs detected
2017-10-13 05:21:59,329 INFO [regionserver/hahd1/10.211.18.39:60020] zookeeper.ZooKeeper: Session: 0x15f141db117000c closed
2017-10-13 05:21:59,329 INFO [regionserver/hahd1/10.211.18.39:60020-EventThread] zookeeper.ClientCnxn: EventThread shut down
2017-10-13 05:21:59,329 INFO [regionserver/hahd1/10.211.18.39:60020] regionserver.CompactSplitThread: Waiting for Split Thread to finish...
2017-10-13 05:21:59,333 INFO [regionserver/hahd1/10.211.18.39:60020] regionserver.CompactSplitThread: Waiting for Merge Thread to finish...
2017-10-13 05:21:59,333 INFO [regionserver/hahd1/10.211.18.39:60020] regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...
2017-10-13 05:21:59,334 INFO [regionserver/hahd1/10.211.18.39:60020] regionserver.CompactSplitThread: Waiting for Small Compaction Thread to finish...
2017-10-13 05:21:59,334 INFO [regionserver/hahd1/10.211.18.39:60020] regionserver.HRegionServer: Waiting on 1 regions to close
2017-10-13 05:21:59,336 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore: Closed info
2017-10-13 05:21:59,336 ERROR [RS_CLOSE_META-hahd1:60020-0] regionserver.HRegion: Memstore size is 80344
2017-10-13 05:21:59,337 INFO [RS_CLOSE_META-hahd1:60020-0] regionserver.HRegion: Closed hbase:meta,,1.1588230740
2017-10-13 05:22:01,135 INFO [regionserver/hahd1/10.211.18.39:60020] regionserver.HRegionServer: stopping server hahd1,60020,1507871462810; all regions closed.
2017-10-13 05:22:01,139 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1403ms
No GCs detected
2017-10-13 05:22:01,151 INFO [regionserver/hahd1/10.211.18.39:60020] retry.RetryInvocationHandler: Exception while invoking complete of class ClientNamenodeProtocolTranslatorPB over hahm1/10.211.18.38:9000. Trying to fail over immediately.
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category WRITE is not supported in state standby
at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1773)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1402)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:3655)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:730)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.complete(AuthorizationProviderProxyClientProtocol.java:243)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:527)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)

at org.apache.hadoop.ipc.Client.call(Client.java:1468)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy19.complete(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:443)
at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy20.complete(Unknown Source)
at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
at com.sun.proxy.$Proxy21.complete(Unknown Source)
at org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2283)
at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2267)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.close(ProtobufLogWriter.java:131)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.shutdown(FSHLog.java:1149)
at org.apache.hadoop.hbase.wal.DefaultWALProvider.shutdown(DefaultWALProvider.java:114)
at org.apache.hadoop.hbase.wal.WALFactory.shutdown(WALFactory.java:215)
at org.apache.hadoop.hbase.regionserver.HRegionServer.shutdownWAL(HRegionServer.java:1248)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1003)
at java.lang.Thread.run(Thread.java:744)
2017-10-13 05:22:03,398 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1759ms
No GCs detected
2017-10-13 05:22:03,395 INFO [regionserver/hahd1/10.211.18.39:60020] regionserver.Leases: regionserver/hahd1/10.211.18.39:60020 closing leases
2017-10-13 05:22:03,399 INFO [regionserver/hahd1/10.211.18.39:60020] regionserver.Leases: regionserver/hahd1/10.211.18.39:60020 closed leases
2017-10-13 05:22:05,427 INFO [RS_OPEN_META-hahd1:60020-0-MetaLogRoller] regionserver.LogRoller: LogRoller exiting.
2017-10-13 05:22:05,428 INFO [regionserver/hahd1/10.211.18.39:60020.periodicFlusher] regionserver.HRegionServer$PeriodicMemstoreFlusher: regionserver/hahd1/10.211.18.39:60020.periodicFlusher exiting
2017-10-13 05:22:05,432 INFO [regionserver/hahd1/10.211.18.39:60020.logRoller] regionserver.LogRoller: LogRoller exiting.
2017-10-13 05:22:05,434 WARN [regionserver/hahd1/10.211.18.39:60020] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=hahd1:2181,hahm2:2181,hahm1:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/replication/rs/hahd1,60020,1507871462810
2017-10-13 05:22:05,434 INFO [regionserver/hahd1/10.211.18.39:60020] util.RetryCounter: Sleeping 1000ms before retry #0...
2017-10-13 05:22:07,393 WARN [regionserver/hahd1/10.211.18.39:60020] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=hahd1:2181,hahm2:2181,hahm1:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/replication/rs/hahd1,60020,1507871462810
2017-10-13 05:22:07,393 INFO [regionserver/hahd1/10.211.18.39:60020] util.RetryCounter: Sleeping 2000ms before retry #1...
2017-10-13 05:22:09,393 WARN [regionserver/hahd1/10.211.18.39:60020] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=hahd1:2181,hahm2:2181,hahm1:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/replication/rs/hahd1,60020,1507871462810
2017-10-13 05:22:09,394 INFO [regionserver/hahd1/10.211.18.39:60020] util.RetryCounter: Sleeping 4000ms before retry #2...
2017-10-13 05:22:13,394 WARN [regionserver/hahd1/10.211.18.39:60020] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=hahd1:2181,hahm2:2181,hahm1:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/replication/rs/hahd1,60020,1507871462810
2017-10-13 05:22:13,395 INFO [regionserver/hahd1/10.211.18.39:60020] util.RetryCounter: Sleeping 8000ms before retry #3...
2017-10-13 05:22:21,587 WARN [regionserver/hahd1/10.211.18.39:60020] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=hahd1:2181,hahm2:2181,hahm1:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/replication/rs/hahd1,60020,1507871462810
2017-10-13 05:22:21,588 ERROR [regionserver/hahd1/10.211.18.39:60020] zookeeper.RecoverableZooKeeper: ZooKeeper getChildren failed after 4 attempts
2017-10-13 05:22:21,588 INFO [regionserver/hahd1/10.211.18.39:60020] ipc.RpcServer: Stopping server on 60020
2017-10-13 05:22:21,588 INFO [RpcServer.listener,port=60020] ipc.RpcServer: RpcServer.listener,port=60020: stopping
2017-10-13 05:22:21,590 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped
2017-10-13 05:22:21,591 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping
2017-10-13 05:22:21,590 WARN [regionserver/hahd1/10.211.18.39:60020] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=hahd1:2181,hahm2:2181,hahm1:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/hahd1,60020,1507871462810
2017-10-13 05:22:21,592 INFO [regionserver/hahd1/10.211.18.39:60020] util.RetryCounter: Sleeping 1000ms before retry #0...
2017-10-13 05:22:23,387 WARN [regionserver/hahd1/10.211.18.39:60020] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=hahd1:2181,hahm2:2181,hahm1:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/hahd1,60020,1507871462810
2017-10-13 05:22:23,388 INFO [regionserver/hahd1/10.211.18.39:60020] util.RetryCounter: Sleeping 2000ms before retry #1...
2017-10-13 05:22:25,388 WARN [regionserver/hahd1/10.211.18.39:60020] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=hahd1:2181,hahm2:2181,hahm1:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/hahd1,60020,1507871462810
2017-10-13 05:22:25,388 INFO [regionserver/hahd1/10.211.18.39:60020] util.RetryCounter: Sleeping 4000ms before retry #2...
2017-10-13 05:22:29,389 WARN [regionserver/hahd1/10.211.18.39:60020] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=hahd1:2181,hahm2:2181,hahm1:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/hahd1,60020,1507871462810
2017-10-13 05:22:29,389 INFO [regionserver/hahd1/10.211.18.39:60020] util.RetryCounter: Sleeping 8000ms before retry #3...
2017-10-13 05:22:37,389 WARN [regionserver/hahd1/10.211.18.39:60020] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=hahd1:2181,hahm2:2181,hahm1:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/hahd1,60020,1507871462810
2017-10-13 05:22:37,390 ERROR [regionserver/hahd1/10.211.18.39:60020] zookeeper.RecoverableZooKeeper: ZooKeeper delete failed after 4 attempts
2017-10-13 05:22:37,390 WARN [regionserver/hahd1/10.211.18.39:60020] regionserver.HRegionServer: Failed deleting my ephemeral node
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/hahd1,60020,1507871462810
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:179)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1286)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1275)
at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1339)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1029)
at java.lang.Thread.run(Thread.java:744)
2017-10-13 05:22:37,390 INFO [regionserver/hahd1/10.211.18.39:60020] regionserver.HRegionServer: stopping server hahd1,60020,1507871462810; zookeeper connection closed.
2017-10-13 05:22:37,390 INFO [regionserver/hahd1/10.211.18.39:60020] regionserver.HRegionServer: regionserver/hahd1/10.211.18.39:60020 exiting
2017-10-13 05:22:37,391 ERROR [main] regionserver.HRegionServerCommandLine: Region server exiting
java.lang.RuntimeException: HRegionServer Aborted
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:68)
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2540)
2017-10-13 05:22:37,394 INFO [Thread-5] regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@28d9e87f
2017-10-13 05:22:37,395 INFO [Thread-5] regionserver.ShutdownHook: Starting fs shutdown hook thread.
2017-10-13 05:22:37,396 INFO [Thread-5] regionserver.ShutdownHook: Shutdown hook finished.
已邀请:

OpenSkill - OpenSkill官方账号

赞同来自: chris

zkstate.png

如上图所示,是hbase和zookeeper之间会话的状态图,出现session expired的情况,可能如下:
 
1、JVM的垃圾回收,如果配置hbase的heap经常发生垃圾回收,暂停状态会导致心跳失败,可以从hbase JVM GC日志来识别这一点,HBase还将在服务日志中包含消息。
 
2、Host-level停顿。节点将JVM的部分移出内存并转换为交换(由磁盘支持),会导致整个JVM没有任何垃圾收集。HBase通常也会注意到这一点,但是在不发生任何垃圾收集的情况下报告暂停。
 
3、zookeepr连接数超出,可以用netstat命令对比maxClientCnxns值,调整zoo.cfg中的maxClientCnxns值大小
 
4、超时时间过小,可以设置zookeepr中minSessionTimeout和maxSessionTimeout来调节,而且要注意,hbase也有连接zk超时的session设置,hbase-site.xml中设置如下:
<property>
<name>zookeeper.session.timeout</name>
<value>120000</value>
</property>
参考:https://community.hortonworks.com/questions/11779/hbase-master-shutting-down-with-zookeeper-delete-f.html  https://www.coder4.com/archives/3181  http://blog.csdn.net/u010316405/article/details/52396932  

要回复问题请先登录注册