Code Monkey home page Code Monkey logo

bwbwchen / mapreduce Goto Github PK

View Code? Open in Web Editor NEW
213.0 213.0 13.0 2.66 MB

An easy-to-use Map Reduce Go parallel-computing framework inspired by 2021 6.824 lab1. It supports multiple workers threads on a single machine and multiple processes on a single machine right now.

Home Page: https://pkg.go.dev/github.com/BWbwchen/MapReduce

License: MIT License

Makefile 5.91% Go 93.74% Nix 0.34%
golang mapreduce mapreduce-go parallel-computing

mapreduce's Introduction

Hello! I am Bo-Wei Chen, Tim. ๐Ÿš€๐Ÿš€

I'm focused on Distributed Systems and Physical Design - Placement.


mapreduce's People

Contributors

bwbwchen avatar kiarash8112 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

mapreduce's Issues

data racing when run ```make test_multi_node ```

when @kiarash8112 run make test_multi_node , he get following error

rm output/*
rm: cannot remove 'output/*': No such file or directory
make: [Makefile:10: clean] Error 1 (ignored)
rm cmd/*.so
rm: cannot remove 'cmd/*.so': No such file or directory
make: [Makefile:11: clean] Error 1 (ignored)
go build -race -buildmode=plugin -o cmd/wc.so ./mrapps/wc.go
go build -race -buildmode=plugin -o cmd/merge.so ./mrapps/merge.go
rm output/*
rm: cannot remove 'output/*': No such file or directory
make: [Makefile:27: test_multi_node] Error 1 (ignored)
rm mr-out-*
rm: cannot remove 'mr-out-*': No such file or directory
make: [Makefile:28: test_multi_node] Error 1 (ignored)
go run -race cmd/master.go -i 'txt/*' -p 'cmd/wc.so' -r 4 -w 8 &
sleep 1
go run -race cmd/worker.go -i 'txt/*' -p 'cmd/wc.so' -r 4 -w 1 &
go run -race cmd/worker.go -i 'txt/*' -p 'cmd/wc.so' -r 4 -w 2 &
go run -race cmd/worker.go -i 'txt/*' -p 'cmd/wc.so' -r 4 -w 3 &
go run -race cmd/worker.go -i 'txt/*' -p 'cmd/wc.so' -r 4 -w 4 &
go run -race cmd/worker.go -i 'txt/*' -p 'cmd/wc.so' -r 4 -w 5 &
go run -race cmd/worker.go -i 'txt/*' -p 'cmd/wc.so' -r 4 -w 6 &
go run -race cmd/worker.go -i 'txt/*' -p 'cmd/wc.so' -r 4 -w 7 &
go run -race cmd/worker.go -i 'txt/*' -p 'cmd/wc.so' -r 4 -w 8 
INFO[0000] [Master] Master gRPC server start            
TRAC[0000] [Master] Wait for enough workers             
INFO[0000] Worker gRPC server start                     
INFO[0000] Worker load plugin finish                    
TRAC[0000] With Time out                                
TRAC[0000] Start RPC call                               
INFO[0000] [Master] Worker register success             
TRAC[0000] End RPC call                                 
INFO[0000] Worker register itself finish                
INFO[0000] Worker gRPC server start                     
INFO[0000] Worker load plugin finish                    
TRAC[0000] With Time out                                
TRAC[0000] Start RPC call                               
INFO[0001] [Master] Worker register success             
TRAC[0000] End RPC call                                 
INFO[0000] Worker register itself finish                
INFO[0000] Worker gRPC server start                     
INFO[0000] Worker gRPC server start                     
INFO[0000] Worker load plugin finish                    
TRAC[0000] With Time out                                
TRAC[0000] Start RPC call                               
INFO[0001] [Master] Worker register success             
TRAC[0000] End RPC call                                 
INFO[0000] Worker register itself finish                
INFO[0000] Worker load plugin finish                    
TRAC[0000] With Time out                                
TRAC[0000] Start RPC call                               
INFO[0001] [Master] Worker register success             
TRAC[0000] End RPC call                                 
INFO[0000] Worker register itself finish                
INFO[0000] Worker gRPC server start                     
INFO[0000] Worker load plugin finish                    
TRAC[0000] With Time out                                
TRAC[0000] Start RPC call                               
INFO[0001] [Master] Worker register success             
TRAC[0000] End RPC call                                 
INFO[0000] Worker register itself finish                
INFO[0000] Worker gRPC server start                     
INFO[0000] Worker gRPC server start                     
INFO[0000] Worker load plugin finish                    
TRAC[0000] With Time out                                
TRAC[0000] Start RPC call                               
INFO[0001] [Master] Worker register success             
TRAC[0000] End RPC call                                 
INFO[0000] Worker register itself finish                
INFO[0000] Worker load plugin finish                    
TRAC[0000] With Time out                                
TRAC[0000] Start RPC call                               
INFO[0001] [Master] Worker register success             
TRAC[0000] End RPC call                                 
INFO[0000] Worker register itself finish                
INFO[0000] Worker gRPC server start                     
INFO[0000] Worker load plugin finish                    
TRAC[0000] With Time out                                
TRAC[0000] Start RPC call                               
TRAC[0001] [Master] Enough workers!                     
INFO[0001] [Master] Worker register success             
TRAC[0001] [Master] Start distribute workload           
TRAC[0001] Add map input file                           
TRAC[0001] Add map input file                           
TRAC[0001] Add map input file                           
TRAC[0001] Add map input file                           
TRAC[0000] End RPC call                                 
INFO[0000] Worker register itself finish                
TRAC[0001] Add map input file                           
TRAC[0001] Add map input file                           
TRAC[0001] Add map input file                           
TRAC[0001] Add map input file                           
TRAC[0001] [Master] End distribute workload             
TRAC[0001] [Master] Start Map task                      
INFO[0001] [Master] Finding available workers to execute 8/8 
INFO[0001] [Worker] Start Map                           
INFO[0000] [Worker] Start Map                           
TRAC[0000] [Worker] Start Mapping                       
TRAC[0000] [Worker] Finish Mapping                      
TRAC[0001] [Worker] Start Mapping                       
TRAC[0001] [Worker] Finish Mapping                      
TRAC[0001] [Worker] Start partition intermediate kv     
INFO[0000] [Worker] Start Map                           
TRAC[0000] [Worker] Start Mapping                       
TRAC[0000] [Worker] Start partition intermediate kv     
TRAC[0000] [Worker] End partition intermediate kv       
TRAC[0001] [Worker] End partition intermediate kv       
TRAC[0000] [Worker] Write intermediate kv to file       
TRAC[0000] [Worker] Finish Mapping                      
INFO[0000] [Worker] Start Map                           
TRAC[0000] [Worker] Start Mapping                       
TRAC[0000] [Worker] End Write intermediate kv to file   
TRAC[0000] [Worker] Start partition intermediate kv     
TRAC[0000] [Worker] End partition intermediate kv       
TRAC[0000] [Worker] Tell Master the intermediate info   
TRAC[0000] [Worker] Finish Mapping                      
TRAC[0000] [Worker] Write intermediate kv to file       
TRAC[0000] [Worker] Start partition intermediate kv     
TRAC[0000] [Worker] End partition intermediate kv       
TRAC[0000] [Worker] Write intermediate kv to file       
INFO[0000] [Worker] Start Map                           
TRAC[0000] [Worker] Start Mapping                       
TRAC[0000] [Worker] Finish Mapping                      
TRAC[0000] [Worker] Start partition intermediate kv     
TRAC[0001] [Worker] Write intermediate kv to file       
TRAC[0000] [Worker] End partition intermediate kv       
TRAC[0000] [Worker] Write intermediate kv to file       
TRAC[0000] [Worker] End Write intermediate kv to file   
TRAC[0000] [Worker] Tell Master the intermediate info   
TRAC[0000] [Worker] End Write intermediate kv to file   
TRAC[0000] [Worker] Tell Master the intermediate info   
TRAC[0000] [Worker] End Write intermediate kv to file   
TRAC[0000] [Worker] Tell Master the intermediate info   
TRAC[0001] [Worker] End Write intermediate kv to file   
TRAC[0001] [Worker] Tell Master the intermediate info   
INFO[0001] [Master] dfbe266c-095f-43c0-ba22-f03c1a3192dc update IMD info success 
TRAC[0000] [Worker] Finish Tell Master the intermediate info 
INFO[0000] [Worker] Finish Map Task                     
INFO[0001] [Master] 52800c38-289e-40ac-98c8-e95b01ba7f8e update IMD info success 
INFO[0001] [Master] 4744d331-a214-4015-abbb-f32e2239dd5d update IMD info success 
INFO[0001] [Master] 84a20685-69ef-4b76-b41d-046ab8fa17ea update IMD info success 
INFO[0001] [Master] 8be49474-e777-48ee-a1cb-4ca5f5b88c30 update IMD info success 
TRAC[0000] [Worker] Finish Tell Master the intermediate info 
INFO[0000] [Worker] Finish Map Task                     
INFO[0000] [Worker] Start Map                           
TRAC[0000] [Worker] Start Mapping                       
TRAC[0000] [Worker] Finish Mapping                      
TRAC[0000] [Worker] Start partition intermediate kv     
TRAC[0000] [Worker] End partition intermediate kv       
TRAC[0000] [Worker] Write intermediate kv to file       
TRAC[0000] [Worker] Finish Tell Master the intermediate info 
INFO[0000] [Worker] Finish Map Task                     
INFO[0000] [Worker] Start Map                           
TRAC[0000] [Worker] Start Mapping                       
TRAC[0000] [Worker] Finish Mapping                      
TRAC[0000] [Worker] Start partition intermediate kv     
TRAC[0001] [Worker] Finish Tell Master the intermediate info 
INFO[0001] [Worker] Finish Map Task                     
TRAC[0000] [Worker] End partition intermediate kv       
TRAC[0000] [Worker] Write intermediate kv to file       
TRAC[0000] [Worker] End Write intermediate kv to file   
TRAC[0000] [Worker] Tell Master the intermediate info   
INFO[0000] [Worker] Start Map                           
TRAC[0000] [Worker] Start Mapping                       
TRAC[0000] [Worker] End Write intermediate kv to file   
INFO[0001] [Master] 09a01a3b-2239-4c63-9428-17bad0ea82f9 update IMD info success 
TRAC[0000] [Worker] Finish Mapping                      
TRAC[0000] [Worker] Start partition intermediate kv     
TRAC[0000] [Worker] End partition intermediate kv       
TRAC[0000] [Worker] Write intermediate kv to file       
TRAC[0000] [Worker] End Write intermediate kv to file   
TRAC[0000] [Worker] Tell Master the intermediate info   
TRAC[0000] [Worker] Tell Master the intermediate info   
TRAC[0000] [Worker] Finish Tell Master the intermediate info 
INFO[0000] [Worker] Finish Map Task                     
INFO[0001] [Master] 07e24826-d4d9-44dd-98ed-0e704c6e2a85 update IMD info success 
TRAC[0000] [Worker] Finish Tell Master the intermediate info 
INFO[0001] [Master] 40d0408b-cacd-41c1-b79d-fec6a7de044f update IMD info success 
INFO[0000] [Worker] Finish Map Task                     
TRAC[0000] [Worker] Finish Tell Master the intermediate info 
INFO[0000] [Worker] Finish Map Task                     
TRAC[0000] [Worker] Finish Tell Master the intermediate info 
INFO[0000] [Worker] Finish Map Task                     
TRAC[0001] [Master] End Map task                        
TRAC[0001] [Master] Start Reduce task                   
INFO[0001] [Master] Finding available workers to execute 4/8 
INFO[0001] [Worker] Start Reduce                        
TRAC[0001] [Worker] Get intermediate file               
INFO[0000] [Worker] Start Reduce                        
TRAC[0000] [Worker] Get intermediate file               
INFO[0000] [Worker] Start Reduce                        
TRAC[0000] [Worker] Get intermediate file               
INFO[0000] [Worker] Start Reduce                        
TRAC[0000] [Worker] Get intermediate file               
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0001] [Worker] RPC Get intermediate file           
INFO[0001] [Worker] RPC Get intermediate file           
INFO[0001] [Worker] RPC Get intermediate file           
INFO[0001] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
TRAC[0001] [Worker] Sort intermediate KV                
TRAC[0001] [Worker] Start Reducing                      
TRAC[0001] [Worker] End Reducing                        
INFO[0001] [Worker] End Reduce                          
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
TRAC[0000] [Worker] Sort intermediate KV                
TRAC[0000] [Worker] Start Reducing                      
TRAC[0000] [Worker] End Reducing                        
INFO[0000] [Worker] End Reduce                          
INFO[0000] [Worker] RPC Get intermediate file           
TRAC[0000] [Worker] Sort intermediate KV                
TRAC[0000] [Worker] Start Reducing                      
TRAC[0000] [Worker] End Reducing                        
INFO[0000] [Worker] End Reduce                          
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
INFO[0000] [Worker] RPC Get intermediate file           
TRAC[0000] [Worker] Sort intermediate KV                
TRAC[0000] [Worker] Start Reducing                      
TRAC[0000] [Worker] End Reducing                        
INFO[0000] [Worker] End Reduce                          
TRAC[0001] [Master] End Reduce task                     
TRAC[0001] [Master] End Workers Start                   
INFO[0001] [Worker] End worker                          
INFO[0000] [Worker] End worker                          
INFO[0000] [Worker] End worker                          
INFO[0000] [Worker] End worker                          
INFO[0000] [Worker] End worker                          
INFO[0000] [Worker] End worker                          
INFO[0000] [Worker] End worker                          
INFO[0000] [Worker] End worker                          
TRAC[0001] [Master] End Workers done                    
go run -race cmd/master.go -i 'mr-out-*' -p 'cmd/merge.so' -r 1 -w 4 &
sleep 1
INFO[0000] [Master] Master gRPC server start            
TRAC[0000] [Master] Wait for enough workers             
go run -race cmd/worker.go -i 'mr-out-*' -p 'cmd/merge.so' -r 1 -w 1 &
go run -race cmd/worker.go -i 'mr-out-*' -p 'cmd/merge.so' -r 1 -w 2 &
go run -race cmd/worker.go -i 'mr-out-*' -p 'cmd/merge.so' -r 1 -w 3 &
go run -race cmd/worker.go -i 'mr-out-*' -p 'cmd/merge.so' -r 1 -w 4
INFO[0000] Worker gRPC server start                     
INFO[0000] Worker load plugin finish                    
TRAC[0000] With Time out                                
TRAC[0000] Start RPC call                               
INFO[0001] [Master] Worker register success             
TRAC[0000] End RPC call                                 
INFO[0000] Worker register itself finish                
INFO[0000] Worker gRPC server start                     
INFO[0000] Worker load plugin finish                    
TRAC[0000] With Time out                                
TRAC[0000] Start RPC call                               
INFO[0002] [Master] Worker register success             
TRAC[0000] End RPC call                                 
INFO[0000] Worker register itself finish                
INFO[0000] Worker gRPC server start                     
INFO[0000] Worker load plugin finish                    
TRAC[0000] With Time out                                
TRAC[0000] Start RPC call                               
INFO[0002] [Master] Worker register success             
TRAC[0000] End RPC call                                 
INFO[0000] Worker register itself finish                
INFO[0000] Worker gRPC server start                     
INFO[0000] Worker load plugin finish                    
TRAC[0000] With Time out                                
TRAC[0000] Start RPC call                               
INFO[0002] [Master] Worker register success             
TRAC[0002] [Master] Enough workers!                     
TRAC[0002] [Master] Start distribute workload           
TRAC[0002] Add map input file                           
TRAC[0002] Add map input file                           
TRAC[0002] Add map input file                           
TRAC[0002] Add map input file                           
TRAC[0002] Add map input file                           
TRAC[0002] Add map input file                           
TRAC[0002] Add map input file                           
TRAC[0002] Add map input file                           
TRAC[0002] Add map input file                           
TRAC[0002] Add map input file                           
TRAC[0002] Add map input file                           
TRAC[0002] Add map input file                           
TRAC[0002] Add map input file                           
TRAC[0002] Add map input file                           
TRAC[0002] Add map input file                           
TRAC[0002] Add map input file                           
TRAC[0002] [Master] End distribute workload             
TRAC[0002] [Master] Start Map task                      
INFO[0002] [Master] Finding available workers to execute 4/4 
TRAC[0000] End RPC call                                 
INFO[0000] Worker register itself finish                
INFO[0000] [Worker] Start Map                           
INFO[0000] [Worker] Start Map                           
TRAC[0000] [Worker] Start Mapping                       
TRAC[0000] [Worker] Start Mapping                       
INFO[0000] [Worker] Start Map                           
TRAC[0000] [Worker] Finish Mapping                      
TRAC[0000] [Worker] Finish Mapping                      
TRAC[0000] [Worker] Start partition intermediate kv     
TRAC[0000] [Worker] Start partition intermediate kv     
TRAC[0000] [Worker] Start Mapping                       
INFO[0000] [Worker] Start Map                           
TRAC[0000] [Worker] Start Mapping                       
TRAC[0000] [Worker] Finish Mapping                      
TRAC[0000] [Worker] Start partition intermediate kv     
TRAC[0000] [Worker] End partition intermediate kv       
TRAC[0000] [Worker] Write intermediate kv to file       
TRAC[0000] [Worker] End Write intermediate kv to file   
TRAC[0000] [Worker] Tell Master the intermediate info   
panic: runtime error: index out of range [1] with length 1

goroutine 27 [running]:
plugin/unnamed-8794b001124bf89ba2b7e1417ecabf0b0c406174.Map({0x0?, 0x0?}, {0x0, 0x0}, {0xc0001808a0})
        /home/user/Desktop/MapReduce/mrapps/merge.go:20 +0x1b1
github.com/BWbwchen/MapReduce/worker.(*Worker).Map.func1(0xc0001948c0, {0x0, 0x0})
        /home/user/Desktop/MapReduce/worker/worker.go:61 +0xc8
created by github.com/BWbwchen/MapReduce/worker.(*Worker).Map in goroutine 35
        /home/user/Desktop/MapReduce/worker/worker.gopanic: runtime error: index out of range [1] with length 1

goroutine 10 [running]:
plugin/unnamed-8794b001124bf89ba2b7e1417ecabf0b0c406174.Map({0x0?, 0x0?}, {0x0, 0x0}, {0xc00028e480})
        /home/user/Desktop/MapReduce/mrapps/merge.go:20 +0x1b1
github.com/BWbwchen/MapReduce/worker.(*Worker).Map.func1(0xc000282690, {0x0, :600x0})
        /home/user/Desktop/MapReduce/worker/worker.go:61 +0xc8
 +created by github.com/BWbwchen/MapReduce/worker.(*Worker).Map in goroutine 6
0x185   /home/user/Desktop/MapReduce/worker/worker.go:60 +0x185

INFO[0002] [Master] 9503430e-b944-4d68-8531-1fd401a48b07 update IMD info success 
TRAC[0000] [Worker] Finish Tell Master the intermediate info 
INFO[0000] [Worker] Finish Map Task                     
panic: runtime error: index out of range [1] with length 1

goroutine 28 [running]:
plugin/unnamed-8794b001124bf89ba2b7e1417ecabf0b0c406174.Map({0x0?, 0x0?}, {0x0, 0x0}, {0xc000094d20})
        /home/user/Desktop/MapReduce/mrapps/merge.go:20 +0x1b1
github.com/BWbwchen/MapReduce/worker.(*Worker).Map.func1(0xc000096eb0, {0x0, 0x0})
        /home/user/Desktop/MapReduce/worker/worker.go:61 +0xc8
created by github.com/BWbwchen/MapReduce/worker.(*Worker).Map in goroutine 25
        /home/user/Desktop/MapReduce/worker/worker.go:60 +0x185
exit status 2
exit status 2
WARN[0002] [Master]: error reading from server: EOF     
WARN[0002] [Master]: error reading from server: EOF     
INFO[0002] [Master] Finding available workers to execute 1/4 
INFO[0002] [Master] Re-execute Map Task from 9c0bc0cd-5305-49f7-87ec-c60dd346ab94 To 9503430e-b944-4d68-8531-1fd401a48b07 
INFO[0002] [Master] Finding available workers to execute 1/4 
INFO[0000] [Worker] Start Map                           
TRAC[0000] [Worker] Start Mapping                       
TRAC[0000] [Worker] Finish Mapping                      
TRAC[0000] [Worker] Start partition intermediate kv     
make: *** [Makefile:44: test_multi_node] Error 1
panic: runtime error: index out of range [1] with length 1                                                                                                                                

goroutine 70 [running]:
plugin/unnamed-8794b001124bf89ba2b7e1417ecabf0b0c406174.Map({0x1?, 0x1?}
, {0x0, 0x0}, {0xc000180420})
        /home/user/Desktop/MapReduce/mrapps/merge.go:20 +0x1b1
github.com/BWbwchen/MapReduce/worker.(*Worker).Map.func1(0xc000194aa0, {0x0, 0x0})
        /home/user/Desktop/MapReduce/worker/worker.go:61 +0xc8
created by github.com/BWbwchen/MapReduce/worker.(*Worker).Map in goroutine 66
        /home/user/Desktop/MapReduce/worker/worker.go:60 +0x185
exit status 2
WARN[0002] [Master]: error reading from server: EOF     
exit status 2
==================
WARNING: DATA RACE
Write at 0x00c000172170 by goroutine 33:
  github.com/BWbwchen/MapReduce/master.(*WorkerInfo).SetState()
      /home/user/Desktop/MapReduce/master/workerInfo.go:36 +0x4d
  github.com/BWbwchen/MapReduce/master.(*Master).distributeMapTask.func1()
      /home/user/Desktop/MapReduce/master/master.go:220 +0x584
  github.com/BWbwchen/MapReduce/master.(*Master).distributeMapTask.func4()
      /home/user/Desktop/MapReduce/master/master.go:228 +0xa6

Previous read at 0x00c000172170 by goroutine 7:
  github.com/BWbwchen/MapReduce/master.(*WorkerInfo).Broken()
      /home/user/Desktop/MapReduce/master/workerInfo.go:41 +0x49e
  github.com/BWbwchen/MapReduce/master.(*Master).availableWorkers()
      /home/user/Desktop/MapReduce/master/master.go:144 +0x450
  github.com/BWbwchen/MapReduce/master.(*Master).distributeMapTask()
      /home/user/Desktop/MapReduce/master/master.go:233 +0x60e
  github.com/BWbwchen/MapReduce/master.StartMaster()
      /home/user/Desktop/MapReduce/master/mainInterface.go:47 +0x330
  github.com/BWbwchen/MapReduce.startMaster.func1()
      /home/user/Desktop/MapReduce/util.go:102 +0x89

Goroutine 33 (running) created at:
  github.com/BWbwchen/MapReduce/master.(*Master).distributeMapTask()
      /home/user/Desktop/MapReduce/master/master.go:215 +0x512
  github.com/BWbwchen/MapReduce/master.StartMaster()
      /home/user/Desktop/MapReduce/master/mainInterface.go:47 +0x330
  github.com/BWbwchen/MapReduce.startMaster.func1()
      /home/user/Desktop/MapReduce/util.go:102 +0x89

Goroutine 7 (running) created at:
  github.com/BWbwchen/MapReduce.startMaster()
      /home/user/Desktop/MapReduce/util.go:101 +0x2a4
  github.com/BWbwchen/MapReduce.StartMaster()
      /home/user/Desktop/MapReduce/masterNode.go:4 +0x27
  main.main()
      /home/user/Desktop/MapReduce/cmd/master.go:8 +0x18
==================
WARN[0002] [Master]: error reading from server: EOF     
==================
WARNING: DATA RACE
Write at 0x00c0001720e0 by goroutine 58:
  github.com/BWbwchen/MapReduce/master.(*WorkerInfo).SetState()
      /home/user/Desktop/MapReduce/master/workerInfo.go:36 +0x4d
  github.com/BWbwchen/MapReduce/master.(*Master).distributeMapTask.func2()
      /home/user/Desktop/MapReduce/master/master.go:248 +0x5ac
  github.com/BWbwchen/MapReduce/master.(*Master).distributeMapTask.func5()
      /home/user/Desktop/MapReduce/master/master.go:255 +0x91

Previous read at 0x00c0001720e0 by goroutine 7:
  github.com/BWbwchen/MapReduce/master.(*WorkerInfo).Broken()
      /home/user/Desktop/MapReduce/master/workerInfo.go:41 +0x49e
  github.com/BWbwchen/MapReduce/master.(*Master).availableWorkers()
      /home/user/Desktop/MapReduce/master/master.go:144 +0x450
  github.com/BWbwchen/MapReduce/master.(*Master).distributeMapTask()
      /home/user/Desktop/MapReduce/master/master.go:233 +0x60e
  github.com/BWbwchen/MapReduce/master.StartMaster()
      /home/user/Desktop/MapReduce/master/mainInterface.go:47 +0x330
  github.com/BWbwchen/MapReduce.startMaster.func1()
      /home/user/Desktop/MapReduce/util.go:102 +0x89

Goroutine 58 (running) created at:
  github.com/BWbwchen/MapReduce/master.(*Master).distributeMapTask()
      /home/user/Desktop/MapReduce/master/master.go:242 +0x5b9
  github.com/BWbwchen/MapReduce/master.StartMaster()
      /home/user/Desktop/MapReduce/master/mainInterface.go:47 +0x330
  github.com/BWbwchen/MapReduce.startMaster.func1()
      /home/user/Desktop/MapReduce/util.go:102 +0x89

Goroutine 7 (running) created at:
  github.com/BWbwchen/MapReduce.startMaster()
      /home/user/Desktop/MapReduce/util.go:101 +0x2a4
  github.com/BWbwchen/MapReduce.StartMaster()
      /home/user/Desktop/MapReduce/masterNode.go:4 +0x27
  main.main()
      /home/user/Desktop/MapReduce/cmd/master.go:8 +0x18
==================
PANI[0002] No enough worker!0 1                         
panic: (*logrus.Entry) 0xc0000ac9a0

goroutine 6 [running]:
github.com/sirupsen/logrus.(*Entry).log(0xc0000ac930, 0x0, {0xc00009e120, 0x14})
        /home/user/Desktop/MapReduce/vendor/github.com/sirupsen/logrus/entry.go:259 +0x97f
github.com/sirupsen/logrus.(*Entry).Log(0xc0000ac930, 0x0, {0xc00019bb80, 0x3, 0x3})
        /home/user/Desktop/MapReduce/vendor/github.com/sirupsen/logrus/entry.go:293 +0x8c
github.com/sirupsen/logrus.(*Logger).Log(0x1305240, 0x0, {0xc00019bb80, 0x3, 0x3})
        /home/user/Desktop/MapReduce/vendor/github.com/sirupsen/logrus/logger.go:198 +0x7a
github.com/sirupsen/logrus.(*Logger).Panic(...)
        /home/user/Desktop/MapReduce/vendor/github.com/sirupsen/logrus/logger.go:247
github.com/sirupsen/logrus.Panic(...)
        /home/user/Desktop/MapReduce/vendor/github.com/sirupsen/logrus/exported.go:129
github.com/BWbwchen/MapReduce/master.(*Master).availableWorkers(0xc000024870, 0x1)
        /home/user/Desktop/MapReduce/master/master.go:149 +0x269
github.com/BWbwchen/MapReduce/master.(*Master).distributeMapTask(0xc000024870)
        /home/user/Desktop/MapReduce/master/master.go:233 +0x60f
github.com/BWbwchen/MapReduce/master.StartMaster({0xc000078c40, 0x4, 0x4}, 0x0?, 0x0?, {0xc00001569a, 0x6})
        /home/user/Desktop/MapReduce/master/mainInterface.go:47 +0x331
github.com/BWbwchen/MapReduce.startMaster.func1()
        /home/user/Desktop/MapReduce/util.go:102 +0x8a
created by github.com/BWbwchen/MapReduce.startMaster in goroutine 1
        /home/user/Desktop/MapReduce/util.go:101 +0x2a5
exit status 2

Originally posted by @kiarash8112 in #1 (comment)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.