Love Go/Rust/C++, distributed systems, blockchain / Web3 tech.
reach me at: [email protected]
concurrent file downloader
License: Apache License 2.0
Love Go/Rust/C++, distributed systems, blockchain / Web3 tech.
reach me at: [email protected]
example, you can expect a config file like .blazer_config in home directory, there you can store defaults like threadcount, save download history option etc
Use Delimiter "|"
blazer -url=https://speed.hetzner.de/100MB.bin t=1
Fetching file meta..
Can't initiate download Head "https://speed.hetzner.de/100MB.bin": EOF
Lot of threads doesn't guarantee optimal concurrency, need to find out the ideal concurrency for a given situation and select that as thread count as default value
blazer -url=https://schooleverywhere-elquds.com/lib/uploadbook/179121479125786First_Children__s_Encyclopedia.pdf -t=200 -out=test.pdf -checksum=ea008f32a265d322bb3956afbd7c39d1b010b28dfc659c7b7ad3ed20bc0de339
Get "https://url/test.pdf": read tcp 192.168.0.113:56940->75.98.168.118:443: read: connection reset by peer
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x646d0a]
goroutine 773 [running]:
github.com/arvpyrna/blazer/network.DownloadSegment(0xc0008eef00, 0x1af, 0xd8d0b8, 0xd9517f)
/home/arv/lab/blazer/network/downloader.go:47 +0x34a
github.com/arvpyrna/blazer/network.ConcurrentDownloader.func1(0xc0005940b0, 0xc0008eef00, 0x1af, 0xd8d0b8, 0xd9517f)
/home/arv/lab/blazer/network/downloader.go:32 +0x75
created by github.com/arvpyrna/blazer/network.ConcurrentDownloader
/home/arv/lab/blazer/network/downloader.go:30 +0x30f
For homebrew users like me it would be very cool if you can publish blazer as a package so that we can do
brew install blazer
and have it installed on our systems. Homebrew is majorly used for package management by mac users. It is also available for linux as well. :-)
-Get "https://logic.pdmi.ras.ru/~gravin/storage/Concrete_Mathematics_2e.pdf": dial tcp 83.149.197.121:443: i/o timeout
Handle improper URL
Get "https://mirror.yandex.ru/fedora/linux/releases/34/Workstation/aarch64/iso/Fedora-Workstation-Live-aarch64-34-1.2.iso": read tcp 192.168.0.113:38734->213.180.204.183:443: read: connection reset by peer
Get "https://sample-videos.com/img/Sample-jpg-image-30mb.jpg": read tcp 192.168.43.62:41606->103.145.51.95:443: read: connection reset by peer
TCP timeout incase of large file
Try to download a large file like 10 or 20 gb and
Get "https://releases.ubuntu.com/20.04.2.0/ubuntu-20.04.2.0-desktop-amd64.iso": EOF
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x64366a]
A segment download can fail due to errors such as network backpressure, it makes sense to retry few times before failing the entire download process.
before merging the file we are not checking if all the segments were downloaded or not. And also moved the segment merging out of the network code for more clarity.
if a segment is not downloaded successfully, push it to a channel, listener of the channel will initiate the download again
CI/CD pipeline seems to be broken, needs to be fixed
gox
Probably, prompt the user with available space and confirm to go ahead with the download or not.
If a segment is not completely downloaded, need to retry that, create a new channel that will run those before merging files, do not merge files if there is not enough segments
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.