sup3rgiu / polidown Goto Github PK

Hi! Great tool, thanks!

I have a problem.
When I launch the tool at home with my wifi connection, the result is always one of these 2 message:
message 1

message 2

I tried using a different wifi connection and also tethering from smartphone and it worked fine. I don't really know what the problem could be (I have no issue navigating the web, except seldom slow connection).

When I try to download a video from Microsoft Streams it gives me that error. I'm on Mac OS Catalina. What should I do?

Hi, first thanks for your nice tool =).

until the download part everything runs smooth.
But when it starts to use ffmpeg i get this error:

Any idear how i can fix it?

I run using the following command (the --username argument is omitted obv)
node polidown.js --videoUrls "https://web.microsoftstream.com/video/5b4d4a6c-a0a8-48d0-8c20-0df9367dc0e9"
but I get this error:

[crypto @ 00000251742672c0] Unable to open resource: aaudio_segments/Fragments(aac_eng_1_60_1_1=2767360000,format=m3u8-aapl)
[hls @ 000002517425de40] Failed to open segment 46 of playlist 0
[hls @ 000002517425de40] Error when loading first segment 'aaudio_segments/Fragments(aac_eng_1_60_1_1=0,format=m3u8-aapl)'
audio_tmp.m3u8: Invalid data found when processing input
(node:23128) UnhandledPromiseRejectionWarning: Error: Command failed: ffmpeg -protocol_whitelist file,http,https,tcp,tls,crypto,data -allowed_extensions ALL -i audio_tmp.m3u8 -protocol_whitelist file,http,https,tcp,tls,crypto,data -allowed_extensions ALL -i video_tmp.m3u8 -async 1 -c copy -bsf:a aac_adtstoasc -n "....\videos\Lesson 26_04_2020 - FRIGO CARLO ALBINO 083148-RIABILITAZIONE MOTORIA [M.S.] (712476).mp4"
at checkExecSyncError (child_process.js:630:11)
at execSync (child_process.js:666:15)
at downloadVideo (C:\Projects\PoliDown\polidown.js:426:22)
at processTicksAndRejections (internal/process/task_queues.js:97:5)
(node:23128) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag --unhandled-rejections=strict (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:23128) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

After #5 (commit e017cd1) I had some problems on Linux; as reported in this issue (Foundry376/Mailspring#681) and other, node-keytar works only with libsecret, and libsecret only works with gnome-keyring. Since I don't use gnome, I think it would be better to have an option not to use it, much like there is a way to not use it if X11 is not installed. I have absolutely no problem with version before commit e017cd1.

I suggest maybe having a command line option to explicitly say not to use the keyring.

Firstly Thank you so much for your effort and generosity.
Can you add support for students outside of Politecnico di Milano?
Thanks!

I maybe did something wrong, or missed a package somewhere but it doesn't work.

It suggested me to npm install but that already went well.

I'm on Windows and I did install node.js, aria2c and ffmpeg and added them to the $PATH variable.

Hey everyone!

I wanted to turn this into a chrome extension but i quicky realized that currently i have more skills at biasing transistors than at writing lines of code (especially high level).

So i'm leaving this idea here for anyone with more javascript programming skills than me, it would be handy to be able to download lessons with just one button.

I recently came across this plugin for Vimeo, that gave me the idea. This specific plugins modifies the page adding the button for download over the video, but i feel that's not necessary, a click on the normal extension icon would be more than enough.

P.S. if possible leave this issue open so that more people will be able to see it and maybe take charge of the development

Sometimes may appear "timeouterror" after "Filling in servizi online". It is probably caused by the PoliMi warning "The password will expire in.." and can be easily solved changing the password.

more than restarting the prompt and setting everything, is it possible for me to do something more?

I'm on WSL Debian kernel 4.4.0-18362-Microsoft

This is what i got after the first run of npm install:

username@LAPTOP~/p/PoliDown> npm install

> [email protected] install /home/username/polidown/PoliDown/node_modules/keytar
> prebuild-install || node-gyp rebuild

prebuild-install WARN install libsecret-1.so.0: cannot open shared object file: No such file or directory
/bin/sh: 1: pkg-config: not found
gyp: Call to 'pkg-config --libs-only-l libsecret-1' returned exit status 127 while in binding.gyp. while trying to load binding.gyp
gyp ERR! configure error
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onCpExit (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:351:16)
gyp ERR! stack     at ChildProcess.emit (events.js:198:13)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)
gyp ERR! System Linux 4.4.0-18362-Microsoft
gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /home/username/polidown/PoliDown/node_modules/keytar
gyp ERR! node -v v10.22.1
gyp ERR! node-gyp -v v5.1.0
gyp ERR! not ok
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: `prebuild-install || node-gyp rebuild`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/username/.npm/_logs/2020-09-22T10_43_37_118Z-debug.log

I solved via sudo apt install libsecret-1-0

Hi, after all the parts were downloaded, I get the message in the image (I think is aria2 related). After the message the program was stuck so, after about 5 minutes, I digited ctrl-c.
input:

output:

ffmpeg.zeranoe.com closed since Sep 18, 2020.

 at checkExecSyncError (child_process.js:630:11)
    at execSync (child_process.js:666:15)
    at downloadVideo (D:\stream-test\PoliDown-master\polidown.js:338:22)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
(node:5588) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:5588) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

I really appreciate your job!
When I download multiple videos, it randomly happens this error.
it's not related to a particular video, because rerunning the program on the same video solve the problem. I normally put 5 videos link in the same command.
thank you

Hello,

It give me this error i don't understand how to solve it. Can anyone please help me ?

After selecting the video quality, the download starts;
After downloading all the files I get this message and in my folder videos I have only a lot of small file, but not an mp4 video

installing on Fedora is necessary to run npm as root flagging --unsafe-perm, which is far from ideal.

hi, the script give me this error when i try to download a video. i think it have something to do with the parsing process of the downloaded pieces.

i edited the const argv = yargs.options (lines 13-18), adding my codice persona, password and output after default:, in this way the only arg is the --videoUrls. i don't think this have anything to do w/ the problem above.
with the older version, everything is fine.

Hello! Can you do a tutorial for Mac? Thank you

When you try to download multiple videos from the same course it save only the first one. To solve this you could add a simple variable that increases in the for at the end of the name of the output file.

Is it possible to download subtitles of the videos ? Sometimes they are useful when the professors are speaking in English. I would suggest that in any future update.

Thanks a lot for your work guys👏🏼💪🏼

HI everyone,
i'm using PoliDown and it's amazing! However i noticed that it ask for every video the resolution at witch it has to be download. It is possible to implement something that automatically select the best available resolution? In that way it will be possible to add a long list of video without being in front of pc and let downloads run during night.

Thanks a lot for that tool
Bye

Dopo che compare "Filling in Servizi Online login form...", a distanza di qualche minuto compare ciò:

(node:14232) UnhandledPromiseRejectionWarning: TimeoutError: waiting for target failed: timeout 90000ms exceeded
at Function.waitWithTimeout (C:\Projects\PoliDown\node_modules\puppeteer\lib\helper.js:228:26)
at Browser.waitForTarget (C:\Projects\PoliDown\node_modules\puppeteer\lib\Browser.js:214:27)
at Browser. (C:\Projects\PoliDown\node_modules\puppeteer\lib\helper.js:112:23)
at downloadVideo (C:\Projects\PoliDown\polidown.js:170:18)
(node:14232) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag --unhandled-rejections=strict (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:14232) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Qualche settimana fa, quando ho scaricato i video per la prima volta, funzionava tutto benissimo. Adesso, a distanza di tempo e dopo aver cambiato browser predefinito da Opera a Firefox (non so se possa essere utile come informazione), non funziona più ed esce quel testo che incollato sopra. Ho provato più volte e ogni tanto compare anche "unable to read cookies", però in questo caso riesce a loggarsi nei servizi online. Grazie in anticipo

As title said, the script does not work on Windows when headless: true
Switching it to: headless: false makes it work.

In particular the script stuck at networkidle2:
await page.goto('https://web.microsoftstream.com/', { waitUntil: 'networkidle2' });
After 30s I got timeout exception.
I tried changing it into:
await page.goto('https://web.microsoftstream.com/', { waitUntil: 'domcontentloaded' });
but the script goes on a bit and then it stuck after on this line:
await page.waitForSelector('input[id="login"]'
with another timeout exception.

It seems that the browser have a different behavior when is in headless mode:
puppeteer/puppeteer#665 (comment)
The comment above suggest to change user agent when you are in headless mode.
Maybe an additional window appears, or Polimi prevents bots from login (???)

I am experiencing this kind of error for days, for single videos and multiple ones.
I cannot download anything anymore

Hi sup3rgiu, nice tool!!

I have a problem when downloading multiple videos, but not when I'm trying only one video. Also the problem is not consistent.
I think the problem is related to some kind of traffic policy that ms streams enforces.
I suggest you to change the behaviour from a single try to multiple tries with a delay between them.

node polidown.js -u [...] -v "https://web.microsoftstream.com/video/80f24f48-cf62-455a-a936-54fb5f497686" "https://web.microsoftstream.com/video/c58c6b67-3224-4397-bca8-b1d7e6b51f3e" "https://web.microsoftstream.com/video/42e2e413-00e7-47b1-9748-b299677b52ea" "https://web.microsoftstream.com/video/d78a3cfa-2e04-410e-a47e-cff748266689" "https://web.microsoftstream.com/video/e4cbec03-1b63-4f4f-bfcd-4239b8072728" "https://web.microsoftstream.com/video/7b0be1ca-e1ac-4ca9-ba69-9357c4268025" "https://web.microsoftstream.com/video/d92aed10-cdda-4dcc-a10f-012ed2bdf4fa" "https://web.microsoftstream.com/video/acaa99c0-8394-4828-b294-30ae46eb74d6"

Start downloading video: https://web.microsoftstream.com/video/d78a3cfa-2e04-410e-a47e-cff748266689
(node:13645) UnhandledPromiseRejectionWarning: TimeoutError: Navigation Timeout Exceeded: 30000ms exceeded
    at /Users/lorenzo/Downloads/PoliDown-master/node_modules/puppeteer/lib/LifecycleWatcher.js:142:21
  -- ASYNC --
    at Frame.<anonymous> (/Users/lorenzo/Downloads/PoliDown-master/node_modules/puppeteer/lib/helper.js:111:15)
    at Page.goto (/Users/lorenzo/Downloads/PoliDown-master/node_modules/puppeteer/lib/Page.js:674:49)
    at Page.<anonymous> (/Users/lorenzo/Downloads/PoliDown-master/node_modules/puppeteer/lib/helper.js:112:23)
    at downloadVideo (/Users/lorenzo/Downloads/PoliDown-master/polidown.js:146:19)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
(node:13645) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:13645) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

TXT files containing more URLs can't correctly be parsed on OSX (and maybe Linux).

The reason is linked to this function:
function readFileToArray(path) {
path = path.substr(1,path.length-2);
return fs.readFileSync(path).toString('utf-8').split('\r\n');
}
Files created on Linux and OSX don't use \r\n but just \n. On Unix, it is necessary to remove only \n. I assume Windows version don't have this problem.