lino-levan / astral Goto Github PK

see: https://github.com/lino-levan/astral/actions/runs/6501910122/job/17660034750

Ran into an issue with this on discord.

Suggested in #16

Support for finding elements by XPath?

Suggested in #16

Suggested by #16

The config should be in XDG_CACHE_HOME https://specifications.freedesktop.org/basedir-spec/basedir-spec-0.6.html

I think you can use https://deno.land/x/[email protected]

Hey. I just tried using astral for the first time, but immediately after launching the browser, the script threw an error and crashed.

Code:

import { launch } from "https://deno.land/x/astral/mod.ts";
const browser = await launch();

Error:

Uncaught Error: Failed to spawn '/home/liam/.cache/astral/121.0.6130.0/chrome-linux64/chrome': Exec format error (os error 8)
    at spawnChildInner (ext:runtime/40_process.js:185:17)
    at spawnChild (ext:runtime/40_process.js:206:10)
    at Command.spawn (ext:runtime/40_process.js:474:12)
    at runCommand (https://deno.land/x/[email protected]/src/browser.ts:13:27)
    at launch (https://deno.land/x/[email protected]/src/browser.ts:254:39)
    at eventLoopTick (ext:core/01_core.js:166:7)
    at async <anonymous>:1:38

Environment Information:
Raspberry Pi 4b+ 4GB + 512 GB SSD
Ubuntu Server 23.10
Deno 1.41.1

See #16

This looks to be the future of puppeteer-like APIs.

ref: https://pptr.dev/guides/locators

Scenario

Currently there is a scenario where if the sandbox does not have write permissions it causes an exception when "launching" the browser, as it tries to create a temporary folder for "user data"

Hi there! Thanks for working on this; it's great to see finally a Deno native package for working with a headless browser. Puppeteer has been very frustrating with Deno, and we're at the point now that it's not working at all. Attempting to use npm:[email protected] results type errors due to globals being set by npm Node.js types conflicting with globals set by Deno types. Attempting to use https://deno.land/x/[email protected] results in strange runtime errors now with modern Deno; and it's a huge number of major versions behind the latest Puppeteer version on npm. It seems to be unmaintained.

Consider this issue a question about how Astral can be used to achieve some things I previously had working with Puppeteer (when it was working with earlier Deno versions), and if there is no way, or an inconvenient way, consider it a feature request.

There are 2 main reasons I have been using headless browsers over the years:

1. Screenshotting web pages, or parts of pages. This is probably straightforward with your current Astral API, so I don't have questions about that right now.
2. Unit testing modules of a published library to check they function as intended in a browser. You want to be able to functionally create a browser page, do unit tests with it in an async callback, and then after it automatically does cleanup:
   • Code coverage is collected by the browser, and then combined with V8 code coverage data collected by Node.js/Deno when the same modules/functions were run directly in tests, to be able to have an accurate code coverage report of the library after the tests have run.
   • Any console messages emitted in the browser page are forwarded to the Node.js/Deno process and logged via console.group under a message indicating the messages came from the headless browser, using what API (warn, error, log, etc.).
   • Page errors are collected and after the tests callback completes, assert the array of page errors is empty. You don't want to end up in a situation where the unit tests are passing but there are uncaught page errors going on undetected.

How can Astral be used to achieve use-case number 2? Here is how it was achieved using Puppeteer:

https://github.com/jaydenseric/ruck/blob/aa068afa7f0630c4e3e8617b2936210c14815964/test/testPuppeteerPage.mjs

It was very difficult for me to work out at first, but that's just because the Puppeteer API doesn't provide a ergonomic ways to do it. Surely it would not be complicated to provide a specific API for this in Astral; an easy way to create a browser page for use in unit testing.

Is it expected to add such apis, I use them frequently like this
ElementHandle.parentElement.children[1] for example

Suggested in #16

Hi there. This seems like a great project.

I'm trying to integrate it into a test suite, but deno test complains that there are operations leaking....

When I run the following code: https://gist.github.com/cowboyd/c5735b5f80caf2e745dd93ee78cc48b7

import { describe, it } from "https://deno.land/[email protected]/testing/bdd.ts"
import { launch } from "https://deno.land/x/[email protected]/mod.ts";

describe("browser", () => {
  it("launches, sleeps, and closes", async () => {
    let browser = await launch();
    await new Promise((resolve) => setTimeout(resolve, 2000));
    browser.close();
  });
});

I consistently see the following:

$ deno test -A browser.test.ts
Check browser.test.ts
running 1 test from browser.test.ts
browser ...
  launches, sleeps, and closes ... FAILED (2s)
browser ... FAILED (due to 1 failed step) (2s)

 ERRORS

browser ... launches, sleeps, and closes => https://deno.land/[email protected]/testing/_test_suite.ts:323:15
error: Leaking async ops:
  - 1 async operation to receive the next message on a WebSocket was started in this test, but never completed. This is often caused by not closing a `WebSocket` or `WebSocketStream`.
  - 1 async operation to op_spawn_wait was started in this test, but never completed.
To get more details where ops were leaked, run again with --trace-ops flag.

Should browser.close() be an async operation that waits until everything has been cleaned up?

I'm not an artist, I would happily accept a better icon for astral. The one right now is kind of eh

Would it be technically possible to catch any navigation (I assume there's an event for it) and validate it against the Deno.permissions API ? Or is this infeasible ?

I guess it could be tricky to handle the navigation from scripts...

await page.evaluate(() => location.href = "https://example.com")

Also, maybe this is more browser-realated rather than directly with astral, but is it possible to disable some protocol schemes ? (like file:// and data:// for example) ?

It would be great if Page had a url() method to grab the current location of the page.

Puppeteer already has this.

See #16

Hi !
I'm looking to possibly use this as a replacement of npm:puppeteer as the later isn't designed for deno which seems to cause some troubles on windows (SIGHUP unsupported, instances not properly closing) and during unit testing (leaking ops)

So I'm considering switching to astral, as the others puppeteer-like libs are either outdated or have some caveats from what I tested.

I have a few questions and possibly features requests (if this is in the project scope) though:

• How reliable is it for unit testing ? Currently I need to pass { sanitizeResources: false, sanitizeOps: false } with puppeteer but the aim is to be able to have proper testing without any leaking ops (including windows).
• Would it be possible to avoid the use of Deno.command for the browser installation ? On windows requiring the access of Powershell which creates a big attack vector since you need to give a shell access. I haven't experimented it but it seems that it's possible to use DecompressionStream to unzip the archive natively (ref: https://medium.com/deno-the-complete-reference/zip-and-unzip-files-in-deno-ee282da7369f)
• Similarly, would it be possible to be able to pass the cache path in getBinary() so it's possible to use the same cache as the app using it and have more controls about deno permissions ?
• There are some methods that doesn't seems to have any equivalent in astral currently:
  • Page.setContent()
  • Page.setViewport()
  • Puppeteer.connect()
    • This one is needed to be able to use it through remote services like https://www.browserless.io/ so it's possible to use astral on services like deno deploy or vercel, since you may not always be able to run the browser on the same host as the script
• How do you achieve page.on("console", (message: { text: () => string }) => console.debug("puppeteer: ${message.text()}")) and page.on("pageerror", (error: { message: string }) => console.warn("puppeteer: ${error.message}")) on astral to log console message from the browser to your app ? Is it through page.addEventListener
• What's the equivalent of omitBackground for screenshot (unless it's enabled by default) ?

Thanks in advance for your answer 😄 !