fefit / visdom Goto Github PK

A library use jQuery like API for html parsing & node selecting & node mutation, suitable for web scraping and html confusion.

License: MIT License

Rust 92.92% Go 3.79% JavaScript 3.13% Shell 0.16%

rust scrape html-parse jquery dom-manipulation confuse css-selector

visdom's People

Contributors

Stargazers

Watchers

Forkers

remimimimimi isgasho zhangxianhong john0x zignis mrxsisyphus

visdom's Issues

error: failed to get `mesdoc` as a dependency of package `visdom v0.1.4

# mesdoc = "0.1.11"
rphtml = "0.3.6"
mesdoc = { path = "../mesdoc" }
# rphtml = { path = "../rphtml"}

please fix mesdoc = { path = "../mesdoc" }

内存泄漏

在一个loop中，使用了let doc = Vis::load(&text)，doc.find()之后，调用了doc.remove(),text.clear()
内存仍会以1~2M的速度持续增加

将`Result<Elements, Box<dyn Error>>` 改为`Result<Elements, Box<dyn Error + Send>>`

可以将Result<Elements, Box<dyn Error>> 改为Result<Elements, Box<dyn Error + Send>>吗。以便与anyhow兼容

Please add support for html method for all elements

Hello, I'd like to get all elements with html tags and attribute. The currently available methods are .html() and .outer_html(). However, these methods only target the first element. The .text() gets all the elements in plain text. Is there a way to achieve this purpose with html?

How to remove a DOM element?

Hi! 请问我该如何删除一个符合要求的节点？以及如何增加节点？

let mut img_list = document.find("img[src]");
img_list.for_each(|_index, ele| {
  // 如何删除符合要求的节点？
  // How should I remove the element
});
let svg = Vis::load("<svg></svg>").unwrap();
// 如何把节点添加到document中？
// How should I append this svg to the document?

如何获取节点的tag name（节点名称）？

非常好Rust库。
在使用过程中，遇到一个问题：遍历所有节点时，希望能获取节点的名称，从源码看有tag_name(),但没看到可调用的接口。

let html = r#"
<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <title>.eq</title>
  </head>
  <body>
    <ul id="menu">
      <li class="item-1">item-1</li>
      <li class="item-2">item-2</li>
      <li class="item-3">item-3</li>
    </ul>
  </body>
</html>
"#;
let root = Vis::load(html)?;
root.each(|_index, ele|{
  println!("tag:{}", ele.tag_name()); // 如何获取tag_name
  true
})

set_html and replace_with seems not work

my cargo.toml

visdom = { version = "0.5.8", features = ["insertion", "full"] }

the function do not work

fn get_images(root: &mut Elements, no: &str) {
root.find("img").for_each(|idx, img| {
if let Some(src) = img.get_attribute("data-src") {
let src = src.to_string();
let re = Regex::new("wx_fmt=([^&]*)").unwrap();
if let Some(cap) = re.captures(&src) {
if let Some(fmt) = cap.get(1) {
let ext = fmt.as_str();
let fname = format!("{no}-{idx}.{ext}");
let new_img = format!("");
// let new_img = Vis::load(new_img).unwrap();
// println!("{}", new_img.outer_html());
// let it = new_img.get_ref().iter().next().unwrap();
// println!("{}", it.as_ref().outer_html());
img.set_html(&new_img);
// img.as_mut().replace_with(&it);
// img.replace_with(it);
println!("1. {}", img.outer_html());
println!("2. {}", img.html());
}
}
}
true
});

()

}

No problem, my bad :(

Navigating sideway with `find` method

Hello, is there a way to navigate sideway with find method?

use std::ops::Index;
use visdom::{Vis, html};
use visdom::types::{BoxDynError, Elements};

fn main() -> Result<(), BoxDynError> {
    let html = r##"
        <div class='main'>
            <p>abc</p>
            <p>def</p>
        </div>
        <div class='main'>
            <p>ghi</p>
            <p>jkl</p>
        </div>
    "##;
    let doc = Vis::load(html)?;
    let div = doc.find("div.main");
    let div_length = div.length();
    div.each(|index, ele| {
        if index < div_length {
        let p = ele.find("p");
        return true;
        } false
    });
    Ok(())
}

# error[E0599]: the method `find` exists for mutable reference `&mut Box<dyn visdom::mesdoc::interface::element::IElementTrait>`, but its trait bounds were not satisfied

for div.each, ele.find("p") is not working. Also, why does .each need index?

doc.find("p:contains('好用')" panicked, only when Chinese characters appear in contains()

use visdom::Vis;

fn main() -> anyhow::Result<()> {
    let html = r#"
    <div>
    <p>Visdom is awesome</p>
    <p>Visdom 很好用</p>
    </div>
    "#;
    let doc = Vis::load(html).unwrap();
    let elements = doc.find("p:contains('Visdom')");  // work
    println!("elements : {}", elements.length());

    let elements = doc.find("p:contains('好用')");  // panic
    println!("elements : {}", elements.length());
    println!("done");
    Ok(())
}


elements : 2
thread 'main' panicked at 'range end index 8 out of range for slice of length 5', library/core/src/slice/index.rs:73:5
stack backtrace:
   0: rust_begin_unwind
             at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/core/src/panicking.rs:142:14
   2: core::slice::index::slice_end_index_len_fail_rt
             at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/core/src/slice/index.rs:73:5
   3: core::ops::function::FnOnce::call_once
             at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/core/src/ops/function.rs:248:5
   4: core::intrinsics::const_eval_select
             at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/core/src/intrinsics.rs:2372:5
   5: core::slice::index::slice_end_index_len_fail
             at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/core/src/slice/index.rs:67:9
   6: <core::ops::range::Range<usize> as core::slice::index::SliceIndex<[T]>>::index
             at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/core/src/slice/index.rs:304:13
   7: <core::ops::range::RangeTo<usize> as core::slice::index::SliceIndex<[T]>>::index
             at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/core/src/slice/index.rs:351:9
   8: core::slice::index::<impl core::ops::index::Index<I> for [T]>::index
             at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/core/src/slice/index.rs:18:9
   9: <visdom::mesdoc::selector::pattern::RegExp as visdom::mesdoc::selector::pattern::Pattern>::matched
             at /home/simon/.cargo/registry/src/github.com-1ecc6299db9ec823/visdom-0.5.2/src/mesdoc/selector/pattern.rs:357:17
  10: visdom::mesdoc::selector::pattern::exec
             at /home/simon/.cargo/registry/src/github.com-1ecc6299db9ec823/visdom-0.5.2/src/mesdoc/selector/pattern.rs:445:26
  11: visdom::mesdoc::selector::rule::Rule::exec_queues
             at /home/simon/.cargo/registry/src/github.com-1ecc6299db9ec823/visdom-0.5.2/src/mesdoc/selector/rule.rs:283:54
  12: visdom::mesdoc::selector::rule::Rule::exec
             at /home/simon/.cargo/registry/src/github.com-1ecc6299db9ec823/visdom-0.5.2/src/mesdoc/selector/rule.rs:276:3
  13: visdom::mesdoc::selector::Selector::from_str
             at /home/simon/.cargo/registry/src/github.com-1ecc6299db9ec823/visdom-0.5.2/src/mesdoc/selector/mod.rs:157:51
  14: visdom::mesdoc::interface::elements::Elements::find
             at /home/simon/.cargo/registry/src/github.com-1ecc6299db9ec823/visdom-0.5.2/src/mesdoc/interface/elements.rs:1098:11
  15: repl::main
             at ./src/bin/repl.rs:17:20
  16: core::ops::function::FnOnce::call_once
             at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/core/src/ops/function.rs:248:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

是否有更好的方式获取 select 元素的值？

let html = r##"
<!doctype html>
<html>
  <body>
	
    <select>
	  <option value="1">1</option>
	  <option value="2" selected="true">2</option>
	  <option value="3">3</option>
	</select>

  </body>
</html>
"##;
let doc = Vis::load(html)?;

let select = doc.find("#select");
println!("select value is {:?}", select.text());
// select value is "\n        1\n        \n        2\n        \n        3\n        \n        "

如何获取select选中的值？当前代码希望获取为”2”

不可用

包变得不可用

错误如下：

error[E0432]: unresolved imports `htmlentity::entity::encode_chars`, `htmlentity::entity::EntitySet`
  |
6 | use htmlentity::entity::{decode_chars_to, encode_chars, EncodeType, Entity, EntitySet};
  |                                           ^^^^^^^^^^^^                      ^^^^^^^^^ no `EntitySet` in `entity`
  |                                           |
  |                                           no `encode_chars` in `entity`
  |
help: a similar name exists in the module
  |
6 | use htmlentity::entity::{decode_chars_to, encode_char, EncodeType, Entity, EntitySet};
  |                                           ~~~~~~~~~~~
help: a similar name exists in the module
  |
6 | use htmlentity::entity::{decode_chars_to, encode_chars, EncodeType, Entity, Entity};
  |

这么好的项目为什么这么少人知道？

我Google了好几天，看过html5ver/craper/lol html之类的。他们的用法非常的复杂。

原本C# 有自己的Document可以直接使用，到了Rust就找不到了，最后在简书发现了作者的项目。

真是非常棒的项目，希望能多点star，希望作者能一直维护下去（Rust好多项目已经没人维护了，悲

最后，附上诚挚的感谢！

带:冒号的怎么解析？

你好，我用visdom去解析xml库，一般是没有问题，但是有些tag含:号就不好使了，如图：

尝试取tag_name，结果发现好像是空的:

缺少类似jquery的clone

测试中发现，好像没有提供jquery中对某个元素拷贝的接口，目前的cloned不是深拷贝，对返回的值进行操作依然会作用到之前的元素上

Stripping class entirely

Hello, thanks for the library

I'm wondering how do I remove class="" attributes? I want to remove anything that class related, but I can't know their class name.

I've tried this but it doesn't work

html.remove_attr("style");
html.remove_attr("class");
        ```

`Vis::load`返回的结果应该自动选中root元素

use visdom::{types::BoxDynError, Vis};

fn main()->Result<(),BoxDynError> {
    const HTML:&str = r#"<div id="contain"><p class="item"></p></div>"#;
	let mut v = Vis::load(HTML)?;
	v.set_attr("class", Some("ppp"));
	println!("{}",v.outer_html());
	Ok(())
}

输出：

期望：

对比Jquery

let d = $(`<div id="contain"><p class="item"></p></div>`);
d.addClass("ppp");
console.log(d[0].outerHTML)

输出：

visdom 需要通过v.find("#contain").set_attr("class", Some("ppp"));手动选择元素才能设置成功

error[E0432]: unresolved imports `rphtml::entity::encode_chars`, `rphtml::entity::EntitySet`
  --> src/lib.rs:32:19
   |
32 |     entity::{encode, encode_chars, EncodeType, EntitySet},
   |                      ^^^^^^^^^^^^              ^^^^^^^^^ no `EntitySet` in `entity`
   |                      |
   |                      no `encode_chars` in `entity`
   |
help: a similar name exists in the module
   |
32 |     entity::{encode, encode_char, EncodeType, EntitySet},
   |                      ~~~~~~~~~~~
help: a similar name exists in the module
   |
32 |     entity::{encode, encode_chars, EncodeType, Entity},
   |                                                ~~~~~~

error[E0308]: arguments to this function are incorrect
    --> src/lib.rs:250:21
     |
250  |                         let content = encode(content, EntitySet::Html, EncodeType::NamedOrDecimal);
     |                                       ^^^^^^ -------                   -------------------------- expected `&CharacterSet`, found enum `EncodeType`
     |                                              |
     |                                              expected slice `[u8]`, found `str`
     |
     = note: expected reference `&[u8]`
                found reference `&str`
note: function defined here
    --> /Users/ckruse/.cargo/registry/src/github.com-1ecc6299db9ec823/htmlentity-1.3.1/src/entity.rs:1126:8
     |
1126 | pub fn encode<'a>(
     |        ^^^^^^

error[E0308]: mismatched types
   --> src/lib.rs:251:50
    |
251 |                         let mut text_node = Node::create_text_node(content, None);
    |                                             ---------------------- ^^^^^^^ expected struct `Vec`, found struct `EncodedData`
    |                                             |
    |                                             arguments to this function are incorrect
    |
    = note: expected struct `Vec<char>`
               found struct `EncodedData<'_>`
note: associated function defined here
   --> /Users/ckruse/.cargo/registry/src/github.com-1ecc6299db9ec823/rphtml-0.5.9/src/parser.rs:511:9
    |
511 |     pub fn create_text_node(content: Vec<char>, code_at: Option<usize>) -> Self {
    |            ^^^^^^^^^^^^^^^^

Some errors have detailed explanations: E0308, E0432.
For more information about an error, try `rustc --explain E0308`.
error: could not compile `visdom` due to 3 previous errors

Can you please adopt the changes and release a new version?

Best regards,
CK