Response method body_string may use wrong character encoding about surf HOT 4 CLOSED

@Hawk777 Thanks for filing this; I hear you.

I think I built it this way because Rust strings are always guaranteed to be valid utf8. But if the encoding of the incoming request is different we should probably convert it first. Does that sound like it would be the right way to go?

Additionally we should probably indeed update the docs with more details of what we're doing. I hadn't really thought of it, but now that you mention it it's not surprising it can be confusing to people. Apologies!

from surf.

Yes, that is exactly what I would expect. It might also be worth deciding and documenting what the behaviour is (panic? return error?) if the response body is not actually text (e.g. application/octet-stream or some other non-textual type), though perhaps that could just fall under the “return error if the encoding is invalid” rule, and rely on callers to not call this function for binary content?

from surf.

I think an Err() is the correct way to go if the encoding listed in the Content-Type header is unsupported or is incorrect. In that case the Err() should ideally (imo) contain the byte array so you can decide to do something else, like passing it around as a byte array for octet-stream, or using a crate like chardet to try to fall back an encoding that does work.

let response = surf::get("https://some.website/whatever").await?;
let body = match response.body_string().await {
    Ok(body) => body,
    Err(BodyStringError::DecodeError(bytes)) => try_detect_decode(bytes)?,
    Err(err) => Err(err)?,
};

do_something_with(body);

fn try_detect_decode(bytes: Vec<u8>) -> Result<String, SomeError> {
    let (encoding_name, _confidence) = chardet::detect(&bytes);
    if let Some(encoding) = encoding_rs::Encoding::for_label(encoding_name) {
        let (decoded, _used_encoding, failed) = encoding.decode(&bytes);
        if failed {
            Err("could not decode body")
        } else {
            Ok(decoded.to_string())
        }
    } else {
        return Err("could not detect encoding")
    }
}

encoding_rs is very nice—perhaps it could be an optional dependency, so folks talking to an HTTP API that they know will return utf8 do not need to ship its conversion tables.

from surf.

Should be addressed by #108! It tries to use the encoding from the content-type and returns an Err() if that doesn't work.

from surf.