caerbannog / esphttpclient Goto Github PK

Thanks fot this great library. But i was wondering that why cant we make this library simple to use like all other arduino libraries. just copy the folder in the libraries folder, and include the header file in the project and you are good to go.
But for this library why do we need to build or do all those other tricks ? Is there some specific reason or am i missing something ?
I have an existing arduino based project, can i simply use this library ? I tried doing what i explained above, but it will throw an error that http_post is undefined.

I try to execute this request:
http_get("http://api.thingspeak.com/update?key=ABCBCACABCBA1234&field1=0&field2=24.0&headers=false", http_callback_example);
and from httpclient.c->http_post I got:

hostname=api.thingspeak.com
port=80
path=/update?key=ABCBCACABCBA1234&field1=0&field2=24.0&headers=false

next executed is: http_raw_request ... req->post_data = esp_strdup(post_data);
give Fatal exception because post_data is NULL

temporary solution are inside static char * esp_strdup(const char * str)
add:
if (str == NULL) return NULL;
with this everything works good

I'm using your library with the 'Weather Underground' API. This responds with a response claiming to be 'HTTP 1.0' [1] and your library throws an error. Whilst technically correct this is in the 'yes but it works...' [2] category so you might want to implement something like some options flags, one of which could be HTTP_1_0_RSP_OK.

[1] Confirmed also with Wireshark and making same requests using browsers.
[2] The contents of the response is the word 'success' and the website shows that the the weather status update was accepted.

Given that there is an open source version of the ESP8266 lwIP port in esp-open-sdk, I think it would be nice to use that instead of the weird espconn interface. espconn doesn't seem to add much value upon raw lwIP but rather makes things like connection lifetime somewhat unclear. Usage of the plain lwIP APIs would make this code also more portable to other platforms.

I've looked at it a little bit. For the most part, the transformation looks pretty trivial and would likely even simplify some parts of the code.

The only complicated thing is supporting secure connections. AFAICT espconn wraps some TLS library (maybe mbedTLS like in this example code) behind the espconn_secure API. There the Espressif example code may prove to be helpful.

I'm interested in working on this. However, I'd like to hear your opinion on this. Obviously this change would break the build for users who don't have the lwIP headers available in their include path (i.e. not using a recent enough version of esp-open-sdk). Users of the library would also have to link against lwip_open instead of lwip.

Do you think these changes are suitable for the mainline code, or would it be better to indeed fork this library at this point?

Furthermore, I'm interested in taking this library further to the direction of static memory allocation and incremental parsing without buffering the response at all (leaving possible buffering up to the end user), possibly by integrating http-parser. Personally I would be inclined to fork the library as these changes would indeed turn the library into an entirely new direction.

This library is not working with the latest esp8266 v12e.

Trying to make a simple http_get returns a -1 status.
In case we pass the host name in the URL then DNS always fails.

Has anyone tested it on 8266 12e and succeeded ?

-Shyam

I'm not sure if this is a problem or not, but I get the following compilation warning when using this library:

.pioenvs/ota/esphttpclient/httpclient.c: In function 'chunked_decode':
.pioenvs/ota/esphttpclient/httpclient.c:178:3: warning: passing argument 2 of 'esp_strtol' from incompatible pointer type [enabled by default]
i = esp_strtol(src, endstr, 16);
^
.pioenvs/ota/esphttpclient/httpclient.c:86:1: note: expected 'char **' but argument is of type 'char *'
esp_strtol(const char *nptr, char **endptr, int base)
^

No heap available, failed to malloc 1592
:(

First of all, thanks for this awesome library, I'm using it and I'm happy with it. But I think I found a bug in case of chunked content

I've put a print of decoded buffer of received content, in disconnect_callback of httpclient.c so you can see from where in the orig file I'm displaying traces

		// FIXME: make sure this is not a partial response, using the Content-Length header.

			const char * version10 = "HTTP/1.0 ";
			const char * version11 = "HTTP/1.1 ";
			if (os_strncmp(req->buffer, version10, strlen(version10)) != 0
			 && os_strncmp(req->buffer, version11, strlen(version11)) != 0) {
				os_printf("Invalid version in %s\n", req->buffer);
			}
			else {
				http_status = atoi(req->buffer + strlen(version10));

				os_printf("Received buffer[%d]\n",req->buffer_size);
				hex_dump(req->buffer, req->buffer_size);

				/* find body and zero terminate headers */
				body = (char *)os_strstr(req->buffer, "\r\n\r\n") + 2;
				*body++ = '\0';
				*body++ = '\0';

				body_size = req->buffer_size - (body - req->buffer);

				if(os_strstr(req->buffer, "Transfer-Encoding: chunked"))
				{
					body_size = chunked_decode(body, body_size);
				}
			}
		}

		os_printf("Decoded buffer[%d]\n",body_size);
		hex_dump(body, body_size+8); // Add last 8 char to see following data

and here the results, as you can see the 1st display in the trace is corect

get http://192.168.1.34:8080/version1
Received buffer[227]
  0000  48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d  HTTP/1.1 200 OK.
  0010  0a 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 74  .Content-Type: t
  0020  65 78 74 2f 6a 73 6f 6e 0d 0a 44 61 74 65 3a 20  ext/json..Date: 
  0030  46 72 69 2c 20 33 31 20 4d 61 72 20 32 30 31 37  Fri, 31 Mar 2017
  0040  20 31 31 3a 30 36 3a 33 32 20 47 4d 54 0d 0a 43   11:06:32 GMT..C
  0050  6f 6e 6e 65 63 74 69 6f 6e 3a 20 63 6c 6f 73 65  onnection: close
  0060  0d 0a 54 72 61 6e 73 66 65 72 2d 45 6e 63 6f 64  ..Transfer-Encod
  0070  69 6e 67 3a 20 63 68 75 6e 6b 65 64 0d 0a 0d 0a  ing: chunked....
  0080  35 37 0d 0a 7b 22 76 65 72 73 69 6f 6e 22 3a 22  57..{"version":"
  0090  31 2e 30 2e 30 22 2c 22 66 69 6c 65 22 3a 22 2f  1.0.0","file":"/
  00a0  75 70 64 61 74 65 2f 75 73 65 72 31 2e 62 69 6e  update/user1.bin
  00b0  22 2c 22 6d 64 35 22 3a 22 34 30 39 65 62 61 63  ","md5":"409ebac
  00c0  35 36 30 39 30 62 33 64 31 66 36 34 33 64 30 62  56090b3d1f643d0b
  00d0  65 37 36 30 31 34 30 62 66 22 7d 0d 0a 30 0d 0a  e760140bf"}..0..
  00e0  0d 0a 00                                         ...

But when decoded (unchuncked) , data has been moved (wich is normal) but not the \0, so we're missing the \0 at the good place and got some more char in the string, even if the body_size var is correct

In the dump below I added 8 char mode than the real body_size to see buffer content after size

Decoded buffer[87]
  0000  7b 22 76 65 72 73 69 6f 6e 22 3a 22 31 2e 30 2e  {"version":"1.0.
  0010  30 22 2c 22 66 69 6c 65 22 3a 22 2f 75 70 64 61  0","file":"/upda
  0020  74 65 2f 75 73 65 72 31 2e 62 69 6e 22 2c 22 6d  te/user1.bin","m
  0030  64 35 22 3a 22 34 30 39 65 62 61 63 35 36 30 39  d5":"409ebac5609
  0040  30 62 33 64 31 66 36 34 33 64 30 62 65 37 36 30  0b3d1f643d0be760
  0050  31 34 30 62 66 22 7d 62 66 22 7d 0d 0a 30 0d     140bf"}bf"}..0.

To correct this I added body[body_size] = '\0' ;

if(os_strstr(req->buffer, "Transfer-Encoding: chunked"))
{
	body_size = chunked_decode(body, body_size);
	body[body_size] = `\0`;
}

And now results are fine, see the 0 at pos 0x57 ;-)

Decoded buffer[87]
  0000  7b 22 76 65 72 73 69 6f 6e 22 3a 22 31 2e 30 2e  {"version":"1.0.
  0010  30 22 2c 22 66 69 6c 65 22 3a 22 2f 75 70 64 61  0","file":"/upda
  0020  74 65 2f 75 73 65 72 31 2e 62 69 6e 22 2c 22 6d  te/user1.bin","m
  0030  64 35 22 3a 22 36 62 37 61 34 37 38 64 36 36 63  d5":"6b7a478d66c
  0040  31 61 30 63 35 36 34 34 38 63 31 37 32 63 33 61  1a0c56448c172c3a
  0050  33 35 30 36 38 22 7d 00 38 22 7d 0d 0a 30 0d     35068"}.8"}..0.

When I compile the http client with the SSL calls, I run out of iram, but if I remove all the espconn_secure_XXXXX calls then I am able to compile on ESP8266 project. Am I doing something wrong? Is the client able to fit in iram even with SSL?

Your response would be greatly appreciated.

Thanks
Sanjaya

Hi
Thankyou for your project

I combine esphttpclient into epshttpd. When compile show error:

CC httpclient.c
httpclient.c: In function 'esp_strdup':
httpclient.c:45:2: error: implicit declaration of function 'pvPortMalloc' [-Werror=implicit-function-declaration]
char * new_str = (char *)os_malloc(os_strlen(str) + 1); // 1 for null character
^
httpclient.c:45:2: error: implicit declaration of function 'ets_strlen' [-Werror=implicit-function-declaration]
httpclient.c:47:3: error: implicit declaration of function 'os_printf_plus' [-Werror=implicit-function-declaration]
os_printf("esp_strdup: malloc error");
^
httpclient.c:50:2: error: implicit declaration of function 'ets_strcpy' [-Werror=implicit-function-declaration]
os_strcpy(new_str, str);
^
httpclient.c: In function 'receive_callback':
httpclient.c:76:2: error: implicit declaration of function 'ets_memcpy' [-Werror=implicit-function-declaration]
os_memcpy(new_buffer, req->buffer, req->buffer_size);
^
httpclient.c:80:2: error: implicit declaration of function 'vPortFree' [-Werror=implicit-function-declaration]
os_free(req->buffer);
^
httpclient.c: In function 'connect_callback':
httpclient.c:119:3: error: implicit declaration of function 'ets_sprintf' [-Werror=implicit-function-declaration]
os_sprintf(post_headers, "Content-Length: %d\r\n", strlen(req->post_data));
^
httpclient.c: In function 'disconnect_callback':
httpclient.c:166:4: error: implicit declaration of function 'ets_strncmp' [-Werror=implicit-function-declaration]
if (os_strncmp(req->buffer, version, strlen(version)) != 0) {
^
httpclient.c:170:5: error: implicit declaration of function 'atoi' [-Werror=implicit-function-declaration]
http_status = atoi(req->buffer + strlen(version));
^
httpclient.c:171:5: error: implicit declaration of function 'ets_strstr' [-Werror=implicit-function-declaration]
body = (char *)os_strstr(req->buffer, "\r\n\r\n") + 4;

^

Can you fix help me. Thanks

I have an SDK which was updated a couple days ago.
Your simple get & simple post examples work fine.
Simple get works great with SSL.

If instead I do an ssl post (which i tested and the site supports)

http_post("https://httpbin.org/post", "first_word=hello&second_word=world", "Content-Type: application/x-www-form-urlencoded\r\n", http_callback_example);

I get:
DNS pending
http_status=-1

If I manually resolve:
http_post("https://54.175.219.8/post", "first_word=hello&second_word=world", "Content-Type: application/x-www-form-urlencoded\r\n", http_callback_example);

I get:
http_status=-1
ESPCONN_OK, 54.175.219.8, /post, 443, 1

This last line I printed to verify the args. Everything looks fine.
Can you double check my results?
Thoughts?

Thanks

Anyone working on enabling PUT requests? Or does the ESP somehow not support PUT requests (which seems hard to believe)? If not, I can add some code to handle HTTP PUT.

Hi,

trying to HTTP PUT a binary buffer.

http.put('http://example.com/api/v1/target',
    'Content-Type: application/octet-stream\r\n',
    payload,
    function(code, data)
        if (code < 0) then
            print("HTTP request failed")
        else
            print(code, data)
        end
end)

The payload sent is always cut off after the first "0" byte in the payload stream. How can I send binary data?

PS: I think the problem is buried in line 194 of http_client.c

    if ( req->post_data != NULL ) /* If there is data then add Content-Length header. */ {
	os_sprintf( post_headers, "Content-Length: %d\r\n", strlen( req->post_data ) );
}

strlen takes the 0 byte seriously...

PPS: The GET API has no problem with binary data at all...

PPPS: And your module is that fixed on "strlen(blabla)", that I suppose it will not be possible at all...

Thanks for this library! I've used it quite successfully, however I have run into an annoying problem:

I find that if the server I connect to has a valid DNS entry, but is not running an http server, then this library crashes the esp8266 right around espconn_connect() inside dns_callback.

Is this expected behaviour? I notice there is a TODO right at that line to change to using espconn_regist_reconcb(). Maybe that's related.

I'm trying to send my dht11 data from my esp8266 to my thingspeak channel but when i try to send data via https i get error -114. I don't know what to to do.
Here is my code:
url = "https://api.thingspeak.com/update.json?api_key=XXXXXXXXXXXXXXXXXXX&field1=" .. temp print(url) http.get(url, nil, function(code, data) if (code < 0) then print("HTTP request failed") else print(code, data) end end)

And here there's the modules in my firmware:
crypto,dht,file,gpio,http,mdns,mqtt,net,node,pwm,sjson,spi,tmr,tsl2561,uart,websocket,wifi,tls

I am using lua, of course. Anyway here's my error:
> HTTP client: Disconnected with error: -114 HTTP client: Connection timeout HTTP request failed

Can you help me?

In file included from user/httpclient.c:12:0:
user/httpclient.c: In function 'chunked_decode':
c:/Espressif/ESP8266_SDK/include/osapi.h:17:20: error: implicit declaration of function 'ets_memmove' [-Werror=implicit-function-declaration]
#define os_memmove ets_memmove
^
user/httpclient.c:184:3: note: in expansion of macro 'os_memmove'
os_memmove(&chunked[dst], src, i);
^
cc1.exe: all warnings being treated as errors
C:/Espressif/examples/ESP8266/esphttpd/Makefile:212: recipe for target 'build/user/httpclient.o' failed
mingw32-make.exe: *** [build/user/httpclient.o] Error 1

Hello for a start thanks a lot for your useful code. But I have a problem, when I call "http_get" function too often (less than one second period) i am loosing heap of esp8266 and if "http_get" period bigger than a second everything is ok. Give me a suggestion what to do?

This is more of a question rather than an issue. Do HTTPS requests to https://www.wikipedia.org work? I'm currently on SDK 1.4 and using esphttpclient but I seem to be getting either of the following errors when I try making HTTPS requests:

DNS found www.wikipedia.org 198.35.26.96
client handshake start.
client handshake failed

OR

DNS found www.github.com 192.30.252.130
client handshake start.
client handshake ok!
Connected
Sending request header
All sent
client's data invalid protocol

Do you hit the same errors?

HTTP/1.1 request is keep-alive by default, so the connection will hold on until timeout. I look into the code, and haven't find any disconnection on client side.