Gen II: Invalid challenge-response about xled HOT 58 CLOSED

I'm seeing this issue with a gen2 string, but as far as I can tell the verification is only important if your client code thinks it might potentially talk to a non-genuine string, and cares about this? If you short-circuit the validation logic to always return true everything works perfectly. Ideally everything should work, but a quick fix would be to make validation optional as I don't think most clients will care. Looks like something they put in so you couldn't use their mobile app to control third party copies of the hardware.

Specifically, I altered ValidatingClientMixin.challenge_response_valid() to return True rather than actually checking. The lights can then be driven by the rest of the API and everything works as expected.

from xled.

Following pacc's comments, I modified lines 402-403 of 0.6.1 auth.py to comment out the calls to log.error() and raise ValidateError(). This resulted in my Gen2 435 light tree working as expected. I can modify the colors of the tree from the cli. I did get an error on trying to get-mode from the cli, have not dug into it yet. My goal is to enable a color picker in home assistant, so this is 'good enough' for me. I'm sorry I'm not very experienced with git/github, or I would put in a pull request to suppress the auth error.
(edited: tree has 435, not 535 lights).

from xled.

Thanks to @magicus who submitted #63 which should fix this issue. That has been merged now in develop branch to get more testing before I make a new release.

Think of this as last chance for something nice in 2020. 🍾

from xled.

I have finally gen2 Twinkly at my hands now. By coincidence it is TWW210SPP as well. And I'm not able to reproduce issue you are seeing. Verification works for me. I have just checked that with your challenge value.

Currently only firmware difference comes to my mind. I haven't upgraded from 2.4.21 yet.

from xled.

Guys, I think we can hit two birds with one stone... I started implementing #59, to use the mac address from the gestalt command. It returned a slightly different mac than what the mac library stated, in my case it ended with f5 instead of f4. When I fed this value into the challenge/response crypto, it turned out to match...

So it seems that the Twinkly unit itself thinks it has a different mac address than it actually has (or does it have dual mac addresses?), and this is what they expect to use for XOR:ing in the crypto. So fixing #59 should also, indirectly, fix this bug...

from xled.

Oh, this would be awesome. If the challenge/response worked, getting BP to work would be quite easy.

from xled.

Hi, I haven't seen this issue so far. What device do you use there? Is that one of the second generation devices?

from xled.

Yes it's 2nd generation.
https://twinkly.com/products/strings-special-edition-250-leds/

from xled.

Unfortunately I don't have second edition available. So I don't know what has changed there.

Frankly I'm waiting for RGB+W wall one to appear on stock and so far I wasn't successfull.

from xled.

What does the app show as the firmware version number?

from xled.

Firmware 2.4.14 (updated yesterday)

from xled.

and the model/product_code - probably starts TW

from xled.

TWS250SPP-B

from xled.

Have you tried the sync with music function?
In V1 it was done via the phone listening and telling the device what to do ... but I think in V2 it is the device itself that listens ...

from xled.

You're right :
Generation II
Generation II controllers has an embedded microphone able to pick up the sound.
from https://twinkly.com/knowledge/music-sync/

from xled.

Thanks for the link.
I also see there "This function is temporarily available only for iOS device, update for Android devices is due on the last week of November 2019."

from xled.

I can try this evening to sync my twinkly with music and tell you if it's recorded from the smartphone or from the controller. I'm currently not near my Twinkly.

from xled.

Report on home-assistant that the basic script in use there is working (for on/off) with updated firmware.
https://community.home-assistant.io/t/twinkly-christmas-lights/31354/96
That script has not been changed recently but does not do the challenge/response calculation in quite the same way (but I thought the end result would be the same).

from xled.

I tried music sync, it currently uses phone microphone (Android Pie).
I read home-assistant thread, I tried to re apply effect but I'm still getting an error with challenge-response.

from xled.

I used your very good API documentation to create my own PHP script and cURL to call Twinkly API.
I can get a valid token, then call /xled/v1/verify with a valid return code and use /xled/v1/led/mode to off/demo the twinkly so it's enough for me, that's what I wanted to do.
I don't know why Python script is not receving a right challenge response as Gen2 seems to use same mechanism as Gen1 uses.
Anyway thank you for excellent documentation!

from xled.

Same issue here with Gen II Twinkly lights. Hoping for an update!

from xled.

I used your very good API documentation to create my own PHP script and cURL to call Twinkly API.
I can get a valid token, then call /xled/v1/verify with a valid return code and use /xled/v1/led/mode to off/demo the twinkly so it's enough for me, that's what I wanted to do.
I don't know why Python script is not receving a right challenge response as Gen2 seems to use same mechanism as Gen1 uses.
Anyway thank you for excellent documentation!

Hi 0rsa, is it possible to share your PHP script? I'm not so experienced in using Python, but I do know a bit of PHP. Thanks in advance!

from xled.

@SKeehnen
A very basic script which turn on / off a twinkly device.

Replace TWINKLY_DEVICE_IP by your twinkly local IP
To switch on, pass GET parameter ?mode=on

`<?php

function RandomToken($length = 32){
if(!isset($length) || intval($length) <= 8 ){
$length = 32;
}
if (function_exists('random_bytes')) {
return bin2hex(random_bytes($length));
}
if (function_exists('mcrypt_create_iv')) {
return bin2hex(mcrypt_create_iv($length, MCRYPT_DEV_URANDOM));
}
if (function_exists('openssl_random_pseudo_bytes')) {
return bin2hex(openssl_random_pseudo_bytes($length));
}
}

function Salt(){
return substr(strtr(base64_encode(hex2bin(RandomToken(32))), '+', '.'), 0, 44);
}

$curl = curl_init();

curl_setopt_array($curl, array(
CURLOPT_URL => "http://TWINKLY_DEVICE_IP/xled/v1/login",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "POST",
CURLOPT_POSTFIELDS => "{"challenge": "".Salt().""}",
CURLOPT_HTTPHEADER => array(
"cache-control: no-cache",
"content-type: application/json",
"postman-token: 93e6a2db-aa7c-d3bc-eb9b-ad99edc8bd9f"
),
));

$response = curl_exec($curl);
$err = curl_error($curl);

curl_close($curl);

if ($err) {
echo "cURL Error #:" . $err;
} else {
$json = json_decode($response);
print_r($json);

$curl = curl_init();

curl_setopt_array($curl, array(
  CURLOPT_URL => "http://TWINKLY_DEVICE_IP/xled/v1/verify",
  CURLOPT_RETURNTRANSFER => true,
  CURLOPT_ENCODING => "",
  CURLOPT_MAXREDIRS => 10,
  CURLOPT_TIMEOUT => 30,
  CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
  CURLOPT_CUSTOMREQUEST => "POST",
  CURLOPT_POSTFIELDS => "{\"challenge-response\": \"".$json->challenge-response."\"}",
  CURLOPT_HTTPHEADER => array(
	"cache-control: no-cache",
	"content-type: application/json",
	"postman-token: a72a1cfe-fa4e-020b-41b4-2edb1372338a",
	"x-auth-token: ".$json->authentication_token
  ),
));

$response = curl_exec($curl);
$err = curl_error($curl);

curl_close($curl);

if ($err) {
  echo "cURL Error #:" . $err;
} else {
  echo $response;
  
	$curl = curl_init();

	curl_setopt_array($curl, array(
	  CURLOPT_URL => "http://TWINKLY_DEVICE_IP/xled/v1/led/mode",
	  CURLOPT_RETURNTRANSFER => true,
	  CURLOPT_ENCODING => "",
	  CURLOPT_MAXREDIRS => 10,
	  CURLOPT_TIMEOUT => 30,
	  CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
	  CURLOPT_CUSTOMREQUEST => "POST",
	  CURLOPT_POSTFIELDS => "{\"mode\":\"".($_GET['mode']=='on'?"demo":"off")."\"}",
	  CURLOPT_HTTPHEADER => array(
		"cache-control: no-cache",
		"content-type: application/json",
		"postman-token: 3dd2a60d-895b-4dd2-320a-de381747e91a",
		"x-auth-token: ".$json->authentication_token
	  ),
	));

	$response = curl_exec($curl);
	$err = curl_error($curl);

	curl_close($curl);

	if ($err) {
	  echo "cURL Error #:" . $err;
	} else {
	  echo $response;
	}
  
}

}

`

from xled.

Thanks so much!

from xled.

I still don't have Gen2 lights so I cannot test this nor confirm this bug is valid.

Recently with #36 library and CLI will retry to authenticate if token is found to be invalid. Could you try if that code won't help you with Gen 2 authentication validation issues?

from xled.

Still not working with Gen II, 150 led RGB strip with firmware 2.2.1 (might update later)

Got the PHP script working, but also in that case 'challenge-response' option was not necessary to turn the strip on or off.

Testing xled again with challenge_response_valid() hardcoded to True everything seems to work except for the initial error message in the first comment of this issue.

from xled.

Still not working with Gen II, 150 led RGB strip with firmware 2.2.1 (might update later)

Have you tried the latest version 0.6.1?

from xled.

Latest, had to clone the git to hardcode the validation to true.

from xled.

Going out on a quick look limb here, but it would appear as though gen 2 doesn't require the security method.

from xled.

Added Pull Request #53 to failover to Gen2 if Gen1 fails.

from xled.

Going out on a quick look limb here, but it would appear as though gen 2 doesn't require the security method.

I don't think statement about requirement of security method is correct. Otherwise Twinkly would be possible to control without authentication.

from xled.

And for anybody else in this thread who is still interested in this - please make sure you don't have any other device connected to to your lights - e.g. phone app, maybe even disconnect them from access to Internet - and test this issue again.

from xled.

I'm poking at this right now (All of my twinkly are gen2)

The basic flow looks to still be the same. I've attached a pcap of the login flow as done by xled: pcap.zip

The challenge-response coming back isn't what xled expects:

ERROR:xled.auth:challenge-response invalid. Received challenge-response: '8f1fd55ffb2428c0d8bc6fe7bdc59210684b6829' but '4eebaae5d19f9b9ad673ab25bb07e90e2fe9c13d' was expected.

If I use the provided response in /verify, the twinkly seems happy:

$ curl -v -X POST -H "X-Auth-Token: zE6n0SpFihM=" -d '{"challenge-response": "8f1fd55ffb2428c0d8bc6fe7bdc59210684b6829"}' http://192.168.4.37/xled/v1/verify
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 192.168.4.37:80...
* TCP_NODELAY set
* Connected to 192.168.4.37 (192.168.4.37) port 80 (#0)
> POST /xled/v1/verify HTTP/1.1
> Host: 192.168.4.37
> User-Agent: curl/7.68.0
> Accept: */*
> X-Auth-Token: zE6n0SpFihM=
> Content-Length: 66
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 66 out of 66 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: esp-httpd/0.5
< Transfer-Encoding: chunked
< Content-Type: application/json
< 
* Connection #0 to host 192.168.4.37 left intact
{"code":1000}

My guess would be the shared secret has changed.

I second some earlier comments - the validation in xled is done to in theory verify that the lights are genuine. Given that it's trivial to fake once the shared secret is known - why don't we just drop the verification of the challenge-response in xled, or at least make that optional?

from xled.

@mikebryant what are models of your gen2 devices?

from xled.

All RGBW, I've got some curtains: TWW210SPP, and some icicles: TWI190SPP

from xled.

I've confirmed removing that challenge response validation in xled (but still sending it back to verify) allows me to control my lights (just checking on/off)

@scrool What would be your inclination here - keep the ability to verify if the lights are responding with the same shared secret we know, and plumb a flag through to make that check optional for Gen 2, or remove it entirely? Both options should keep gen 1 devices working because the validation is only on the python side, and I doubt that anyone is relying on trying to avoid anyone attempting to badly spoof being a twinkly light.

from xled.

{"product_name":"Twinkly","hardware_version":"100","bytes_per_led":3,"hw_id":"xxx","flash_size":64,"led_type":14,"product_code":"TWS250STP","fw_family":"F","device_name":"Igloo Lights","uptime":"672055336","mac":"xx:xx:xx:xx:xx:xx","uuid":"xxx","max_supported_led":1200,"number_of_led":250,"led_profile":"RGB","frame_rate":20,"measured_frame_rate":25,"movie_capacity":992,"wire_type":1,"copyright":"LEDWORKS 2018","code":1000}

from xled.

Ah, that's interesting. I don't have any V1 to compare to. I'm on 2.5.6

from xled.

My app updated firmware when I first got setup. 2.5.6 here as well

from xled.

I'm wondering if any of you could help me with low level debugging. Most likely you'll need rooted phone, adb and you'll reveal your MAC address of Twinkly.

Install frida server on your phone https://frida.re/docs/android/

Save following code on your machine, e.g. as trace.js

var Color = {
    RESET: "\x1b[39;49;00m", Black: "0;01", Blue: "4;01", Cyan: "6;01", Gray: "7;11", Green: "2;01", Purple: "5;01", Red: "1;01", Yellow: "3;01",
    Light: {
        Black: "0;11", Blue: "4;11", Cyan: "6;11", Gray: "7;01", Green: "2;11", Purple: "5;11", Red: "1;11", Yellow: "3;11"
    }
};

/**
 *
 * @param input. 
 *      If an object is passed it will print as json 
 * @param kwargs  options map {
 *     -l level: string;   log/warn/error
 *     -i indent: boolean;     print JSON prettify
 *     -c color: @see ColorMap
 * }
 */
var LOG = function (input, kwargs) {
    kwargs = kwargs || {};
    var logLevel = kwargs['l'] || 'log', colorPrefix = '\x1b[3', colorSuffix = 'm';
    if (typeof input === 'object')
        input = JSON.stringify(input, null, kwargs['i'] ? 2 : null);
    if (kwargs['c'])
        input = colorPrefix + kwargs['c'] + colorSuffix + input + Color.RESET;
    console[logLevel](input);
};

function traceMethod(targetClassMethod) {
    var delim = targetClassMethod.lastIndexOf('.');
    if (delim === -1)
        return;

    var targetClass = targetClassMethod.slice(0, delim);
    var targetMethod = targetClassMethod.slice(delim + 1, targetClassMethod.length);

    var hook = Java.use(targetClass);
    var overloadCount = hook[targetMethod].overloads.length;

    LOG({ tracing: targetClassMethod, overloaded: overloadCount }, { c: Color.Green });

    for (var i = 0; i < overloadCount; i++) {
        hook[targetMethod].overloads[i].implementation = function () {
            var log = { '#': targetClassMethod, args: [] };

            for (var j = 0; j < arguments.length; j++) {
                var arg = arguments[j];
                // quick&dirty fix for java.io.StringWriter char[].toString() impl because frida prints [object Object]
                if (j === 0 && arguments[j]) {
                    if (arguments[j].toString() === '[object Object]') {
                        var s = [];
                        for (var k = 0, l = arguments[j].length; k < l; k++) {
                            s.push(arguments[j][k]);
                        }
                        arg = s.join('');
                    }
                }
                log.args.push({ i: j, o: arg, s: arg ? arg.toString(): 'null'});
            }

            var retval;
            try {
                retval = this[targetMethod].apply(this, arguments); // might crash (Frida bug?)
                log.returns = { val: retval, str: retval ? retval.toString() : null };
            } catch (e) {
                console.error(e);
            }
            LOG(log, { c: Color.Blue });
            return retval;
        }
    }
}

var Main = function() {
    traceMethod('com.twinkly.security.Security.getSecret');
    traceMethod('com.twinkly.security.Security.calculateChallange');
};

Java.perform(Main);

and start frida with that file:

frida -U -f com.twinkly -l trace.js --no-pause

It should print arguments and return values of those two methods listed at the end the script. Please share those with me. If you would like to reach out to me privately let me know.

PS: majority of that javascript code lays around on internet on various places. E.g. https://stackoverflow.com/a/57474349

from xled.

I'll attempt via nox for the moment.

from xled.

Looks like won't be compatible, I don't have a rootable device handy otherwise.
Won't connect and I suppose more issues trying past that.
frida/frida#403

from xled.

public static final int[] KEY_CHALLENGE = {101, 118, 101, 110, 109, 111, 114, 101, 115, 101, 99, 114, 101, 116, 33, 33};
evenmoresecret!!

public static String calculateChallange(String str, String str2) {
        return TUtils.bytesToHex(sha1(new RC4().encrypt(getSecret(SecurityConfig.KEY_CHALLENGE, getMac(str2)), Base64.decode(str, 2))), false);
    }

private static byte[] getSecret(int[] iArr, byte[] bArr) {
        byte[] bArr2 = new byte[iArr.length];
        int i = 0;
        for (int i2 = 0; i2 < bArr2.length; i2++) {
            byte b = iArr[i2] ^ bArr[i];
            i++;
            if (i == bArr.length) {
                i = 0;
            }
            bArr2[i2] = (byte) (b & UByte.MAX_VALUE);
        }
        return bArr2;
    }

private static byte[] getMac(String str) {
        int[] iArr = SecurityConfig.FILTER_MAC;
        String[] split = str.split(":");
        byte[] bArr = new byte[6];
        for (int i = 0; i < split.length; i++) {
            bArr[i] = (byte) (Integer.parseInt(split[i], 16) & iArr[i]);
        }
        return bArr;
    }

from xled.

@Emerica this doesn't look like JavaScript code from Frida. What is that code about?

from xled.

Already mentioned above, not going to work from the environment I'm at now, maybe someone else can help you out there.
Above is parts of the Android Java Securtiy config from the APK.

from xled.

Above is parts of the Android Java Securtiy config from the APK.

I'm afraid this could be illegal in some countries if that's made by reverse engineering of the code. Also Twinkly's code is copyrighted and I don't think they have allowed to redistribute this.

from xled.

You do what you need to do.

from xled.

I don't agree with that statement. I have decided to hide that content for now.

from xled.

You followed that statement to a tee, you did what you needed to do.

from xled.

I have this same issue and the suggestion above seemed to work for me:

I'm seeing this issue with a gen2 string, but as far as I can tell the verification is only important if your client code thinks it might potentially talk to a non-genuine string, and cares about this?

[...]

Specifically, I altered ValidatingClientMixin.challenge_response_valid() to return True rather than actually checking. The lights can then be driven by the rest of the API and everything works as expected.

Is there any reason this code needs to exist?

from xled.

I just got myself a 210 led curtain, after checking that this very python library existed (I tried to do my research...), but ran into this very problem. :-(

I have a {'product_name': 'Twinkly', 'hardware_version': '100', 'bytes_per_led': 4, 'hw_id': '387df4', 'flash_size': 64, 'led_type': 12, 'product_code': 'TWW210SPP', 'fw_family': 'G', 'device_name': 'Twinkly_387DF5', 'uptime': '74448916', 'mac': '98:f4:ab:**:**:**', 'uuid': '0A1E721B-AE33-****-****-D72A22D76425', 'max_supported_led': 1200, 'number_of_led': 210, 'led_profile': 'RGBW', 'frame_rate': 12.99, 'movie_capacity': 992, 'wire_type': 4, 'copyright': 'LEDWORKS 2018', 'code': 1000}

The Twinkly app wanted me to upgrade firmware but I have as of yet not done that.

I can confirm that by disabling the authentication check in auth.py I can access the lights; I can change brightness and mode. For the moment, I have not tried anything else. My goal is to be able to upload movies from a script instead of the app; I do hope this works as well (need to look into the movie format first to try that). If not, I'm definitely interested in helping getting this resolved properly. (I'm quite well versed in both python and hobby electronics so I think I might be of help, but I don't have any Android devices, so sorry, no help there :-().

from xled.

Some quick updates: Yes, I can indeed set a movie. In fact, all tested commands work fine. It seem to me that the verification is not intended as a way for the hardware to check that there is a valid client, but instead a way for the official Tinkly app to ensure that it is not operating third party hardware. So I can't really see the need for it in this python client lib.

Secondly, the provided high-level functions does not work with RGB+W devices. I got quite a shock at my initial attempt to run set_static_color and got a fruit salad on display. 😮 I'll try to provide a PR to fix this. For the record, the byte order for RGB+W is white, red, green, blue.

from xled.

Secondly, the provided high-level functions does not work with RGB+W devices. I got quite a shock at my initial attempt to run set_static_color and got a fruit salad on display. I'll try to provide a PR to fix this. For the record, the byte order for RGB+W is white, red, green, blue.

Yes, code hasn't been updated for Gen II's RGB+W. See also #51 before you try to look deeper into that.

I'd like to get that documented first in https://github.com/scrool/xled-docs/ .

from xled.

Actually just yesterday I have found out that Gen II uses multiple MAC addresses - one with WiFi adapter, second with Bluetooth one. And there might be something like base address for hwid. But the base address from ESP32 docs: https://docs.espressif.com/projects/esp-idf/en/v3.1.7/api-reference/system/base_mac_address.html isn't actually used.

I have just pushed my changes to scrool/xled-docs@b08e4ac

from xled.

When I fed this value into the challenge/response crypto, it turned out to match...

I wanted to test this out later. Amazing news that crypto works against that address!

from xled.

FWIW, I also observed the off-by-one MAC address when initially trying to set up my tree (just using the twinkly app, didn't know xled existed at the time). I was trying to set a DHCP reservation for its IP in my router. Ended up having to go back in and modify the mac in the router config. Really drove me crazy when it kept refusing to use the first mac after restarting it. As mentioned above, it's a Gen II 435 light tree.

from xled.

Nice catch !

from xled.

I'm able to reproduce the issue now.

One catch is that a Twinkly in access point mode has same MAC address as mac key in gestalt API call. In usual client (station) mode those addresses differ.

That won't be primary reason - because you were reporting issue with a discovery. I suspect that discovery has similar quirk. I need to double check that though.

Edit: I have documented this in scrool/xled-docs@5cf7b60

from xled.