Code Monkey home page Code Monkey logo

Comments (12)

ijl20 avatar ijl20 commented on June 25, 2024 2

I see the same or similar issue - in my case a connection to a mostly default (apart from logging) configured localhost Ubuntu mosquitto broker has timeout disconnects resulting in missed messages. It looks to me (logs below) like the gmqtt client sends PINGREQ messages every (default) 60 seconds, which maintains the connection to the mosquitto broker fine. But when the subscription receives a message it then (usually, I think) stops sending the PINGREQs, the mosquitto broker drops the client (by closing the socket) after 1.5 x the keepalive period, the gmqtt client immediately wakes up on the disconnect and reconnects and resubscribes (taking ~5 seconds to reconnect and missing any messages during that period). When the gmqtt client reconnects, it starts sending PINGREQs again. In this non-PINGREQ period the gmqtt client subscription continues to work fine, i.e. I can publish more messages and they will all be received. But the mqtt client will still be kicked off the mosquitto broker after 1.5 x the keepalive and have to reconnect.

Reproducible with the 'simple example' subscriber program given on the homepage README of this repo. I modified this provided gmqtt example simply to include timestamps and to connect to my default localhost mosquitto broker in my github here.

I published simple test messages on topic TEST/user about every 10 seconds using mosquitto_pub, giving the seconds timestamp as the message (seen as the mosqpub client connections in the mosquitto log).

I ran mosquitto_sub on the same host (seen as the mosqsub client), listing all topics/messages, giving:
Local subscription to mosquitto: me@my-machine:~$ mosquitto_sub -v -t '#'

TEST/user 1585751829
TEST/user 1585751840
TEST/user 1585751851
TEST/user 1585751861
TEST/user 1585751871

The example program subscribes to TEST/# and prints out these messages.
Running the 'example program': me@my-machine:~$ python3 test_timeout.py

1585751818.500474 RECV MSG: b'1585751818'
1585751829.868452 RECV MSG: b'1585751829'
1585751840.392649 RECV MSG: b'1585751840'
1585751851.807200 RECV MSG: b'1585751851'
1585751859.924842 Disconnected
[TRYING WRITE TO CLOSED SOCKET]
1585751865.934178 Connected
1585751865.934926 SUBSCRIBED
1585751871.934515 RECV MSG: b'1585751871'

The example program output above shows it missing the 1585751861 message.

The mosquitto broker drops the GMQTT client at 1585751859, as shown in the
mosquitto broker log. Note the example program is client 'client-id', and also there is mosqpub and mosqsub to send messages and subscribe respectively.

me@my-machine:~$ tail -f /var/log/mosquitto/mosquitto.log

1585751769: New client connected from 127.0.0.1 as client-id (c0, k60, u'foo').
1585751769: Sending CONNACK to client-id (1, 0)
1585751769: Received SUBSCRIBE from client-id
1585751769:     TEST/# (QoS 0)
1585751769: client-id 0 TEST/#
1585751769: Sending SUBACK to client-id
1585751796: Received PINGREQ from mosqsub|4950-ijl20-tosh
1585751796: Sending PINGRESP to mosqsub|4950-ijl20-tosh
1585751818: New connection from 127.0.0.1 on port 1883.
1585751818: New client connected from 127.0.0.1 as mosqpub|5983-ijl20-tosh (c1, k60).
1585751818: Sending CONNACK to mosqpub|5983-ijl20-tosh (0, 0)
1585751818: Received PUBLISH from mosqpub|5983-ijl20-tosh (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751818: Sending PUBLISH to mosqsub|4950-ijl20-tosh (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751818: Sending PUBLISH to client-id (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751818: Received DISCONNECT from mosqpub|5983-ijl20-tosh
1585751818: Client mosqpub|5983-ijl20-tosh disconnected.
1585751829: New connection from 127.0.0.1 on port 1883.
1585751829: New client connected from 127.0.0.1 as mosqpub|5993-ijl20-tosh (c1, k60).
1585751829: Sending CONNACK to mosqpub|5993-ijl20-tosh (0, 0)
1585751829: Received PUBLISH from mosqpub|5993-ijl20-tosh (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751829: Sending PUBLISH to mosqsub|4950-ijl20-tosh (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751829: Sending PUBLISH to client-id (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751829: Received DISCONNECT from mosqpub|5993-ijl20-tosh
1585751829: Client mosqpub|5993-ijl20-tosh disconnected.
1585751840: New connection from 127.0.0.1 on port 1883.
1585751840: New client connected from 127.0.0.1 as mosqpub|6002-ijl20-tosh (c1, k60).
1585751840: Sending CONNACK to mosqpub|6002-ijl20-tosh (0, 0)
1585751840: Received PUBLISH from mosqpub|6002-ijl20-tosh (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751840: Sending PUBLISH to mosqsub|4950-ijl20-tosh (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751840: Sending PUBLISH to client-id (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751840: Received DISCONNECT from mosqpub|6002-ijl20-tosh
1585751840: Client mosqpub|6002-ijl20-tosh disconnected.
1585751851: New connection from 127.0.0.1 on port 1883.
1585751851: New client connected from 127.0.0.1 as mosqpub|6009-ijl20-tosh (c1, k60).
1585751851: Sending CONNACK to mosqpub|6009-ijl20-tosh (0, 0)
1585751851: Received PUBLISH from mosqpub|6009-ijl20-tosh (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751851: Sending PUBLISH to mosqsub|4950-ijl20-tosh (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751851: Sending PUBLISH to client-id (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751851: Received DISCONNECT from mosqpub|6009-ijl20-tosh
1585751851: Client mosqpub|6009-ijl20-tosh disconnected.
1585751856: Received PINGREQ from mosqsub|4950-ijl20-tosh
1585751856: Sending PINGRESP to mosqsub|4950-ijl20-tosh
1585751859: Client client-id has exceeded timeout, disconnecting.
1585751859: Socket error on client client-id, disconnecting.
1585751861: New connection from 127.0.0.1 on port 1883.
1585751861: New client connected from 127.0.0.1 as mosqpub|6022-ijl20-tosh (c1, k60).
1585751861: Sending CONNACK to mosqpub|6022-ijl20-tosh (0, 0)
1585751861: Received PUBLISH from mosqpub|6022-ijl20-tosh (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751861: Sending PUBLISH to mosqsub|4950-ijl20-tosh (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751861: Received DISCONNECT from mosqpub|6022-ijl20-tosh
1585751861: Client mosqpub|6022-ijl20-tosh disconnected.
1585751865: New connection from 127.0.0.1 on port 1883.
1585751865: Client client-id disconnected.
1585751865: New client connected from 127.0.0.1 as client-id (c0, k60, u'foo').
1585751865: Sending CONNACK to client-id (1, 0)
1585751865: Received SUBSCRIBE from client-id
1585751865:     TEST/# (QoS 0)
1585751865: client-id 0 TEST/#
1585751865: Sending SUBACK to client-id
1585751871: New connection from 127.0.0.1 on port 1883.
1585751871: New client connected from 127.0.0.1 as mosqpub|6034-ijl20-tosh (c1, k60).
1585751871: Sending CONNACK to mosqpub|6034-ijl20-tosh (0, 0)
1585751871: Received PUBLISH from mosqpub|6034-ijl20-tosh (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751871: Sending PUBLISH to mosqsub|4950-ijl20-tosh (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751871: Sending PUBLISH to client-id (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585751871: Received DISCONNECT from mosqpub|6034-ijl20-tosh
1585751871: Client mosqpub|6034-ijl20-tosh disconnected.
1585751915: Received PINGREQ from mosqsub|4950-ijl20-tosh
1585751915: Sending PINGRESP to mosqsub|4950-ijl20-tosh
1585751925: Received PINGREQ from client-id
1585751925: Sending PINGRESP to client-id
1585751976: Received PINGREQ from mosqsub|4950-ijl20-tosh
1585751976: Sending PINGRESP to mosqsub|4950-ijl20-tosh
1585751985: Received PINGREQ from client-id
1585751985: Sending PINGRESP to client-id

Another run with logging set to debug, this time with keepalive=20:

(venv) me@my-machine:~/src/acp_mqtt_decoders$ python3 test_timeout.py
2020-04-01 17:04:40,044 [CONNECTION MADE]
2020-04-01 17:04:40,045 [CMD 0x20] b'\x00\x00'
2020-04-01 17:04:40,045 [CONNACK] flags: 0x0, result: 0x0
1585757080.045449 Connected
2020-04-01 17:04:40,045 NEW ID: 1
2020-04-01 17:04:40,045 [SEND SUB] 1 ['TEST/#']
2020-04-01 17:04:40,045 [QoS query IS EMPTY]
2020-04-01 17:04:40,045 Sending PUBLISH (q1), 'TIME', ... (18 bytes)
2020-04-01 17:04:40,045 NEW ID: 2
2020-04-01 17:04:40,046 [CMD 0x90] b'\x00\x01\x00'
2020-04-01 17:04:40,046 [SUBACK] 1 (0,)
1585757080.046218 SUBSCRIBED
2020-04-01 17:04:40,046 FREE MID: 1
2020-04-01 17:04:40,046 [CMD 0x40] b'\x00\x02'
2020-04-01 17:04:40,046 [RECEIVED PUBACK FOR] 2
2020-04-01 17:04:40,046 FREE MID: 2
2020-04-01 17:04:40,046 [REMOVE MESSAGE] 2
2020-04-01 17:04:45,051 [QoS query IS EMPTY]
2020-04-01 17:04:50,052 [QoS query IS EMPTY]
2020-04-01 17:04:55,056 [QoS query IS EMPTY]
2020-04-01 17:05:00,056 [CMD 0xd0] b''
2020-04-01 17:05:00,057 [PONG REQUEST] 0xd0 b''
2020-04-01 17:05:00,059 [QoS query IS EMPTY]
2020-04-01 17:05:05,065 [QoS query IS EMPTY]
2020-04-01 17:05:07,567 [CMD 0x30] b'\x00\tTEST/user1585757107'
2020-04-01 17:05:07,567 [RECV TEST/user with QoS: 0] b'1585757107'
1585757107.568310 RECV MSG: b'1585757107'
2020-04-01 17:05:07,568 FREE MID: None
2020-04-01 17:05:10,066 [QoS query IS EMPTY]
2020-04-01 17:05:15,072 [QoS query IS EMPTY]
2020-04-01 17:05:20,073 [QoS query IS EMPTY]
2020-04-01 17:05:25,079 [QoS query IS EMPTY]
2020-04-01 17:05:29,939 [RECV EMPTY] Connection will be reset automatically.
2020-04-01 17:05:29,940 [CONN CLOSE NORMALLY]
2020-04-01 17:05:29,940 [CMD 0xe0] b''
1585757129.941526 Disconnected
2020-04-01 17:05:29,942 [TRYING WRITE TO CLOSED SOCKET]
2020-04-01 17:05:35,949 [CONNECTION MADE]
2020-04-01 17:05:35,951 [CMD 0x20] b'\x00\x00'
2020-04-01 17:05:35,951 [CONNACK] flags: 0x0, result: 0x0
1585757135.952271 Connected
2020-04-01 17:05:35,952 NEW ID: 3
2020-04-01 17:05:35,952 [SEND SUB] 3 ['TEST/#']
2020-04-01 17:05:35,953 [QoS query IS EMPTY]
2020-04-01 17:05:35,953 [CMD 0x90] b'\x00\x03\x00'
2020-04-01 17:05:35,954 [SUBACK] 3 (0,)
1585757135.954416 SUBSCRIBED
2020-04-01 17:05:35,954 FREE MID: 3
2020-04-01 17:05:40,960 [QoS query IS EMPTY]
2020-04-01 17:05:45,961 [QoS query IS EMPTY]
^C2020-04-01 17:05:47,423 [CONN CLOSE NORMALLY]
2020-04-01 17:05:47,424 [CMD 0xe0] b''
1585757147.424661 Disconnected

For which the mosquitto broker log (last PING at 1585757100, dropped at 1585757129 :

1585757080: New client connected from 127.0.0.1 as client-id (c1, k20, u'foo').
1585757080: Sending CONNACK to client-id (0, 0)
1585757080: Received SUBSCRIBE from client-id
1585757080:     TEST/# (QoS 0)
1585757080: client-id 0 TEST/#
1585757080: Sending SUBACK to client-id
1585757080: Received PUBLISH from client-id (d0, q1, r0, m2, 'TIME', ... (18 bytes))
1585757080: Sending PUBACK to client-id (Mid: 2)
1585757080: Sending PUBLISH to mosqsub|4950-ijl20-tosh (d0, q0, r0, m0, 'TIME', ... (18 bytes))
1585757100: Received PINGREQ from client-id
1585757100: Sending PINGRESP to client-id
1585757107: New connection from 127.0.0.1 on port 1883.
1585757107: New client connected from 127.0.0.1 as mosqpub|10410-ijl20-tos (c1, k60).
1585757107: Sending CONNACK to mosqpub|10410-ijl20-tos (0, 0)
1585757107: Received PUBLISH from mosqpub|10410-ijl20-tos (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585757107: Sending PUBLISH to mosqsub|4950-ijl20-tosh (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585757107: Sending PUBLISH to client-id (d0, q0, r0, m0, 'TEST/user', ... (10 bytes))
1585757107: Received DISCONNECT from mosqpub|10410-ijl20-tos
1585757107: Client mosqpub|10410-ijl20-tos disconnected.
1585757129: Client client-id has exceeded timeout, disconnecting.
1585757129: Socket error on client client-id, disconnecting.
1585757135: New connection from 127.0.0.1 on port 1883.
1585757135: New client connected from 127.0.0.1 as client-id (c0, k20, u'foo').
1585757135: Sending CONNACK to client-id (0, 0)
1585757135: Received SUBSCRIBE from client-id
1585757135:     TEST/# (QoS 0)
1585757135: client-id 0 TEST/#
1585757135: Sending SUBACK to client-id
1585757136: Received PINGREQ from mosqsub|4950-ijl20-tosh
1585757136: Sending PINGRESP to mosqsub|4950-ijl20-tosh
1585757147: Received DISCONNECT from client-id
1585757147: Client client-id disconnected.
1585757196: Received PINGREQ from mosqsub|4950-ijl20-tosh
1585757196: Sending PINGRESP to mosqsub|4950-ijl20-tosh
1585757256: Received PINGREQ from mosqsub|4950-ijl20-tosh
1585757256: Sending PINGRESP to mosqsub|4950-ijl20-tosh
1585757316: Received PINGREQ from mosqsub|4950-ijl20-tosh

from gmqtt.

ijl20 avatar ijl20 commented on June 25, 2024 2

Quick heads up... on further testing with patch #105 I think there remains a similar timeout issue for a client publisher. (I haven't properly debugged yet, just wanted to post quickly)

I modified my earlier test_timeout.py (itself based on the gmqtt homepage sample) to remove the subscription in on_connect and add another co-routine to publish every 10 seconds ( full code ):

def ts_string():
    return '{:.6f}'.format(time.time())

async def publisher(client):
    while True:
        await asyncio.sleep(10)
        ts = ts_string()
        print("{} [test_pub_timeout] publishing".format(ts))
        client.publish('TIME', ts, qos=0)

async def main(broker_host):
    client = MQTTClient("client-id")

    client.on_connect = on_connect
    client.on_message = on_message
    client.on_disconnect = on_disconnect
    client.on_subscribe = on_subscribe

    client.set_auth_credentials('mqtt-user','mqtt-password')
    await client.connect(broker_host, keepalive=20, version=MQTTv311)

    client.publish('TIME', str(time.time()), qos=0)

    asyncio.ensure_future(publisher(client))

    await STOP.wait()
    await client.disconnect()

Although the regular publishes succeed, the responses from mosquitto do not cause the connection.py self._last_data_in to be updated, and the gmqtt client disconnects. Pls note for this test the client has keepalive=20.

2020-04-07 13:20:46,746 [CONNECTION MADE]
2020-04-07 13:20:46,746 [CMD 0x20] b'\x01\x00'
2020-04-07 13:20:46,747 [CONNACK] flags: 0x1, result: 0x0
1586262046.747338 Connected
2020-04-07 13:20:46,747 [QoS query IS EMPTY]
2020-04-07 13:20:51,748 [QoS query IS EMPTY]
1586262052.676289 [test_pub_timeout] publishing
2020-04-07 13:20:52,676 Sending PUBLISH (q0), 'TIME', ... (17 bytes)
2020-04-07 13:20:56,748 [_keep_connection fix] since: 10.001
2020-04-07 13:20:56,749 [QoS query IS EMPTY]
2020-04-07 13:21:01,752 [QoS query IS EMPTY]
1586262062.678739 [test_pub_timeout] publishing
2020-04-07 13:21:02,678 Sending PUBLISH (q0), 'TIME', ... (17 bytes)
2020-04-07 13:21:06,752 [_keep_connection fix] since: 20.006
2020-04-07 13:21:06,753 [QoS query IS EMPTY]
2020-04-07 13:21:11,758 [QoS query IS EMPTY]
1586262072.680279 [test_pub_timeout] publishing
2020-04-07 13:21:12,680 Sending PUBLISH (q0), 'TIME', ... (17 bytes)
2020-04-07 13:21:16,757 [_keep_connection fix] since: 30.010
2020-04-07 13:21:16,760 [QoS query IS EMPTY]
2020-04-07 13:21:21,765 [QoS query IS EMPTY]
1586262082.682315 [test_pub_timeout] publishing
2020-04-07 13:21:22,682 Sending PUBLISH (q0), 'TIME', ... (17 bytes)
2020-04-07 13:21:26,761 [_keep_connection fix] since: 40.014
2020-04-07 13:21:26,761 [LOST HEARTBEAT FOR 40 SECONDS, GOING TO CLOSE CONNECTION]
2020-04-07 13:21:26,761 [CONN CLOSE NORMALLY]
2020-04-07 13:21:26,762 [CMD 0xe0] b''
1586262086.762310 Disconnected
2020-04-07 13:21:26,762 [TRYING WRITE TO CLOSED SOCKET]
1586262092.689112 [test_pub_timeout] publishing
2020-04-07 13:21:32,689 Sending PUBLISH (q0), 'TIME', ... (17 bytes)
2020-04-07 13:21:32,689 [TRYING WRITE TO CLOSED SOCKET]
2020-04-07 13:21:32,763 [CONNECTION MADE]
2020-04-07 13:21:32,763 [CMD 0x20] b'\x01\x00'
2020-04-07 13:21:32,764 [CONNACK] flags: 0x1, result: 0x0
1586262092.764317 Connected
2020-04-07 13:21:32,764 [QoS query IS EMPTY]
2020-04-07 13:21:37,770 [QoS query IS EMPTY]
1586262102.695539 [test_pub_timeout] publishing
2020-04-07 13:21:42,695 Sending PUBLISH (q0), 'TIME', ... (17 bytes)
2020-04-07 13:21:42,763 [_keep_connection fix] since: 10.000

From a simple mosquitto_sub output, the messages published to TIME were as below, i.e. 1586262092 was lost :

1586262052.676289
1586262062.678739
1586262072.680279
1586262082.682315
1586262102.695539

from gmqtt.

dgambin avatar dgambin commented on June 25, 2024 2

I have just tried and for now I don't see any problem :-)

from gmqtt.

Lenka42 avatar Lenka42 commented on June 25, 2024 1

From both broker and client sides it looks like socket was closed unexpectedly :( I guess more research on system and network settings is needed

from gmqtt.

Lenka42 avatar Lenka42 commented on June 25, 2024

@dgambin could you please set debug-level logging?
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(message)s')

from gmqtt.

dgambin avatar dgambin commented on June 25, 2024

@Lenka42 debug log attached
debug.log

from gmqtt.

Mixser avatar Mixser commented on June 25, 2024

Hi @ijl20, thanks for your investigation, it will help us to understand and find out where is the mistake.

from gmqtt.

ijl20 avatar ijl20 commented on June 25, 2024

On further debugging the issue seems to be that gmqtt is treating an incoming message from a mosquitto subscription as ok to reset internal timestamp MQTTConnection._last_data_in, but mosquitto is not using that publish to reset its internal keepalive timer.

I modified MQTTConnection._keep_connection as below,

  1. to add the additional logger writes to show the internal timestamp (which shows the gmqtt _last_data_in time getting reset so the next PINGREQ not getting sent),

  2. to comment out the _last_data_in test so the PINGREQ is always sent - the timeout at mosquitto then no longer occurs.

    def _keep_connection(self):
        logger.debug('[_keep_connection] since: {:.3f}'.format(time.monotonic() - self._last_data_in))
        
        if self.is_closing():
            return

        if time.monotonic() - self._last_data_in >= 2 * self._keepalive:
            logger.warning("[LOST HEARTBEAT FOR %s SECONDS, GOING TO CLOSE CONNECTION]", 2 * self._keepalive)
            asyncio.ensure_future(self.close())
            return

        #if time.monotonic() - self._last_data_in >= 0.8 * self._keepalive:
        #    logger.debug('[_keep_connection] send_ping_request()')
        self._send_ping_request()
            
        logger.debug('[_keep_connection] resetting own loop.call_later()]')
        self._keep_connection_callback = asyncio.get_event_loop().call_later(self._keepalive / 2, self._keep_connection)

I think using _last_data_in from MQTTConnection.put_package is questionable to assume the broker has also reset its timeout (certainly doesn't work with mosquitto). Maybe _last_data_out would work but that variable is initialized in MQTTConnection but never used AFAIK.

Thanks for the effort supporting this - IMHO gmqtt is a nice product which sensibly aims at providing just the client well rather than client+broker.

from gmqtt.

Lenka42 avatar Lenka42 commented on June 25, 2024

@ijl20 thanks for investigation 🙂
This is an interesting case, when messages from client (Publish qos 0) are coming quit often, but broker does not respond in any way. I pushed commit with a fix for this case #105 , could you please check if looks good to you?

from gmqtt.

ijl20 avatar ijl20 commented on June 25, 2024

Fix looks good to me (I cloned master from Lenka42/gmqtt), and my pub & sub test code all behaves fine now. I hadn't considered the significance of frequent + qos=0 publishes being the special case in my 'publisher' issue but see exactly what you mean.

from gmqtt.

Lenka42 avatar Lenka42 commented on June 25, 2024

@dgambin Could you please check the newest version 0.6.4 and see if it solves your problem?

from gmqtt.

Lenka42 avatar Lenka42 commented on June 25, 2024

@dgambin @ijl20 it seems to me that issue is solved. If you have any other consideration, feel free to reopen 🙂

from gmqtt.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.