The toxcore library uses pthread_mutex_* functions in net_crypto. I wonder why it does that, given that the public API is not supposed to be thread-safe. I'd argue that not being thread-safe is a desirable feature, because it greatly simplifies the implementation by not having to deal with locking. If a client wants to use multiple threads to access a single tox instance, it can do its own locking around the calls (tox4j does this internally, so in Java you won't see it).

Is there any reason for having mutex locks inside toxcore if we can assume no concurrent calls to public API functions? If not, I would like to see all uses of pthread functions removed.

I can't find explicit alignment of SAVED_FRIEND struct
This structure is written into tox_save file as is. That is why very important to provide same alignment of this struct across all clients. What happens if I set default align to 1 byte for toxcore?
Ok, you answer me: "all good guys build toxcore using in-lib scripts and they always get same align"
But I build toxcore using msvc and I'm bad guy. Do you know, that it is very bad practice - save structures to file as is, without portable serialization?

Tentative spec => https://docs.google.com/document/d/1Y415qZcZQeKVsqZJ7gfH62pwOqQAAplXxcBSuvfOiV4/edit#heading=h.59m74p2xlbqq

Gist version (may go out of date) => https://gist.github.com/GrayHatter/409cae0c1f788b4eaeab9eaad695432b

Apparently, there are some problems with the toxcore tests because some of them have failed. I didn't have a Tox client running in the background.
Here the report:

Run toxcore tests:

git clone https://github.com/TokTok/toxcore
cd toxcore
autoreconf -i
./configure
make check

Result:

PASS: encryptsave_test
PASS: messenger_autotest
PASS: crypto_test
PASS: network_test
PASS: assoc_test
FAIL: onion_test
PASS: TCP_test
FAIL: tox_test
FAIL: dht_autotest
PASS: toxav_basic_test
PASS: toxav_many_test
============================================================================
Testsuite summary for tox 0.0.0
============================================================================
# TOTAL: 11
# PASS:  8
# SKIP:  0
# XFAIL: 0
# FAIL:  3
# XPASS: 0
# ERROR: 0

Further information:

=====================================
   tox 0.0.0: build/test-suite.log
=====================================

# TOTAL: 11
# PASS:  8
# SKIP:  0
# XFAIL: 0
# FAIL:  3
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: onion_test
================

Running suite(s): Onion
50%: Checks: 2, Failures: 0, Errors: 1
../auto_tests/onion_test.c:386:E:announce:test_announce:0: (after this point) Test timeout expired
FAIL onion_test (exit status: 1)

FAIL: tox_test
==============

Running suite(s): Tox
Toxes are online, took 10 seconds
tox clients connected took 0 seconds
tox clients messaging succeeded
3427
Toxes are online again after reloading, took 8 seconds
tox clients connected took 2 seconds
Starting file transfer test.
100MB file sent in 27 seconds
Starting file streaming transfer test.
Starting file 0 transfer test.
test_few_clients succeeded, took 38 seconds
33%: Checks: 6, Failures: 0, Errors: 4
../auto_tests/tox_test.c:909:E:many_clients:test_many_clients:0: (after this point) Test timeout expired
../auto_tests/tox_test.c:1008:E:many_clients_tcp:test_many_clients_tcp:0: (after this point) Test timeout expired
../auto_tests/tox_test.c:1107:E:many_clients_tcp_b:test_many_clients_tcp_b:0: (after this point) Test timeout expired
../auto_tests/tox_test.c:1225:E:many_group:test_many_group:0: (after this point) Test timeout expired
FAIL tox_test (exit status: 4)

FAIL: dht_autotest
==================

Running suite(s): DHT
50%: Checks: 2, Failures: 0, Errors: 1
../auto_tests/dht_test.c:609:E:DHT_test:test_DHT_test:0: (after this point) Test timeout expired
FAIL dht_autotest (exit status: 1)

Disable ipv6 on travis if it breaks too often, but have at least one test that exercises both paths.

this could be disabled by the user, but when enabled, a similar confirmation like the receipt confirmation that already exists, should be sent and upon receiving of such the client be notified, such that the the sender of a message can see that a message has been read.

Guys, I hope you will understand me correctly.
I really want to toxcore flourished.
I very sad to see how the project dies. (Do not argue! It is so indeed)
I was happy to know that TokTok continue toxcore developing, but the happiness was short-lived.
You guys make the same mistake that most of the dead opensource project: Pessimistic Merging
Please read this article and think about it.
http://hintjens.com/blog:106
I wish good
PS. That is why I don't want to make any PR's to toxcore.

Originally posted by @tux3 in irungentoo#1479:

Currently as I understand it toxcore shoves all the video data over UDP and hopes it reaches the destination.

This creates horrible video corruption when a key frame is lost. On desktop streaming, intra frames are super rare so we can easily go 10s at a time with completely garbled video output. It's a common thing when calling EchoBot for example, with both uTox and qTox.

Toxcore should try on the sending end to retransmit iframes until it either gets an ACK or the next iframe is ready, and on the receiving end it shouldn't give Tox clients inter frames that depend on missing intra frames.

For more information: qTox/qTox#2529

This needs to do roughly:

cd .. # Get out of the toxcore clone (=> /home/travis).
git clone --recursive --depth=1 https://github.com/TokTok/hstox # Fetch hstox and its submodules.
rm -rf hstox/test/toxcore/toxcore # Remove the toxcore submodule
mv toxcore hstox/test/toxcore/ # Move the to-be-tested toxcore (from PR/branch) to the test directory.
cd hstox
make check-toxcore TEST=cov-asan # Build with coverage and address sanitizer.

This involves a bit more than just the above script:

• Install ghc (>= 7.8). If downloading and installing ghc takes too much time, put it into $CACHE_DIR (~/cache).
• Make two builds, one with BUILD=toxcore, one BUILD=hstox, and do different things on each. The toxcore build will do what the current build does, the hstox build does the above script. Both upload their coverage data.

In 2015 the new ToxAV API had been merged.

Among other things, the introduced bitrate management hasn't really worked well.

Because of that, it had been disabled in qTox: qTox/qTox@db52ef6

@tux3 wrote:

For some users on poor connections, toxav would suggest lower and lower bitratres until reaching zero and disabling audio/video entirely, toxav would then not raise the bitrate back and the only way to re-enable audio/video would have been to change A/V sources or restart the call

We now log toxav's recommendation, but ignore it

Suggestion

Operating under the assumption that toxcore will follow Semantic Versioning. I suggest we take on the uTox style of sem-ver.

There will be two version "tracks" available. The first will exist ~only in code, and will won't be exposed to git. The second will be publicly release, tagged in git, and serve as "supported" versions.

This scheme allows for rapid development outside of a specific version, and when trouble shooting bugs or issues, quick identification of where and what could have caused the bug.

The actual flow and tags I suggest follow. (And will be updated from accepted comments made in this thread)

Example:

git tag v0.0.0 e6af3a7825e8307a501bc7c3e7b1ff323f081870

Changes are made to the toxcore api (already done), version identified in code should be v0.1.0 but this version will not be exposed to (tagged in) git.

Following the completion of the ToxAV stateless update. The version in code will be v0.2.0, with the addition of git tag v0.2.0 (this commit may be optionally pushed to a new branch stable)

The very next commit to master will change the version to v0.2.1 and any/all commits will be merged into this branch.**

Once a new version is ready to be pushed to stable, the version in code will be incremented to v0.2.2 (optionally v0.3.0) with git tag v0.2.2. This branch will be pushed to stable.**

The very next commit to master will be v0.2.3 and there will be no git tag for this version.

Discussion

In as instant developers can know if a bug exists in a stable version, or only in the development branch, or again, if it has been fixed already in a development branch.

It allows (forces) package maintainers to use only stable versions if they wish when they package and release a version, (because only stable versions will get tagged.) The one caveat to this is packages may be versioned in inconstantly with the version reported by toxcore if packagers blindly use the version reported by git tag.

Finally, developers can enable debug mode by default, if they detect that they are using a development build of toxcore. (toxcore_version_patch() % 2)

The user @yurivict reported (irungentoo#1550):

After talking to users and then being myself idle for many hours I came back to see all users offline. But after qtox restart users showed with online statuses. For some reason qtox stopped updating user status, so it created the false impression that everybody is offline.

Reporting it here because it appears to me the toxcore problem, and not a qtox problem.

qTox-1.3.0
toxcore-0.151112_1
FreeBSD

@GrayHatter wrote:

Can reproduce in µTox. seems to happen if the internet connection is down for an extended time.

With a ~stable API for clients to use.

I've left this proposal on the Tox blog and I will just copypaste it here because I'm lazy and cheap like that:

Say, I’m walking home, toxing on my phone. Then, when I come home, I want
to continue talking from my PC. It would be awesome to pass the call to my
other device seamlessly, without having to re-dial that person I’m talking to.

Guess, that wouldn’t be too hard to implement as long as all devices know
about all other devices: you just give user a list, and then he decides what
device to pass the call to.

Also, I thought a little bit further and realized, that this function shouldn't really stop at devices belonging to one person. There also should be possibility to redirect a call to another person entirely. The mechanism is basically the same, it only should differ visually.
Say, I get a call from a customer. I run into issue or a question, that I can't solve and answer, but my friend on the other desk can. Telling him to come over here is lame. Giving the customer his Tox ID and hoping that they call back is also lame. I just throw this call to my friend via menu.

So basically, it works like PBX without actually having a PBX.

May be this is a bit too much, but that's just a proposal, make of it what you will.

Currently, network.h includes a whole lot of platform specific headers (like u.h, windows.h, and unistd.h). We should confine these to a single file (or a few files).

I suggest we create a thin abstraction on the OS facilities for networking, and another on the ones for the monotonic clock. We could then have one file per platform kind (e.g. ${module}_win32.c, _unix.c, _osx.c, _posix.c, ...) and conditionally compile them according to preprocessor symbols, just like we do already. The downside is that it's a little bit more declaration overhead, because currently those #ifdefs are inside functions that will need to be redefined for each platform. The advantage is that there are no or few #ifdefs in the code itself, and the platform specific code is in a set of small files instead of (as it is now) in a 1100 line file mixing platform abstractions with application code.

something like make commit which will run the apidsl, style, direct decendant of current (local?) master, and other misc checks that Travis does.

#93 spawned a discussion around whether we should expose internal APIs to clients to give more control.

Where we are now.

We already split the library into 8 layers:

1. crypto, containing some abstractions for the libsodium crypto functions
2. network, basic networking abstractions (e.g. packet kind callbacks)
3. dht, the distributed hash table
4. netcrypto, containing TCP, onion, and net_crypto (basic encrypted end to end connections)
5. friends, friend connections and requests
6. messenger, almost all toxcore messenger functionality such as messages, typing notifications, and file transfers
7. group, old group chats built on top of (and a little integrated in) messenger
8. toxcore, the public API shim on top of everything

Currently, these layers are somewhat separated, and indeed don't call functions from layers above them, but some do have some knowledge of layers above them. Also, netcrypto is a bit large, containing all of TCP, onion, and net_crypto, because these three form a dependency cycle.

Where do we want to go?

If we want to expose internal APIs to clients, I would like to have clear interfaces between these layers and use them in the upper layers. Ideally, we would apidsl-generate each layer's public interface. That interface can then be used by clients (linking e.g. libtoxdht and using #include <tox/dht.h>). We would also use that public API in e.g. the netcrypto layer. For example, we would no longer directly access struct members, but instead expose APIs for all the things we want to be able to do.

What does the future hold?

When we have well-defined APIs like that, we can write tests against them, and since we only depend on the public API of each layer, that layer can be swapped out for an alternative implementation (e.g. in Rust or Haskell) or a mock for testing.

Currently, at least one test fails on Travis, which means we need to ignore all test failures.

Task: fix the test and remove the || true from the make check || true line.

We should change realloc_tox_array to error-out on overflow.
This is good for maintainability (we don't need to keep half the codebase in mind to know it is not overflowing) and for security.

Moreover, array (re)allocations should be changed to take advantage of it.

• Portable: it should work the same across architectures and compiler flags.
• Simple: it should be easy to parse and understand.
• Standard: it should use a standard format, e.g. msgpack or protobuf.

See #215 for a related discussion.

In #13, the group chat tests were dropped For Travis-CI.

Group chats will 'split' unpredictably, while only occasionally during local tests. On Travis the fail 40-80% of the time, making the whole test system itself, unusable.

IMO merging the new version of group chats, would by a better expenditure of time, rather than trying to fix these tests (or fix group chats in general because the reason they fail is problem.)

Many pointers are only set once before going out-of-scope, and could be constified.

Example at https://reviewable.io/reviews/toktok/toxcore/143#-KRuxior-8qxWa2CFtp5

• This is an invalid value for that enum type.
• All that code will silently break when the enum datatype is changed.

Do we have similar instances of that “idiom” in the current codebase?

Toxcore should have tests that try to connect and exchange data with an existing node in the wild running any previous version of toxcore. E.g. echobot.

@JFreegman mentioned an article about assertions that contains some useful thoughts on how to write good assertions and what to pay attention to.

As discussed in #5, artificially restricting the interface of a function to include sanity checks about layers on top of it is not ideal, since it breaks module abstractions. Also, swallowing severe code errors tends to hide bugs that could go unnoticed for some time.

Therefore, I would suggest we make a gradual move towards having our APIs accept their whole function domain and encoding bug-checking checks with knowledge of higher layers as assertions.

For a specific first case and its discussion, see #5.

@iphydf wrote:

problem: the tox public interface defines a lot of enumerations that are duplicated in the implementation

Proposed solutions:

• let directly include tox.h (carefully checking that only enums/types are used from there)
• split tox.h into two files: one for enums/types, one for functions

See #131. This change allows toxcore to support low latency applications like VPNs.

I have written following example code:

#include <stdio.h>
#include <stdlib.h>

#include <tox/tox.h>

int main(void)
{
    size_t i;
    uint8_t addr[TOX_ADDRESS_SIZE];
    char* nns="AABBCCDD";
    uint32_t ns;
    Tox* t;
    struct Tox_Options to;

    tox_options_default(&to);
    t=tox_new(&to, NULL);

    ns=tox_self_get_nospam(t);
    tox_self_get_address(t, addr);

    printf("current addr: ");
    for(i=0; i<TOX_ADDRESS_SIZE; i++)
        printf("%02X", addr[i]);
    putchar('\n');
    printf("current ns: %08X\n", ns);

    ns=strtoul(nns, NULL, 16);

    tox_self_set_nospam(t, ns);

    ns=tox_self_get_nospam(t);
    tox_self_get_address(t, addr);

    printf("current addr: ");
    for(i=0; i<TOX_ADDRESS_SIZE; i++)
        printf("%02X", addr[i]);
    putchar('\n');
    printf("current ns: %08X\n", ns);

    return 0;
}

When compiled with
gcc -o tns tns.c -ltoxcore

and executed, this prints

current addr: 5A9712242692889C310A7227CD78D3D9F0F9F5AE9608DDD3640175419093A33368C3A8A5173B
current ns: A5A8C368
current addr: 5A9712242692889C310A7227CD78D3D9F0F9F5AE9608DDD3640175419093A333DDCCBBAAB13B
current ns: AABBCCDD

As you can see, the checksum (the addresses last 4 bytes-2) is reversed, instead AABBCCDD they are DDCCBBAA.

I am running Arch linux, Kernel 4.8.1, gcc 6.2.1, toxcore-toktok-git 3911-1.
If this is intended behaviour, ignore this.

Hello. Whonix (whonix.org) developer here. We are an anonymity distro that some of you are familiar with. We have a vested interest in seeing Tox succeed as a secure/private communication alternative to the status quo.

IMHO moving the project binary builds to an online build server is a bad move especially in absence of a reproducible build system. You cannot rely on a cloud server to not work against your interests and that of your users. The Certificate Authority/HTTPS is badly broken system and not a guarantee against nation states trying to poison binaries.

I think that at the very least you should run your own build infrastructure until repro builds are realized so you can verify no one messed with binaries you distribute.

cc/ @adrelanos

How to reproduce:

1. Alice begin call to Bob and Bob begin call to Alice simultaneously.
2. Alice pressed hangup button and get crash.
3. Bob pressed hangup button and get crash.

Root of problem:
in file msi.c, in function handle_init FAILURE occurred. This mean unexpected kill_call called. So, MSICall object suddenly destroyed. Now we get dead pointer: ToxAV->calls[friend_number].msi_call
And final action: toxav_call_control get this pointer by call_get and crashed.

What are we doing?

We are moving towards stateless callbacks. This means that when registering a
callback, you no longer pass a user data pointer. Instead, you pass a user data
pointer to tox_iterate. This pointer is threaded through the code, passed to
each callback. The callback can modify the data pointed at. An extra indirection
will be needed if the pointer itself can change.

Why?

Currently, callbacks are registered with a user data pointer. This means the
library has N pointers for N different callbacks. These pointers need to be
managed by the client code. Managing the lifetime of the pointee can be
difficult. In C++, it takes special effort to ensure that the lifetime of user
data extends at least beyond the lifetime of the Tox instance. For other
languages, the situation is much worse. Java and other garbage collected
languages may move objects in memory, so the pointers are not stable. Tox4j goes
through a lot of effort to make the Java/Scala user experience a pleasant one by
keeping a global array of Tox+userdata on the C++ side, and communicating via
protobufs. A Haskell FFI would have to do similarly complex tricks.

Stateless callbacks ensure that a user data pointer only needs to live during a
single function call. This means that the user code (or language runtime) can
move the data around at will, as long as it sets the new location in the
callback.

How?

We are doing this change one callback at a time. After each callback, we ensure
that everything still works as expected. This means the toxcore change will
require 15 Pull Requests.

I am not sure if I should post issues here now, but I guess it makes sense because it's the only active repository.

There has been the consensus that toxcore should remove ToxDNS at all.

In general, the goal of toxcore is to provide a lighweight, reliable and secure codebase for the clients. The current solution with ToxDNS isn't both secure and decentralized.
Moreover, to use Tox IDs is not such a problem as it probably seems in my opinion. Nonetheless, the clients can of course implement HTTPS lookup services - but that's something that shouldn't be in toxcore.

Here the old issue -> irungentoo#1491

Added by iphy:

• Antox
• qTox
• Ricin
• Toxic
• uTox

Astyle is rather limited, and behaves differently in different versions. Create a .clang-format file that ideally produces a minimal diff from the current formatting. Create two PRs: one with changes that affect lines (git diff -w > my.patch; git checkout .; patch -p1 < my.patch), and another on top of that with the remaining changes (git diff -w shows an empty diff).

There is an issue filed against the nodes.tox.chat web app about it not being able to detect a bootstrap node as being online (Tox/ToxStatus#17). The reason why the web app can't detect that the node is online is because the web app sends a get-nodes request to an IPv6 address of the bootstrap node (one received by resolving the domain name), but the bootstrap node sends back the get-nodes reply through a different IPv6 address. The machine the bootstrap node runs on has several IPv6 addresses assigned to a single network interface. The web app has no knowledge that both of those IPv6 addresses belong to the same node, so it just ignores the reply coming from a different IPv6 address, thinking that it's a reply from a node different from the one it has sent the request to.

The solution for this would be to make sure that the replies to requests are sent using the same IPv6 address that the requests arrived though.

This is likely to be true for IPv4 addresses too.

Always-on local peer detection apparently can lead to a VPN leak, so having an option to disable it would be nice. (A warning in the tooltip might be a good idea, as well.)

See also qTox/qTox#2965 and irungentoo#1530.

At the moment, every client implements an own system how to save aliases for contacts. Because every client should support aliases (they are really important because of how Tox works), it would makes sense to add it to the .tox file.

If you use client X with aliases and export the Tox profile because you'd like to use it on another device, everybody expect that the names in the contact list are the same as before. But that's not the case currently.

Pros:
✅ It will be much easier for client developers to implement alias support
✅ Because of the consistent save system, import/export of a profile will be easier and more comfortable for the users
✅ Global alias support and no client patchwork rug solution
✅ Toxic gets alias support -> JFreegman/toxic#382

As far as I can remember, @JFreegman agreed to implement it when he has time. :) Unfortunately, he didn't start because there was no chance to get it in irungentoo/toxcore. But because we have a new repository now, it's time to discuss it again, I think.

In your vision you already recognize the value of Freenet-like systems for the future of the private and secure web: https://toktok.github.io/vision.html

I propose implementing an offline messaging system that relies on offloading messages to Freenet without reinventing the wheel.

Tox Bootstrap nodes can install Freenet and use its python bindings lib (pyFreenet) to upload/fetch message ciphertext on behalf of peers who could not connect to their contacts.

This will need some way for the Tox DHT to hold an advertisement for offline peers that they have pending messages/files. These advertisement messages will contain the Freenet keys for the uploaded media.

You should probably limit the permitted offline message number (and sizes) so advertisement messages don't get too big. Probably also limit the period you advertise to some number of days to avoid ballooning your DHT and because Freenet inserts are ephemeral and disappear if material is not requested after some time. The size limitation is needed because Freenet inserts take a long time.

When the intended recepient goes online they can ping the Tox DHT for the advertisement and ask for the bootstrap node to fetch them on their behalf - only if the sender is offline, otherwise the fallback to a direct transfer as normally happens.

cc/ @adrelanos

[1] https://github.com/freenet/pyFreenet
[2] https://pypi.python.org/pypi/pyFreenet/0.4.1

I don't think it needs to be updated every commit, but every once in a while I'd like something rolled into an existing pull.

Originally posted here by a deleted user:

If I make a file transfer with one or more contacts during an audio or video call, the call quality gets worse and sometimes the audio/video hangs for short moments. Normally, the call should still working great and the file transfers should be slowed down in return. It's not acceptable that the call is suffering.

fixed in 949ef78

@cleverca22 has given me a pcap dump that caught this bug which may or may not be helpful when dx this bug.

Can build toxcore with tests. Have an error:

CMakeFiles/av_test.dir/testing/av_test.c.o: In function `cvPointFrom32f':
av_test.c:(.text+0x549): undefined reference to `cvRound'
av_test.c:(.text+0x55f): undefined reference to `cvRound'
CMakeFiles/av_test.dir/testing/av_test.c.o: In function `cvReadInt':
av_test.c:(.text+0xda5): undefined reference to `cvRound'
CMakeFiles/av_test.dir/testing/av_test.c.o: In function `cvEllipseBox':
av_test.c:(.text+0x1011): undefined reference to `cvRound'
av_test.c:(.text+0x1033): undefined reference to `cvRound'

Seems, OpenCV 3.1 hasn't cvRound function

Ideally, I'd like to see some padding lines with the test that's about to run echo,'d before each test so I can see at a short glance what happened in what order.

================================
==   Running ASTYLE           ==
================================

After the introduction of nat traversal code, there are some functions that can be removed (adapting the rest of the code).
The functions are:

• new_networking()
• new_networking_ex()
• new_TCP_server()

The solution is to rename new_networking_nat() to new_networking() and get rid of all its occurrences into the code; then doing the same for new_TCP_server_nat().

Hi we did a test with 10 routers and qtox ( windows ) killed 6 out of 10 routers . We had to disable udp to make it work, which brings the questions , is the implementation of hole-punching correct ? How do other programs holepunch without killing the (same) routers. Why do we need udp holepunching when it could first try to upnp ? The router list includes tplink netgear ZTE thomson and others ( common routers where I live).

At least you could default udp off for now since I cant even recommend it to somone without have to be like " Oh and you need to go uncheck that udp thing... "

Many thanks,
Lait