ndpi-netfilter (netfilter-2.6 branch) doesn't recognize YouTube, GMail, sometimes WhatsApp about ndpi HOT 27 CLOSED

The flow_info branch is based on nDPI 2.8-stable. Maybe the problem is fixed there?

from ndpi.

from ndpi.

By the way, I have opened an issue on ntop/nDPI regarding WhatsApp, I don't know when and if it would be treated someday : ntop#683

from ndpi.

I confirm that with the branch flow_info, it still doesn't work correctly.

I have compiled nDPI 2.8, dev, 2.6, 2.2 and fed ndpiReader with my pcaps for Maps, Youtube, Whatsapp Gmail recognition, and I couldn't get them detected in any version.

Do you have these protocols detected ?

from ndpi.

egrep -i 'whatsapp|youtube|maps|gmail' /proc/net/xt_ndpi/proto
7a        7a/000001ff GMail            # 0 debug=0
7b        7b/000001ff GoogleMaps       # 0 debug=0
7c        7c/000001ff YouTube          # 0 debug=0
88        88/000001ff YouTubeUpload    # 0 debug=0
8e        8e/000001ff WhatsApp         # 1658 debug=0
bd        bd/000001ff WhatsAppVoice    # 1689751 debug=0
f2        f2/000001ff WhatsAppFiles    # 117 debug=0

Gmail, GoogleMaps and YouTube* not detected :(
Perhaps the file ndpi_content_match.c.inc is very outdated.
Can you send sample traffic files?

from ndpi.

Oups... my bad !
I was using the VM as gateway but... for IPv4 ! And I have a dual stack enabled. Facebook, Whatsapp, Google/Maps/Drive/Docs/Gmail are provided on IPv6 too ! So the traffic wasn't caught by the VM during tcpdump.
Now I have deactivated IPv6 on my computer, these traffics are well recognized by ndpiReader.

Unfortunately, xt_ndpi doesn't detect them. Any google service is accounted only by the Google entry. Whatsapp, Youtube, Docs, Drive, Gmail as still not detected.

from ndpi.

What's your GLIBC version ?

from ndpi.

glibc 2.21 and 2.27 (Slackware 14.1+ and Slackware-15.0 pre)
When building with glibc-2.27, a strange error occurs, but it can be avoided if you add the tzset () call at the beginning of main() in ndpiReader.c. Then there is no sigsegv in the output of statistical data after processing the file.
IPv6 is supported by xt_ndpi. I have not had time to test IPv6 operation.
It is unfortunate that the nTOP project does not collect ipv6 traffic samples for testing.

from ndpi.

Yep, I wasn't saying that IPv6 was not caught by xt_ndpi, just that my ipv6 traffic was not routed through this VM because I only added an ipv4 default route to it and because the services like Google and WhatsApp are also IPv6, the usual ipv6 gateway was used instead of the VM.

from ndpi.

I've just tested on Ubuntu Server 18.04 LTS (first time of my life I install it :-D), kernel 4.15 glibc 2.27, GCC 7.3, ndpi-netfilter flow_info branch. It gets built without error but It still doesn't work.

from ndpi.

Part of the traffic cannot be determined by signatures, it can be determined indirectly by addresses ( ip/AS ) and ports. This information in ntop/nDPI is extremely rarely updated.
Can you send traffic samples that are defined incorrectly?
If you have data on the availability of ip / port for a specific protocol or on the inconsistency of the available data in ndpi_content_match.c.inc, make a “Push request”.
This is the only way to improve this project.

from ndpi.

from ndpi.

from ndpi.

But the various services are correctly detected by ndpiReader with the pcaps recorded when these apps where not detected in live by xt_ndpi. So it doesn't seem related to ndpi itself.

Can you send me traffic to the address vel21ripn at gmail dot com in pcap format which is not detected by xt_ndpi and is defined in ndpiReader?

from ndpi.

I applied ntop#687 and ntop#688 (branch flow_info).

from ndpi.

from ndpi.

I can process the pcap file in ndpi-netfilter. I have a working set of scripts and a program similar to tcpreplay, with which I tested the identity of the ndpiReader and ndpi-netfilter results.
The scheme is as follows:
pcap -> tapsend -> netfilter -> -j NDPI - set-mark -j NFLOG -> ndpiReader
I have a modified ndpiReader that can get a protocol label from the NFLOG and compare it with the protocol detected by ndpiReader.
ndpi-nefilter gives bad results if it does not see traffic in both directions (for example, only incoming or outgoing traffic passes through it).

from ndpi.

See #54

from ndpi.

This seems everything is mostly solved. I can get YouTube, WhatsApp, GMail and other services recognized but WhatsApp voice is still not detected by xt_ndpi. It is however correctly detected by ndpiReader.

from ndpi.

I did not backporting changes from flow_info to netfilter-2.6

from ndpi.

from ndpi.

We have "WhatsApp voice" traffic examples ?

from ndpi.

sure !
whatsapp_voice.pcap.gz

from ndpi.

I deleted the packages with addresses 239.255.255.250 and 10.144.172.200.
ndpiReader result

Detected protocols:
        Unknown              packets: 41            bytes: 3526          flows: 3
        Google               packets: 4             bytes: 252           flows: 1
        WhatsApp             packets: 260           bytes: 31767         flows: 3
        Messenger            packets: 61            bytes: 6996          flows: 13
        WhatsAppVoice        packets: 1804          bytes: 281947        flows: 2

        1       UDP 192.168.0.175:40613 <-> 157.240.21.51:3478 [proto: 189/WhatsAppVoice][cat: VoIP/10][844 pkts/162698 bytes <-> 882 pkts/105582 bytes]
        3       UDP 192.168.0.175:40911 <-> 157.240.21.51:3478 [proto: 189/WhatsAppVoice][cat: VoIP/10][8 pkts/896 bytes <-> 70 pkts/12771 bytes]

/proc/net/xt_ndpi/flows

1557734980 1557734980 4 17 192.168.0.175 40613 157.240.21.51 3478 150630 93234 844 882 I=129,0 P=WhatsAppVoice
1557734980 1557734980 4 17 192.168.0.175 40911 157.240.21.51 3478 772 11791 8 70 I=129,0 P=WhatsAppVoice

I do not see a problem yet.

from ndpi.

Do you use your modified ndpiReader that reads labels from NFLOG ?

How can I read /proc/net/xt_ndpi/flows ? I can't cat on it. I have tried to set ndpi_enable_flow at insmod time but it's giving me something that is not what I'm looking for.

from ndpi.

Do you use your modified ndpiReader that reads labels from NFLOG ?

With the advent of flow_info, reading the NFLOG can only be useful for debugging purposes.
ndpiReader from my branches can read NFLOG.

How can I read /proc/net/xt_ndpi/flows ? I can't cat on it. I have tried to set ndpi_enable_flow at insmod

Please read the first 20 lines from ndpi-netfilter/FLOW_INFO.txt

from ndpi.

Oups, sorry, wonderful explanation :)

from ndpi.