dns-oarc / dnsjit Goto Github PK

• Add producer interface
• Add receiver for the different layers or split the module into (OSI model?) layer 1, 2, 3 etc(?)
• Add receiver for IP fragments(?)

The own PCAP parser in input.fpcap and input.mmpcap needs to translate the link type found in the PCAP to the proper DLT, see https://github.com/the-tcpdump-group/libpcap/blob/90543970fd5fbed261d3637f5ec4811d7dde4e49/pcap-common.c#L1212 .

If I feed drool respdiff a pcap with 1909 packet or more it produces the following error:

output.respdiff[0x40688428] fatal: mdb_put meta.version failed

• Add TCP reassembling module based on dnscap code
• Maybe as lib.tcpreasm?

Support for TCP fast open is missing in underlying libuv library (libuv/libuv#1136) and its implementation would be quite complex.

• Remove private struct
• Pull in omg-dns parse code
• Remove omg-dns dependency
• Add support for core.object.udp
• Add support for core.object.tcp
• Add support for core.object.payload
• Add class, type, opcode, rcode etc to constants in module
• Add class, type, opcode, rcode etc to string mapping
• Add RR data interface
• Mock RR type objects

• Add support for core.object.dns

nil is returned instead of 0-length payload in non-blocking mode of TCP client.

• Create PCAP output using libpcap
  • pcap_dump_open()
  • pcap_dump_close()
  • pcap_dump_flush()
  • pcap_dump_ftell()

• Add mode fixed, set a fixed nanosecond pause between packets
• Add support for other objects then packet
• Add producer interface

RFC7918 has not been implemented

• Rework current code to :map_whole(true)
• Map only 2 pages at a time (unless map whole)
• Add conf for how big pages to map
• Add support for MAP_HUGETLB, MAP_HUGE_2M and MAP_HUGE_1GB

Certificates aren't verified, since checking that isn't really relevant to the primary purpose of the tool.

However, adding the possibility to select between opportunistic and strict mode RFC8310 might be useful for certain use cases, as well as enforcing strict mode for HTTP.

Related to DNS-OARC/drool#80:

• Add an unique ID to each query read/created so they can be correlated when sending to multiple targets

checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for style of include used by make... GNU
checking dependency style of gcc... gcc3
checking whether gcc and cc understand -c and -o together... yes
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking for ar... ar
checking the archiver (ar) interface... ar
./configure: line 4358: syntax error near unexpected token `disable-static'
./configure: line 4358: `LT_INIT(disable-static)'

centos version: CentOS Linux release 7.7.1908 (Core)

• Move queue code from filter.thread to core.channel
• Add different modes, current code as fast mode
• Add safe mode with shared read/write positions
• Add even mode
  • Add indicators of want_write, want_read
  • Use indicators to evenly distribute producers and consumers
• Add atomic locks (if supported)

• Expose core_object_copy()
• Add helper functions for object reference
• Add copy of all objects
  • Perhaps move copy function into the objects themselves
• Add uncast() to all objects to cast it back to core_object_t*
• Add free() to all objects

• for example set client id

Rework logging so that there are 3 levels of settings:

• global
• module
• object

When logging, if the objects log level is not set the check if the modules log level is set or use the global settings.

Overhaul all code to:

• fatal on input that is not external
• negative return values on errors if possible, fatal on error if returning pointers
• fatal on out of memory
• warning/critical on error with return

UDP: it should be possible to configure the client to re-transmit the query after a certain period.

Depends on #158

rehandshake / reauthentication hasn't been implemented. I don't see it as very relevant for the purpose of the tool - so this is most likely wontfix, just documenting it.

• settings
• throughput
• targets
• results
• output JSON etc

When using tcpcli, it seems that every response it produces is identical to the first one, regardless of the query sent. The same code works as expected when using udpcli instead.

Example:

dnsjit examples/replay.lua -Rt

query:
id:	742
...
questions:	class	type	labels
	IN	AAAA	<12>www.<16>japonec.<24>eu.
...
response:
id:	742
...
questions:	class	type	labels
	IN	AAAA	<12>www.<16>japonec.<24>eu.
...

query:
id:	33778
...
questions:	class	type	labels
	IN	AAAA	<12>www.<16>canoe42.<24>ca.
...
response:
id:	742
...
questions:	class	type	labels
	IN	AAAA	<12>www.<16>japonec.<24>eu.
...

query:
id:	10831
...
questions:	class	type	labels
	IN	AAAA	<12>ic.<15>mestodobruska.<29>cz.
...
response:
id:	742
...
questions:	class	type	labels
	IN	AAAA	<12>www.<16>japonec.<24>eu.
...

• Add libpcap functions
  • pcap_create()
  • pcap_activate()
  • pcap_findalldevs()
  • pcap_lookupdev()
  • pcap_set_snaplen()
  • pcap_set_promisc()
  • pcap_set_rfmon()
  • pcap_can_set_rfmon()
  • pcap_set_timeout()
  • pcap_set_buffer_size()
  • pcap_major_version()
  • pcap_minor_version()
  • pcap_setfilter()
  • pcap_setdirection()
  • pcap_stats()
  • pcap_statustostr()
  • pcap_lib_version()

RFC8446#section-2.3 has not been implemented.

It would be useful to match QID of the response and ignore mismatching DNS messages. Currently, if an unexpected packet arrives while waiting for a response, it will be accepted.

This can happen if the server doesn't respond within the configured dnsjit timeout, but the response still arrives later. The original query will be marked as timeout, but when the answer still arrives later, it will be assigned to the current ongoing query. Its response will in turn be assigned to the next query. This will effectively shift all the remaining responses, causing them to be assigned to incorrect queries.

• Implement adaptive realtime, see discussion in #122

• over TLS
• over DTLS
• % of traffic over configurable transport

Add the possibility to retry a UDP query that has TC=1 in the answer over TCP.

Depends on #158

• Add IP fragmentation handling module based on pcap-thread extension
• Maybe as lib.ipfrag?

• Document contrib/respdiff-dnsjit2answers.py
• Document the usage more clearly, example the conversion of tables
• Fix msec, response time
• Add support for dynamically increasing the database (see #92)

• Use malloc() instead of LuaJIT's metatype constructor

In the FreeBSD section you mention the manual install of libck. On FreeBSD, libck can be installed by

pkg install devel/concurrencykit

Maybe adapt the README ?

• Rework as an object
• Add a global object for all Lua states

• Add support for core.object.dns
• Add support for core.object.tcp
• Add connect() and use it in new object if host and port is given
• Add nonblocking(bool) default true
• Add wait_on_reply(bool) default false, if true then clear recvbuf before sending
• Remove checking for DNS queries (or option?)
• Add recvfrom() and receiver

• Add example modules as stand-alone compiled modules that can be used as templates for new modules
  • input.example
  • filter.example
  • output.example

Related to DNS-OARC/drool#80

• Add bitmap of available bytes for objects mask = -1 >> (len - need)
• Add macros to check if parts are available
• Add macros to get parts
• Change filter.layer to use mask (speed up ~10-15%)

• Create simple DNS TCP client
• Add connect() and use it in new object if host and port is given
• Add nonblocking(bool) default true
• Add wait_on_reply(bool) default false, if true then clear recvbuf before sending
• Add autoreconnect(bool) default false
• Add reconnect()
• Remove checking for DNS queries (or option?)
• Add recvfrom() and receiver
• Rewrite _produce() to be able to handle timeouts during length retrieval

Currently, it's not possible to push an empty string to a thread, because it has zero length. Attempting to do so triggers a fatal error.

Passing an empty string may be useful in some cases, e.g. when using it as a default value for getopt.

#!/usr/bin/env dnsjit
local thread = require("dnsjit.core.thread")
local getopt = require("dnsjit.lib.getopt").new({
	{ "s", "string", "", "optional string", "?" },
})
getopt:parse()

local opt_string = getopt:val("string")
local thr = thread.new()
thr:push(opt_string)

core/thread.c[176] core.thread[0xdeadbeef] fatal: len is zero

Queries are sent over TLS without any padding.

The harder part would be to request server padding via RFC 7830, since that would require modifying the queries themselves.

• Rewrite output.udpcli/output.tcpcli to a simpler socket() output that does not understand DNS, that logic can be handled elsewhere

Related to DNS-OARC/drool#80:

• add filter to send a query to multiple receivers

It seems that simple make install does not install all necessary files:

$ ./dumpdns-qr.lua /home/pspacek/pcaps/test.pcap
<< dnsjit v0.9.3 https://github.com/DNS-OARC/dnsjit/issues >>
core critical: ./dumpdns-qr.lua: ./dumpdns-qr.lua:9: module 'dnsjit.core.object' not found:
	no field package.preload['dnsjit.core.object']
	no file './dnsjit/core/object.lua'
	no file '/usr/share/luajit-2.1.0-beta3/dnsjit/core/object.lua'
	no file '/usr/local/share/lua/5.1/dnsjit/core/object.lua'
	no file '/usr/local/share/lua/5.1/dnsjit/core/object/init.lua'
	no file '/usr/share/lua/5.1/dnsjit/core/object.lua'
	no file '/usr/share/lua/5.1/dnsjit/core/object/init.lua'
	no file './dnsjit/core/object.so'
	no file '/usr/lo

$ ls /usr/local/share/lua/5.1/dnsjit
ls: cannot access '/usr/local/share/lua/5.1/dnsjit': No such file or directory

I did not specify any arguments to configure/make/make install.

To allow retry logic (or TCP fallback), the code has to be refactored to allow associating multiple queries with a single request.

Since linked-lists are used in multiple places (e.g. queries associated with connection), it might be worth investigating whether a different data structure with faster lookup would be beneficial.

The C-DNS (RFC8618) format looks quite useful for storing and processing large amount of DNS data.

Are there any plans to add reader/writer for C-DNS?

The output/dnssim component should support sending queries over IPv4 as well.

• Use core.channel to pass objects

If a connection is closed while there were pending queries, how should it be handled?

Currently, it stays dormant until it either times out, or another query triggers a new connection and the orphaned query is re-sent over that connection as well.

Re-sending orphaned queries by immediately by establishing new connections would have to have an upper limit of attempts and/or backoff time.