kismetwireless / python-kismet-db Goto Github PK

Getting back a buffer object instead of a string for the json field.

Also increment version and test Github's issue-to-commit tracking with new process.

When taking a large Kismet sqlite3 database and using kismet_log_to_pcap, a typeerror is returned:

# kismet_log_to_pcap --in Kismet-20190904-10-18-58-1.kismet --outtitle small --limit-packets 100000
Limiting to 100000 packets per file in small-X.pcap
DLT 0 for all packets
Logging to None
Traceback (most recent call last):
  File "/usr/local/bin/kismet_log_to_pcap", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/kismetdb/scripts/log_to_pcap.py", line 110, in main
    logf = open(results.outfile, file_mode)
TypeError: coercing to Unicode: need string or buffer, NoneType found

The kismet database has plenty of packets:

# ls -la | grep Kismet
-rw-r--r--  1 root root 196722688 Sep  4 10:36 Kismet-20190904-10-18-58-1.kismet

Installed on Kali per instructions at https://www.kismetwireless.net/docs/readme/kismetdb_to_pcap/ which references the kismetwireless.net git repo as opposed to this one.

Need to improve docs on included scripts.

at a minimum, packets.py doesn't support new columns from v7 and v8

v7 added column "datarate"
v8 added columns "hash" and "packetid"

i added an entry for each value in column_reference of packets.py and seems to work, not sure if more is needed

edit: i'll try to submit a pull request with some fixes

Is there a way to attach to a running kismet session? I know I could try to open the .kismet file it generates, but I was wondering if there was a way to use the kismetdb against the kismet session like how the REST API's work?

I received the message exported 0 devices after running kismet_log_to_xml. I confirmed the .kismet file did have device records.

I added more verbose error logging to the script and turns out there's an issue dereferencing the lon/lat data. It looks like Kismet has changed the data structure to a geopoint instead of individual lon/lat fields. Here's an example:

"kismet.common.location.avg_loc": {
  "kismet.common.location.time_usec": 0,
  "kismet.common.location.time_sec": 0,
  "kismet.common.location.fix": 0,
  "kismet.common.location.alt": 0,
  "kismet.common.location.geopoint": [0, 0]
}

I captured this data on Kismet 2020-12-R3.

I have a python script that pulls out data from a generated kismet file so that the information is presented in a summarised format. This kismet file is generated when you stop running kismet and has an extension of .kismet. This kismet file is actually a SQLite database underneath. However I have an issue whereby when I try to run my python script on a kismet file generated on newer versions of kismet I get quite a few errors, whereas If I run on an older version of kismet I was using before it works fine. I should also point out that I updated all my other libraries and software on my system too. My python script makes use of the kismetdb python wrapper library and a few others to accomplish this extraction of SQLite data.

I have done some investigating and tested a few things. Firstly I checked the version of SQLite that I had used before and what I had now. The one I was using before was version SQLite 3027002 and the one I’m using now is SQLite 3038002. I tested both versions of SQLite with the new version of kismet and it made no difference.
Next I tested different versions of python. My original python version was python 3.7.3 and my new one is now python 3.9.2. I ran my script with both python versions on the new kismet version. On python 3.7.3 I got a different error that says KeyError: 8. Whereas with python 3.9.2 I got a UnicodeDecodeError which I've posted the full error log after my code snippet.

The new kismet version I’m using now is 2022-01-R3 and the one I was using before was 2020-12-R3. Looking at the changelog didn’t really give me an information that I found useful. I also looked at the version of the actual kismet db_versions on both versions of kismet. The newer kismet uses version 8 whereas the older kismet is using version 6 and according to this information version 8 introduces the hash and packetid attributes to the packets table within the generated kismet file.

This is my python script code, I’ve commented out most of it so I can narrow down what’s causing the error on the new kismet version. I’ve managed to determine that it’s when I call the get_all() function that the error occurs. When I run this code with the old kismet version the print(KIS_DEVICES.get_all(**query_args)) outputs all of the devices that kismet detected, which corresponds to the devices table in the generated kismet SQLite file, which is too long to show. But that’s what I’m trying to do but with the new kismet version that uses kismetdb version 8.

import json, sys, kismetdb
from datetime import datetime

# Check if KismetDB is Specified as an Argument
if not len(sys.argv) == 2:
	print("[!] No KismetDB Specified")
	sys.exit(0)

# Set Input and Output Files
KIS_IN = sys.argv[1]
KIS_OUT = "%ssummary" % (KIS_IN[:-6])

query_args = {}
# Get Kismet Devices from DB
KIS_DEVICES = kismetdb.Devices(KIS_IN)
print(KIS_DEVICES.get_all(**query_args))

KIS_DB = [row["device"] for row in KIS_DEVICES.get_all(**query_args)]

sys.exit(0)

This is the error I get when I try to run that same code with the newer kismet version:

sudo python3 KismetDB_to_Summary\ copy.py new\ pi\ build/Kismet-20220411-23-01-06-1.kismet 
Traceback (most recent call last):
  File "/Users/user/Desktop/Apolloo /KismetDB_to_Summary copy.py", line 53, in <module>
    print(KIS_DEVICES.get_all(**query_args))
  File "/usr/local/lib/python3.9/site-packages/kismetdb/base_interface.py", line 155, in get_all
    return self.get_rows(self.column_names, sql, replacements)
  File "/usr/local/lib/python3.9/site-packages/kismetdb/base_interface.py", line 325, in get_rows
    for row in cur.fetchall():
  File "/usr/local/lib/python3.9/site-packages/kismetdb/utility.py", line 473, in device_field_parser
    retval = json.dumps(json.loads(device))
  File "/usr/local/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/__init__.py", line 341, in loads
    s = s.decode(detect_encoding(s), 'surrogatepass')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf5 in position 1286: invalid start byte

I’ve tried looking through the library code that the errors point to but most of it goes over my head and I’m really not sure what could be causing the errors. I have a feeling it’s something to do with the fact that the newer version of kismetdb (version 8) introduces new attributes in the packets table, but I’m not actually sure if that what the issue is.

Any help would be appeciated as I really want to use a newer version of kismet but still be able to extract and filter the generated SQLite data into my summary output.

Whenever I run kismet_log_to_pcap --in my_rowless_file.kismet --out output.pcap, the output file is not actually created.

This is likely currently intentional, but I'm not sure that is the right default behavior to have. This was causing issues in my script where I was trying to read data from a file I expected to exist, even if the file was just empty. I've implemented a workaround for the time being, but I really think a solution like this should be the library's default:

filename = await self.convert_kismet_to_pcap(kismet_in=filename)
with open(csv_filename) as read_obj:
    extra_info_kismet_df = pd.read_csv(read_obj, low_memory=False, delimiter='\t')

# Handle cases where there is no generated file (because the Kismet file has no rows)
if not os.path.isfile(filename):
    with open(filename, "w") as empty_file:
        empty_file.write("")

The output file of kismet_log_devices_to_json is a correct json file but it is a list of strings, which is not usefull.

It seems each object of the list has been converted to a string by out of placed quote at the start and end of each line of the file, and all the quotes inside the "object" are escaped with \.

For example:

[
"{<object1>}",
"{<object2>}",
"{<object3>}"
]

To get a usefull json file, one can use:

sed -e 's/\\"/"/g' -e 's/ "{/ {/g' -e 's/}",$/},/g' -e 's/}"$/}/g'  broken.json > fixed.json

The correct output should have been:

[
{<object1>},
{<object2>},
{<object3>}
]

where each object has quote not escaped.

Tested with a KismetDB version: 6

Improve timestamp handling:

Add a synthetic column for handling timestamps. This will add a column to output which will be a python datetime object (composed from ts_sec and ts_usec).

Provide functionality to make time-related queries using a Python datetime object instead of having to reduce query args to two timestamp-related fields.

Add the ability to query using ISO 8601-formatted timestamp. Prioritize so that if multiple timestamps are provided with kwargs, the priority (in order of importance) should be:

1. ts_sec, ts_usec
2. datetime object
3. ISO 8601