marin-m / pbtk Goto Github PK

View Code? Open in Web Editor NEW

1.3K 1.3K 164.0 8.03 MB

A toolset for reverse engineering and fuzzing Protobuf-based apps

License: GNU General Public License v3.0

Python 100.00%

decompiler fuzzing protobuf python reverse-engineering

pbtk's People

Contributors

Stargazers

Watchers

Forkers

terry2012 h0wl mainvolume-hq ycmint wflk minkione ksmaheshkumar techlord-rce redragonvn dongaxis ssdtfarm jancajthaml jancajthaml-javascript srgraham cheaphunter alexxnica nani1337 pdkyll pombredanne ezhangle ccsysadmin cherscarlett csxqli zbx91 mitakeck hax0rg1rl bingwin paynejacob pehelwan spacepro exiahan agametov gipsyd victorhdamian zhencang phpdude qsdj yuknight vistone sasqwatch ppmim slooppe ouerhub raimundojimenez starrify donfanning munelear f0829 hello-xbugs delthas i5anoff xtiankisutsa asarasetheru91 rajivraj fishso agent00049 allinonecyberteamindia seabreg yetanotherpeng keyman9848 angrykobe killvxk gorpo fengjixuchui 5l1v3r1 timoilya hyww leepapa daydayup40 throne6g zjwhy kingking888 fathui hanger23 musicode1 shadowce pbxdev tienquanutc sahwar anylayer nhattm3006 brinedfish reverse-man maodijim netmaxt3r ksw9722 zhuangchaoxi luodaoyi fakeshinoda jellykkk zzhichen jeepperscreeppers icgp qipeng090 heckerstone hyglvy zyzhen shuixi2013 wuyadie zbx911

pbtk's Issues

Please correct the magic comment # --coding: utf-8 -.-

Hi, many thanks for your code!
Just a note about the magic comment your have in your code, I see that you use:

#-*- encoding: Utf-8 -*-

but my editor highligth it as a mistake. It should be:

# -*- coding: utf-8 -*-

Many sources also state that if the code is Python3, the magic quote above is not longer required, because assumed per default.

Please include a copy of the license in the repo root

README states the license used as GPLv3. If you include a copy the license in your repository as the file LICENSE, GitHub will be able to automatically tag your project with the license used. This makes users happy :)

Error on start of the program

This is what I get, any suggestions? All the dependencies are installed

kali@kali:~/pbtk$ ./gui.py Traceback (most recent call last): File "./gui.py", line 20, in <module> from views.fuzzer import ProtobufItem, ProtocolDataItem File "/home/kali/pbtk/views/fuzzer.py", line 5, in <module> from PyQt5.QtWebEngineWidgets import QWebEngineView ImportError: libQt5WebEngineWidgets.so.5: cannot open shared object file: No such file or directory

dex2jar creates case-sensitive files on windows resulting in missing class files

dex2jar creates case-sensitive class filenames for example it generates "mL2.class" while in the same apk another class with the name "Ml2" will result in the override of the first file thus the proto search is done on less classes then there actually is

Unexpected end-group tag.

Hey, could you tell me what causes the following error?

Traceback (most recent call last):
  File ".\gui.py", line 328, in launch_fuzzer
    self.get_params = self.transport.load_sample(sample, self.pb_request)
  File "C:\Users\Ghxst\Downloads\pbtk-master\pbtk-master\utils\transports.py", line 30, in load_sample
    pb_msg.ParseFromString(bytes.fromhex(sample))
  File "C:\Program Files\Python38\lib\site-packages\google\protobuf\message.py", line 199, in ParseFromString
    return self.MergeFromString(serialized)
  File "C:\Program Files\Python38\lib\site-packages\google\protobuf\internal\python_message.py", line 1148, in MergeFromString
    raise message_mod.DecodeError('Unexpected end-group tag.')
google.protobuf.message.DecodeError: Unexpected end-group tag.

It's possible I'm doing something wrong when fuzzing, I am taking the request from HexView in Fiddler and using that as the raw POST data.

Use pacaur instead of yaourt

Yaourt is dangerous, as it executes the PKGBUILD instead of letting the user view it before executing. Pacaur should be used instead.

References:
Arch Wiki
Reddit explanation

Is also the Google Maps information is in Protobuf

Google Maps has a very strange format to convey the information to the customer.
This is a lot of nested arrays, which contain a lot of NULL values.
It looks like this:

{"c":0,"d":")]}'\n[[\"white house\",[[null,null,null,null,null,null,null,null,\"L1m9X6rhA5CKlwSauKDQBw\",\"0ahUKEwjqtNnQ75vtAhUQxYUKHRocCHoQmBkIAigA\",[\"white house\",0]\n]\n]\n,null,1,null,[\"1606244655560\"]\n,null,null,null,null,null,0]\n,[[5.345248603008451E7,-20.522437649999997,35.14462475]\n,[0.0,0.0,0.0]\n,[1024,768]\n,13.1]\n,null,null,null,null,null,\"L1m9X6rhA5CKlwSauKDQBw\",null,[]\n,[[[\"m\",[3,0,0]\n,8,[534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534247556,534247556,534255710,534255710,534254714,534254714,534227504,534240512,534240512,534240512,534253622,534254714]\n]\n,[\"m\",[2,0,0]\n,4,[534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255614,534255710,534255710,534255710]\n]\n,[\"m\",[4,3,1]\n,8,[534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710,534255710]\n]\n]\n]\n,null,null,null,null,null,[[[2,\"spotlight\",null,null,null,null,null,[null,null,null,null,null,null,null,null,null,null,null,[\"white house\",[\"112308031950842969741\",\"\"]\n,[[null,null,\"0x0:0xb3e6e80a69d78dc1\"]\n,[null,null,\"0x0:0x67ae4478a534c4d6\"]\n,[null,null,\"0x0:0xc83d885f202b2ecf\"]\n,[null,null,\"0x0:0x424daceea73a1f68\"]\n,[null,null,\"0x0:0xe13ae4daf2eec66b\"]\n,[null,null,\"0x0:0xf5a81e78fa2e0d78\"]\n,[null,null,\"0x0:0x53fd99e4f5b8c7f\"]\n,[null,null,\"0x0:0x1d89849223d7f90d\"]\n,[null,null,\"0x0:0xc5edb6b6c28d4154\"]\n,[null,null,\"0x0:0x62ff5e6af80250a0\"]\n,[null,null,\"0x0:0x30e1c1aa4416e2c0\"]\n,[null,null,\"0x0:0x713dc1909209bc1a\"]\n,[null,null,\"0x0:0x50d2a43e03c42a73\"]\n,[null,null,\"0x0:0x5860d8d32225a85\"]\n,[null,null,\"0x0:0xd0b2a5a1da50756e\"]\n,[null,null,\"0x0:0x715969d86d0b76bf\"]\n,[null,null,\"0x0:0xe48589933a9c28e1\"]\n,[null,null,\"0x0:0x30162385ca0fbd7d\"]\n,[null,null,\"0x0:0x95ca0169317472f2\"]\n,[null,null,\"0x0:0x3e49841d60cf0fec\"]\n]\n,null,null,null,null,null,null,0]\n,[null,\"a\",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,[null,null,null,null,0,0,null,null,1,null,null,1,null,null,null,0,null,null,null,1,1]\n,null,null,null,[null,null,null,null,null,2,3,2]\n]\n,null,null,null,null,null,[12,14,29,30,61,70]\n,null,null,null,null,null,[null,null,null,2]\n]\n]\n]\n,[[52]\n]\n]\n,null,null,null,null,[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,\"IAE\\u003d\"]\n,null,null,null,null,null,[4]\n,null,null,null,null,null,null,null,\"Q2dBd0Fn\",null,null,0,[\"white house\"]\n]\n","e":"L1m9X6rhA5CKlwSauKDQBw","p":true,"u":"https:\/\/www.google.com\/search?tbm=map\u0026hl=en\u0026q=white%20house\u0026tch=1\u0026ech=1"}/*""*/

Sample link: https://www.google.com/search?tbm=map&hl=en&q=white%20house&tch=1&ech=1

Is it also Protobuf?
And how do you decipher it?

FileNotFoundError classes-dex2jar.jar

PS C:\Users\misti\Desktop\Begin\PrivacySecurityReversProtection\revers\pbtk-1.0.5> ./gui.py
C:\Users\misti\Desktop\Begin\PrivacySecurityReversProtection\revers\pbtk-1.0.5\gui.py:442: DeprecationWarning: an integer is required (got type float). Implicit conversion to integers using int is deprecated, and may be
removed in a future version of Python.
view.move((resolution.width() / 2) - (view.frameSize().width() / 2),
Traceback (most recent call last):
File "C:\Users\misti\Desktop\Begin\PrivacySecurityReversProtection\revers\pbtk-1.0.5\gui.py", line 462, in run
for name, contents in self.extractor'func':
File "C:\Users\misti\Desktop\Begin\PrivacySecurityReversProtection\revers\pbtk-1.0.5\extractors\jar_extract.py", line 51, in handle_jar
with JarWrapper(path) as jar:
File "C:\Users\misti\Desktop\Begin\PrivacySecurityReversProtection\revers\pbtk-1.0.5\utils\java_wrapper.py", line 25, in init
self.handle_file(fname)
File "C:\Users\misti\Desktop\Begin\PrivacySecurityReversProtection\revers\pbtk-1.0.5\utils\java_wrapper.py", line 43, in handle_file
self.handle_file(self.name + '/' + cls)
File "C:\Users\misti\Desktop\Begin\PrivacySecurityReversProtection\revers\pbtk-1.0.5\utils\java_wrapper.py", line 34, in handle_file
with ZipFile(fname) as jar:
File "C:\Python\Python38\lib\zipfile.py", line 1251, in init
self.fp = io.open(file, filemode)
FileNotFoundError: [Errno 2] No such file or directory: 'C:\Users\misti\AppData\Local\Temp\tmpkxi_mopf/classes-dex2jar.jar' ****

I don't know how to solve this problem

File "/home/kali/pbtk/extractors/from_binary.py", line 8, in
from utils.common import register_extractor, extractor_main
ModuleNotFoundError: No module named 'utils.common

Can not extract proto file

File "gui.py", line 462, in run
for name, contents in self.extractor'func':
File "c:\pbkt\pbtk\extractors\jar_extract.py", line 51, in handle_jar
with JarWrapper(path) as jar:
File "c:\pbkt\pbtk\utils\java_wrapper.py", line 25, in init
self.handle_file(fname)
File "c:\pbkt\pbtk\utils\java_wrapper.py", line 43, in handle_file
self.handle_file(self.name + '/' + cls)
File "c:\pbkt\pbtk\utils\java_wrapper.py", line 35, in handle_file
jar.extractall(self.name)
File "c:\users\user\appdata\local\programs\python\python37\lib\zipfile.py", line 1636, in extractall
self._extract_member(zipinfo, path, pwd)
File "c:\users\user\appdata\local\programs\python\python37\lib\zipfile.py", line 1690, in _extract_member
open(targetpath, "wb") as target:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\Users\user\AppData\Local\Temp\tmprh1yw3mr\prn.class'

ValueError: VectorTown.proto:1405:16: Expected field name.

Last Wednesday I still can use pbtk to fiddling with google maps's protobuf (step 1 and step 3)
But then on Friday I got ValueError on most of endpoints on step 3:
Only two of them still works

/maps/@{coods}/data={data}
/maps/dir/{dir1}/{dir2}/@{cỏods}/data={data}

My error with /s

Traceback (most recent call last):
  File "gui.py", line 310, in launch_fuzzer
    self.pb_request = dict(self.pb_request)[data['request']['proto_msg']]()
  File "/home/thanhlv/workspace/decode_pb/pbtk/utils/common.py", line 179, in load_proto_msgs
    raise ValueError(cmd.stderr.decode('utf8'))
ValueError: Search.proto:1884:62: Field numbers must be positive integers.
Search.proto:636:18: "undefined" is not defined.
Search.proto:637:18: "undefined" is not defined.
Search.proto:638:29: Field number 6 has already been used in "Search.E" by field "f".
Search.proto:639:18: "undefined" is not defined.
Search.proto:640:18: Field number 8 has already been used in "Search.E" by field "h".
Search.proto:641:20: Field number 9 has already been used in "Search.E" by field "i".
Search.proto:828:22: "undefined" is not defined.
Search.proto:1879:50: "undefined" is not defined.
Search.proto:1880:50: "undefined" is not defined.
Search.proto:1883:41: "undefined" is not defined.
Search.proto:2011:14: "undefined" is not defined.
Search.proto:2013:5: "undefined" is not defined.
Aborted (core dumped)

and /maps/vt

Traceback (most recent call last):
  File "gui.py", line 310, in launch_fuzzer
    self.pb_request = dict(self.pb_request)[data['request']['proto_msg']]()
  File "/home/thanhlv/workspace/decode_pb/pbtk/utils/common.py", line 179, in load_proto_msgs
    raise ValueError(cmd.stderr.decode('utf8'))
ValueError: VectorTown.proto:1405:16: Expected field name.
Aborted (core dumped)

Is it only my problem or did Google recently change the response format ?

Thanks for your awesome tool, it helps me a lot !

UnicodeDecodeError

Traceback (most recent call last):
File "C:\main\soft\protobuf\pbtk-master\gui.py", line 462, in run
for name, contents in self.extractor'func':
File "C:\main\soft\protobuf\pbtk-master\extractors\jar_extract.py", line 51, in handle_jar
with JarWrapper(path) as jar:
File "C:\main\soft\protobuf\pbtk-master\utils\java_wrapper.py", line 25, in init
self.handle_file(fname)
File "C:\main\soft\protobuf\pbtk-master\utils\java_wrapper.py", line 46, in handle_file
self.bonus_protos[cls] = jar.read(cls).decode('utf8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x86 in position 5: invalid start byte

APK: com.google.android.gm_2023.04.02.523594694.release_63820521.apk (https://trashbox.ru/files30/1792705/com.google.android.gm_2023.04.02.523594694.release_63820521.apk/)

Commercial development

Hello,
I'm interesting in commercial development,
how can I contact you?

google-chrome binary is missing

I have install all the tools

Installing collected packages: PyQt5-sip, PyQt5-Qt, PyQtWebEngine-Qt, PyQt5, pyqtwebengine
Successfully installed PyQt5-5.15.3 PyQt5-Qt-5.15.2 PyQt5-sip-12.8.1 PyQtWebEngine-Qt-5.15.2 pyqtwebengine-5.15.3

but i still getting ImportError: You are missing the binary "google-chrome" for this.

install on Debian

FYI, I had to run the following command to get the program to run.

pip3 install PyQtWebEngine

Might consider revising README.md

Nothing found in YouTube app

You can download the apk from:
https://m.apkpure.com/youtube/com.google.android.youtube/download?from=details
and then, extract the apk from xapk file.

The output:

python3 extractors/jar_extract.py com.google.android.youtube.apk output
In aakr:

In amxc:

In anad:

In amyn:

In zpn:

In zwk:

IndexError: tuple index out of range

when i parse a apk file, error in jar_extract.py show IndexError: tuple index out of range

`use_namer = all(len(i[4]) < 3 or i[4].endswith('_fld') for i in fields.values())

all_vars = [field[4] for field in fields.values()]`

Can't decompile APK when one of the .class files are named "aux" on Windows

Windows does not allow any files to be called aux. This is a problem when the disassembled version of the APK I'm trying to use has a file named aux.class. This causes the following error to occur:

FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\...\\AppData\\Temp\\tmp...\\aux.class

Is there anything I can do to bypass this?

Using decompiled data as intput

Hi,

I have already decompiled a jar binary using fernflower and other tools because it fails with JAD. Is it somehow possible to use these decompiled files as input for .proto generation?

Thanks.

not support cpu_64

OSError: [Errno 86] Bad CPU type in executable: '/Users/zx/pbtk/utils/external/jad/jad_osx'

jadx good

Core dumps when fuzzing

In "Step 3" after choosing an endpoint, the app crashes.

Here's a backtrace from the core dump:

#0  0x0000000000000000 in ?? ()
#1  0x00007fe10ddbab8f in QOpenGLContext::makeCurrent(QSurface*) ()
   from /usr/local/lib/python3.5/dist-packages/PyQt5/Qt/lib/libQt5Gui.so.5
#2  0x00007fe0fe61b5e3 in ?? () from /usr/local/lib/python3.5/dist-packages/PyQt5/Qt/lib/libQt5QuickWidgets.so.5
#3  0x00007fe0fe61bd70 in QQuickWidget::showEvent(QShowEvent*) ()
   from /usr/local/lib/python3.5/dist-packages/PyQt5/Qt/lib/libQt5QuickWidgets.so.5
#4  0x00007fe1068e4fd9 in ?? () from /usr/local/lib/python3.5/dist-packages/PyQt5/Qt/lib/libQt5WebEngineWidgets.so.5
#5  0x00007fe10e5929b8 in QWidget::event(QEvent*) ()
   from /usr/local/lib/python3.5/dist-packages/PyQt5/Qt/lib/libQt5Widgets.so.5
#6  0x00007fe1068e520b in ?? () from /usr/local/lib/python3.5/dist-packages/PyQt5/Qt/lib/libQt5WebEngineWidgets.so.5
#7  0x00007fe10e5565ec in QApplicationPrivate::notify_helper(QObject*, QEvent*) ()
   from /usr/local/lib/python3.5/dist-packages/PyQt5/Qt/lib/libQt5Widgets.so.5
#8  0x00007fe10e55da17 in QApplication::notify(QObject*, QEvent*) ()
   from /usr/local/lib/python3.5/dist-packages/PyQt5/Qt/lib/libQt5Widgets.so.5
#9  0x00007fe10ef9f24e in sipQApplication::notify(QObject*, QEvent*) ()
   from /usr/local/lib/python3.5/dist-packages/PyQt5/QtWidgets.so
#10 0x00007fe10d7922b8 in QCoreApplication::notifyInternal2(QObject*, QEvent*) ()
   from /usr/local/lib/python3.5/dist-packages/PyQt5/Qt/lib/libQt5Core.so.5
#11 0x00007fe10e58f963 in QWidgetPrivate::show_helper() ()
   from /usr/local/lib/python3.5/dist-packages/PyQt5/Qt/lib/libQt5Widgets.so.5
#12 0x00007fe10e592505 in QWidget::setVisible(bool) ()
   from /usr/local/lib/python3.5/dist-packages/PyQt5/Qt/lib/libQt5Widgets.so.5
#13 0x00007fe10e578147 in QStackedLayout::setCurrentIndex(int) ()
   from /usr/local/lib/python3.5/dist-packages/PyQt5/Qt/lib/libQt5Widgets.so.5
#14 0x00007fe10e5787ca in QStackedLayout::insertWidget(int, QWidget*) ()
   from /usr/local/lib/python3.5/dist-packages/PyQt5/Qt/lib/libQt5Widgets.so.5
#15 0x00007fe10e578815 in QStackedLayout::addItem(QLayoutItem*) ()
   from /usr/local/lib/python3.5/dist-packages/PyQt5/Qt/lib/libQt5Widgets.so.5
#16 0x00007fe1068e48ba in ?? () from /usr/local/lib/python3.5/dist-packages/PyQt5/Qt/lib/libQt5WebEngineWidgets.so.5
#17 0x00007fe1068e2bd4 in ?? () from /usr/local/lib/python3.5/dist-packages/PyQt5/Qt/lib/libQt5WebEngineWidgets.so.5
#18 0x00007fe1068e3826 in QWebEngineView::page() const ()
   from /usr/local/lib/python3.5/dist-packages/PyQt5/Qt/lib/libQt5WebEngineWidgets.so.5
#19 0x00007fe1068e3ff1 in QWebEngineView::showEvent(QShowEvent*) ()
   from /usr/local/lib/python3.5/dist-packages/PyQt5/Qt/lib/libQt5WebEngineWidgets.so.5
#20 0x00007fe106b18a83 in sipQWebEngineView::showEvent(QShowEvent*) ()
   from /usr/local/lib/python3.5/dist-packages/PyQt5/QtWebEngineWidgets.so
#21 0x00007fe10e5929b8 in QWidget::event(QEvent*) ()
   from /usr/local/lib/python3.5/dist-packages/PyQt5/Qt/lib/libQt5Widgets.so.5
#22 0x00007fe106b172f3 in sipQWebEngineView::event(QEvent*) ()
   from /usr/local/lib/python3.5/dist-packages/PyQt5/QtWebEngineWidgets.so
#23 0x00007fe10e5565ec in QApplicationPrivate::notify_helper(QObject*, QEvent*) ()
   from /usr/local/lib/python3.5/dist-packages/PyQt5/Qt/lib/libQt5Widgets.so.5
#24 0x00007fe10e55da17 in QApplication::notify(QObject*, QEvent*) ()
   from /usr/local/lib/python3.5/dist-packages/PyQt5/Qt/lib/libQt5Widgets.so.5
#25 0x00007fe10ef9f24e in sipQApplication::notify(QObject*, QEvent*) ()
   from /usr/local/lib/python3.5/dist-packages/PyQt5/QtWidgets.so

Is there perhaps some library or Python module missing?

Support for com.squareup.wire

It would be great, if pbtk was able to extract proto-files from java-source generated for square's wire.

no joy; I know this app uses protobuf but nothing is apparently found

This is all I see when running an extract. I'm I doing something wrong?

./extractors/jar_extract.py com.starlink.mobile.apk

In com.google.android.gms.internal.measurement.zzjn:

Error extracting from jar or apk

I'm trying to extract proto files from an APK, but it fails when extracting content from the jar:

In vfo:
Traceback (most recent call last):
  File "./gui.py", line 462, in run
    for name, contents in self.extractor['func'](input_):
  File "/home/jack/Desktop/phone/pbtk/extractors/jar_extract.py", line 236, in handle_jar
    extract_lite(jar, cls, enums, gen_classes_nodollar, codedinputstream, codedoutputstream, map_entry_cls, out_additional_cls,
  File "/home/jack/Desktop/phone/pbtk/extractors/jar_extract.py", line 326, in extract_lite
    ftype = {0: 'int32', 1: 'fixed64', 2: 'bytes', 3: 'group', 5: 'fixed32'}[lazy_tag & 7]
KeyError: 4

I checked the code and the lazy_tag I'm getting when this error occurs is 68, so it's trying to find key 4 in the ftype dict, but it doesn't exist. Is this intentional?

Format converter for JsProtoUrl?

Hi,

Wondering if you have a converter to go from the text proto url thingy that google uses, into an object in memory or JSON?

Ive been trying to do this myself, but are having some difficulty with the nested fields.

Good job with the XSS by the way. = )