log2timeline / l2tscaffolder Goto Github PK

View Code? Open in Web Editor NEW

6.0 6.0 9.0 659 KB

Scaffolders for bootstrapping development of open source forensics tools.

Home Page: http://l2tscaffolder.readthedocs.io/

License: Apache License 2.0

Python 89.68% Jinja 10.32%

developer-tools development-workflow scaffolder

l2tscaffolder's People

Contributors

Stargazers

Watchers

Forkers

onager kiddinn aguilajesus berggren wajihyassine roshanmaskey jkppr studiawan sydp

l2tscaffolder's Issues

Remove old code from the project

There are few steps for this to be done:

~~Complete the backend for v2 of the scaffolder~~
Complete the CLI for v2 of scaffolder: (#40)
~~Delete / remove old tests: #36~~
Delete all the old code: client and diff

Add a plugin for Windows Registry plugin creation.

Plugin needed to create Windows registry plugins in plaso.

Timesketch crashes with current "ts_sketch_analyzer.jinja2" template

Describe the bug

When using l2t_scaffolder.py to create a new timesketch sketch analyzer the template ts_sketch_analyzer.jinja2 is used. However, the argument for the class in line 9 interface.BaseSketchAnalyzer seems to be deprecated and renamed to interface.BaseAnalyzer. Trying to run timesketch with the analyzer file generated by the l2t_scaffolder.py tool does crash with AttributeError: module 'timesketch.lib.analyzers.interface' has no attribute 'BaseSketchAnalyzer'.

To Reproduce

Steps to reproduce the behavior:

$ l2t_scaffolder.py 
   == Starting the scaffolder ==
Gathering required information.

Available definitions: 
  [0] plaso
  [1] timesketch
  [2] turbinia
Definition choice: 1
timesketch chosen.

Path to the project root: .
Path [.] set as the project path.

Name of the module to be generated. This can be something like "foobar sqlite" or "event analytics".

This will be used for class name generation and file name prefixes.
Module Name: screenshot
About to create a new feature branch to store newly generated code.
ERROR:root:Running: "git show-ref --verify --quiet refs/heads/"screenshot"" failed with error: .
Creating feature branch: screenshot inside .
Switching to feature branch screenshot

Available scaffolders for timesketch:
  [0] index_analyzer
  [1] sketch_analyzer
Scaffolder choice: 1
Ready to generate files? [Y/n]: 
File: ./timesketch/lib/analyzers/screenshot.py written to disk.
File: ./timesketch/lib/analyzers/screenshot_test.py written to disk.
File: ./timesketch/lib/analyzers/__init__.py written to disk.

Error
As soon as I try to run timesketch after creating the new analyzer template the timesketch container throws the following error:

[2022-06-07 15:26:01 +0000] [156] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/gunicorn/arbiter.py", line 586, in spawn_worker
    worker.init_process()
  File "/usr/local/lib/python3.8/dist-packages/gunicorn/workers/base.py", line 135, in init_process
    self.load_wsgi()
  File "/usr/local/lib/python3.8/dist-packages/gunicorn/workers/base.py", line 144, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/usr/local/lib/python3.8/dist-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/usr/local/lib/python3.8/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load
    return self.load_wsgiapp()
  File "/usr/local/lib/python3.8/dist-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/local/lib/python3.8/dist-packages/gunicorn/util.py", line 350, in import_app
    __import__(module)
  File "/usr/local/src/timesketch/timesketch/wsgi.py", line 39, in <module>
    from timesketch.app import configure_logger
  File "/usr/local/src/timesketch/timesketch/app.py", line 32, in <module>
    from timesketch.api.v1.routes import API_ROUTES as V1_API_ROUTES
  File "/usr/local/src/timesketch/timesketch/api/v1/routes.py", line 18, in <module>
    from .resources.aggregation import AggregationGroupResource
  File "/usr/local/src/timesketch/timesketch/api/v1/resources/aggregation.py", line 30, in <module>
    from timesketch.api.v1 import utils
  File "/usr/local/src/timesketch/timesketch/api/v1/utils.py", line 29, in <module>
    from timesketch.lib import ontology
  File "/usr/local/src/timesketch/timesketch/lib/ontology.py", line 17, in <module>
    from timesketch.lib.analyzers import interface
  File "/usr/local/src/timesketch/timesketch/lib/analyzers/__init__.py", line 29, in <module>
    from timesketch.lib.analyzers import screenshot
  File "/usr/local/src/timesketch/timesketch/lib/analyzers/screenshot.py", line 9, in <module>
    class ScreenshotSketchPlugin(interface.BaseSketchAnalyzer):
AttributeError: module 'timesketch.lib.analyzers.interface' has no attribute 'BaseSketchAnalyzer'
[2022-06-07 15:26:01 +0000] [156] [INFO] Worker exiting (pid: 156)
[2022-06-07 15:26:02 +0000] [115] [INFO] Shutting down: Master
[2022-06-07 15:26:02 +0000] [115] [INFO] Reason: Worker failed to boot.

Expected behavior

When using the l2t_scaffolder.py tool to create a new analyzer template I would expect it not to crash the timesketch server.

Possible solution
Change ts_sketch_analyzer.jinja2 line 9 to class {{ class_name }}SketchPlugin(interface.BaseAnalyzer):

Desktop (please complete the following information):

OS: Debian
Version 5.17.6-1rodete1 (2022-05-12)

**l2tscaffolder Version``

$ pip3 list | grep l2tscaffolder
l2tscaffolder               20200511

Disable Python2.7 in Travis

We're not targeting compatibility

Fix issues in templates

missing import statement
misplaced , before'

assertIsEqual doesn't exist

Describe the bug

jinja template for tests should call assertEqual instead of AssertIsEqual

Create a new definition for timesketch and add a timesketch analysis scaffolder

Add a timesketch scaffolder for analysis plugins in timesketch.

l2tscaffolder fails to create parser

l2tscaffolder has syntax error when creating parser
ERROR:root:Syntax error while attempting to generate parser, error message: invalid syntax (, line 30)

Did some testing and did a generic SQL "SELECT * from table;" and "SELECT * from table"

A clear and concise description of what the bug is.

To Reproduce

Steps to reproduce the behavior:

What definition was selected
2 - plaso
Answers to questions

$ l2t_scaffolder.py
== Starting the scaffolder ==
Gathering required information.

Available definitions:
[0] timesketch
[1] turbinia
[2] plaso
Definition choice: 2
plaso chosen.

Path to the project root: /home/forensics/plaso
Path [/home/forensics/plaso] set as the project path.

Name of the module to be generated. This can be something like "foobar sqlite" or "event analytics".

This will be used for class name generation and file name prefixes.
Module Name: android_googlemail
About to create a new feature branch to store newly generated code.
ERROR:root:Running: "git show-ref --verify --quiet refs/heads/"android_googlemail"" failed with error: .
Creating feature branch: android_googlemail inside /home/forensics/plaso
Switching to feature branch android_googlemail

Available scaffolders for plaso:
[0] sqlite
Scaffolder choice: 0

Define the name of the callback function (key) that will be
called for every row returned from the SQL query (value).
The plugin will execute the SQL query and call the callback
once for each resulting row. The name of the function should
follow style guide and be descriptive. An example of that is
a SQL statement that fetches bookmarks, the key name should be
Bookmark, or if the SQL statement collects GPS coordinates
it could be called Location.
Callback function name [#1]: Messages
SQL Statement [#1]: SELECT * from messages
More entries? [Y/n]: n

List of required tables
Value to add [#1]: messages
Add more values? [Y/n]: n

Absolute or relative path to the file that will be used for tests.
Value: /home/forensics/google_db/[email protected]
Ready to generate files? [Y/n]: y
File: /home/forensics/plaso/test_data/[email protected] written to disk.
ERROR:root:Syntax error while attempting to generate parser, error message: invalid syntax (, line 30)
File: /home/forensics/plaso/tests/parsers/sqlite_plugins/android_googlemail.py written to disk.
File: /home/forensics/plaso/plaso/formatters/android_googlemail.py written to disk.
File: /home/forensics/plaso/tests/formatters/android_googlemail.py written to disk.
File: /home/forensics/plaso/plaso/formatters/init.py written to disk.
File: /home/forensics/plaso/plaso/parsers/sqlite_plugins/init.py written to disk.

Error message, or output that was unexpected

"File: /home/forensics/plaso/plaso/parsers/sqlite_plugins/android_googlemail.py written to disk"

OS: Ubuntu 16.04

l2tscaffolder Version
l2tscaffolder==20190103

Update output code to match latest plaso

Update output code to match latest plaso version

Migrate codebase to l2tdevtools

Instead of being a separate project within the log2timeline org, migrate this to be part of l2tdevtools.

Since this is a development tool, it should just be part of that.

Add a plugin interface

Plugin interface

Add a plugin manager.

A plugin manager is required to register new plugins and provide easy mechanism for the CLI to discover and extract information to build CLI options from available plugins

Add a way to identify projects

This scaffolder project should expand to support other projects than plaso, initially plaso will the only supported project but it needs to be abstracted away so that other projects can be supported in the near future, eg. timesketch.

Add a SQlite plugin

Add a plugin that handles SQLite plugin generation for plaso.

Add end-to-end tests

Idea for how to do this:

Check out project to scaffold (Turbinia, Plaso etc.)
Run scaffolder on project
See if tests still pass for project to scaffold

We might need a way to pass answers to a scaffold non-interactively.

Additions to init files is not sorted, with a TODO in there and no newline character appended.

Describe the bug
The scaffolders produce new plugins/parsers/modules but when they are registered in init files they do not alphabetically sort them. Also the scaffolders don't add a newline character at the end, leaving some linters to complain.

To Reproduce

Generate files with the scaffolders, look at init.py additions.

Expected behavior

It would be better if the init files would be correct and wouldn't need any other additions/changes by devs, that is that the entries would be inserted into the correct order, etc.

look into yapf failing in tests on appveyor

look into yapf failing in tests on appveyor
https://ci.appveyor.com/project/joachimmetz/plasoscaffolder/build/4

Comment out all tests that are not working ATM in order for the tests to successfully run.

ATM the tests are all failing. This will be "fixed" by commenting all failed testing out with a TODO of this code being removed in future CLs.

pip3 install of l2tscaffolder is missing the ".style.ts.yapf" file

Describe the bug

When running l2t_scaffolder.py to create the setup for a new timesketch analyzer the tool crashes because it cannot find the .style.ts.yap style file.
It looks like installing l2tscaffolder via pip3 install l2tscaffolder does not place the .style.ts.yap file in the expected place ~/.local/lib/python3.9/site-packages/l2tscaffolder/.style.ts.yapf. After copying the file there manually from github, it works without issues.

To Reproduce

Steps to reproduce the behavior:

Make a fresh install via pip3 install l2tscaffolder
Run via l2t_scaffolder.py
Select [1] timesketch, provide . as path, select [1] sketch_analyzer
Error output:

$ cd ~/timesketch
$ l2t_scaffolder.py
   == Starting the scaffolder ==
Gathering required information.

Available definitions: 
  [0] plaso
  [1] timesketch
  [2] turbinia
Definition choice: 1
timesketch chosen.

Path to the project root: .
Path [.] set as the project path.

Name of the module to be generated. This can be something like "foobar sqlite" or "event analytics".

This will be used for class name generation and file name prefixes.
Module Name: screenshot_test
About to create a new feature branch to store newly generated code.
Switching to feature branch screenshot_test

Available scaffolders for timesketch:
  [0] index_analyzer
  [1] sketch_analyzer
Scaffolder choice: 1
Ready to generate files? [Y/n]:
Traceback (most recent call last):
  File "~/.local/bin/l2t_scaffolder.py", line 26, in <module>
    StartCLI()  # pylint: disable=no-value-for-parameter
  File "~/.local/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "~/.local/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "~/.local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "~/.local/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "~/.local/bin/l2t_scaffolder.py", line 22, in StartCLI
    cli.Start(definition)
  File "~/.local/lib/python3.9/site-packages/l2tscaffolder/frontend/frontend.py", line 375, in Start
    for file_path in scaffolder_engine.GenerateFiles():
  File "~/.local/lib/python3.9/site-packages/l2tscaffolder/lib/engine.py", line 74, in GenerateFiles
    for file_path, content in self._scaffolder.GenerateFiles():
  File "~/.local/lib/python3.9/site-packages/l2tscaffolder/scaffolders/timesketch.py", line 103, in GenerateFiles
    plugin_content = self._GeneratePlugin()
  File "~/.local/lib/python3.9/site-packages/l2tscaffolder/scaffolders/timesketch.py", line 54, in _GeneratePlugin
    return self._mapping_helper.RenderTemplate(
  File "~/.local/lib/python3.9/site-packages/l2tscaffolder/lib/mapping_helper.py", line 124, in RenderTemplate
    formatted = self.formatter.Format(template)[0]
  File "~/.local/lib/python3.9/site-packages/l2tscaffolder/lib/code_formatter.py", line 28, in Format
    return yapf_api.FormatCode(code, style_config=self.yapf_path)
  File "~/.local/lib/python3.9/site-packages/yapf/yapflib/yapf_api.py", line 124, in FormatCode
    style.SetGlobalStyle(style.CreateStyleFromConfig(style_config))
  File "~/.local/lib/python3.9/site-packages/yapf/yapflib/style.py", line 498, in CreateStyleFromConfig
    config = _CreateConfigParserFromConfigFile(style_config)
  File "~/.local/lib/python3.9/site-packages/yapf/yapflib/style.py", line 527, in _CreateConfigParserFromConfigFile
    raise StyleConfigError(
yapf.yapflib.style.StyleConfigError: "~/.local/lib/python3.9/site-packages/l2tscaffolder/.style.ts.yapf" is not a valid style or file path

Expected behavior

Instead of crashing I would expect l2tscaffolder to create the required files. Expected output:

[...]
Ready to generate files? [Y/n]: 
File: ./timesketch/lib/analyzers/screenshot_test_2.py written to disk.
File: ./timesketch/lib/analyzers/screenshot_test_2_test.py written to disk.
File: ./timesketch/lib/analyzers/__init__.py written to disk.

Desktop (please complete the following information):

OS: Debian
Version 5.17.6-1rodete1 (2022-05-12)

l2tscaffolder Version

$ pip3 list | grep l2tscaffolder
l2tscaffolder               20200511

timesketch project root not identified correctly

Describe the bug
running the scaffolder and entering the path to the repo results in "Path [.] does not lead to a valid project for timesketch. Do you want to try again? [Y/n]:"

presumably because this test fails:

if not os.path.isfile(os.path.join(root_path, 'timesketch.conf')):
return False
@

l2tscaffolder/l2tscaffolder/definitions/timesketch.py

Line 31 in 5a6e7dd

if not os.path.isfile(os.path.join(root_path, 'timesketch.conf')):

To Reproduce

Steps to reproduce the behavior:

What definition was selected
timesketch
Answers to questions

1
./ and absolute path to repository

Error message, or output that was unexpected
Path [.] does not lead to a valid project for timesketch. Do you want to try again? [Y/n]:

Expected behavior

Screenshots

If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS: Debian GNU/Linux rodete

Add an engine.

The manager is initiated by the frontend and takes in a plugin object.

The manager then takes care of accepting results from the frontend, handling all file operations and requesting data from the plugin that are necessary to generate all files and then finally generating the necessary files.

Migrate CI and coverage

Migrate documentation

Migrate documentation:

https://claudiasaxer.github.io/PlasoScaffolder/
http://plasoscaffolder.readthedocs.io/en/latest/ - make sure all maintainers have access

first run of timesketch scaffolder

while testing timesketch code generation PYTHONPATH=. python3 tools/l2t_scaffolder.py I encountered the following error[0], it did not affect the end results but it could be related to the use of relative paths.

Path [../timesketch] set as the project path.

[0] - Error:
ERROR:root:Running: "git checkout test_analizer" failed with error: error: pathspec 'test_analizer' did not match any file(s) known to git

add support for plaso dsv parser generation

For plaso support generating a DSV-based (delimiter separated) parser

Update CLI to work with new updates.

The CLI only works with the v1.0 of the scaffolder, need to upgrade the CLI to work with the new 2.0 version

Create initial version based on previous work

Create initial version based on previous work by Claudia:
https://github.com/ClaudiaSaxer/PlasoScaffolder

Add a turbinia job scaffolder

Add a scaffolder for turbinia jobs, needs two things:

~~Definition for turbinina: #51~~
The scaffolder itself: #56

Rename the github project l2t_scaffolder

Since this is not only for plaso development, rename the project into l2t_scaffolder.

Create a PR that renames plasoscaffolder to l2tscaffolder in every file and rename directory.
Rename the github project
Create a release
Push the new version to pypi.

This is all dependent on:

#40: New frontend
#42: Deleting/removing old code

These dependencies need to be checked in first.

ImportError: cannot import name 'eventdata' from 'plaso.lib'

Describe the bug

I develop an Sqlite plugin using l2tscaffolder. The scaffolder successfully built the plugin. After that, I rebuilt the plaso and got an error ImportError: cannot import name 'eventdata' from 'plaso.lib'

To Reproduce

Steps to reproduce the behavior:

What definition was selected

$ l2t_scaffolder.py 
   == Starting the scaffolder ==
Gathering required information.

Available definitions: 
  [0] plaso
  [1] timesketch
  [2] turbinia
Definition choice: 0
plaso chosen.

Path to the project root: /home/parallels/git/plaso
Path [/home/parallels/git/plaso] set as the project path.

Name of the module to be generated. This can be something like "foobar sqlite" or "event analytics".
                                                                                                                                                                                
This will be used for class name generation and file name prefixes.                                                                                                             
Module Name: iOS Accounts
About to create a new feature branch to store newly generated code.
ERROR:root:Running: "git show-ref --verify --quiet refs/heads/"ios_accounts"" failed with error: .
Creating feature branch: ios_accounts inside /home/parallels/git/plaso
Switching to feature branch ios_accounts

Available scaffolders for plaso:
  [0] sqlite
Scaffolder choice: 0

Define the name of the callback function (key) that will be
called for every row returned from the SQL query (value).                                                                                                                       
The plugin will execute the SQL query and call the callback                                                                                                                     
once for each resulting row. The name of the function should                                                                                                                    
follow style guide and be descriptive. An example of that is                                                                                                                    
a SQL statement that fetches bookmarks, the key name should be                                                                                                                  
Bookmark, or if the SQL statement collects GPS coordinates                                                                                                                      
it could be called Location.                                                                                                                                                    
Callback function name [#1]: User
SQL Statement [#1]: select zdate, zaccounttypedescription, zusername, zaccountdescription, zaccount.zidentifier, zaccount.zowningbundleid from zaccount, zaccounttype where zaccounttype.z_pk=zaccount.zaccounttype
More entries? [Y/n]: n

List of required tables
Value to add [#1]: zaccount
Add more values? [Y/n]: Y
Value to add [#2]: zaccounttype
Add more values? [Y/n]: n

Absolute or relative path to the file that will be used for tests.
Value: /home/parallels/Desktop/Accounts3.sqlite
Ready to generate files? [Y/n]: Y
File: /home/parallels/git/plaso/test_data/Accounts3.sqlite written to disk.
File: /home/parallels/git/plaso/plaso/parsers/sqlite_plugins/ios_accounts.py written to disk.
File: /home/parallels/git/plaso/tests/parsers/sqlite_plugins/ios_accounts.py written to disk.
File: /home/parallels/git/plaso/plaso/formatters/ios_accounts.py written to disk.
File: /home/parallels/git/plaso/tests/formatters/ios_accounts.py written to disk.
File: /home/parallels/git/plaso/plaso/formatters/__init__.py written to disk.
File: /home/parallels/git/plaso/plaso/parsers/sqlite_plugins/__init__.py written to disk.

Error message, or output that was unexpected
After the scaffolder finished the plugin, I rebuilt plaso using command python3 setup.py build && python3 setup.py install. Then, I run psteal.py to build a timeline from an sqlite given for the test. I got the error messages below.

File "/home/parallels/git/virtualenv/plaso/lib/python3.9/site-packages/pkg_resources/__init__.py", line 656, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/home/parallels/git/virtualenv/plaso/lib/python3.9/site-packages/pkg_resources/__init__.py", line 1453, in run_script
    exec(code, namespace, namespace)
  File "/home/parallels/git/virtualenv/plaso/lib/python3.9/site-packages/plaso-20220501-py3.9.egg/EGG-INFO/scripts/psteal.py", line 21, in <module>
    from plaso.cli import psteal_tool
  File "/home/parallels/git/virtualenv/plaso/lib/python3.9/site-packages/plaso-20220501-py3.9.egg/plaso/cli/psteal_tool.py", line 11, in <module>
    from plaso.cli import extraction_tool
  File "/home/parallels/git/virtualenv/plaso/lib/python3.9/site-packages/plaso-20220501-py3.9.egg/plaso/cli/extraction_tool.py", line 18, in <module>
    from plaso import parsers  # pylint: disable=unused-import
  File "/home/parallels/git/virtualenv/plaso/lib/python3.9/site-packages/plaso-20220501-py3.9.egg/plaso/parsers/__init__.py", line 83, in <module>
    from plaso.parsers import sqlite_plugins
  File "/home/parallels/git/virtualenv/plaso/lib/python3.9/site-packages/plaso-20220501-py3.9.egg/plaso/parsers/sqlite_plugins/__init__.py", line 20, in <module>
    from plaso.parsers.sqlite_plugins import ios_accounts
  File "/home/parallels/git/virtualenv/plaso/lib/python3.9/site-packages/plaso-20220501-py3.9.egg/plaso/parsers/sqlite_plugins/ios_accounts.py", line 13, in <module>
    from plaso.lib import eventdata
ImportError: cannot import name 'eventdata' from 'plaso.lib' (/home/parallels/git/virtualenv/plaso/lib/python3.9/site-packages/plaso-20220501-py3.9.egg/plaso/lib/__init__.py)

Expected behavior

The plaso should successfully parse the given sqlite file.

Desktop (please complete the following information):

OS: Kali Linux
Version: 2021.3 arm64

l2tscaffolder Version

l2tscaffolder 20200511.
I installed l2tscaffolder from source.

Possible solution

eventdata is not in plaso.lib anymore. If I am not mistaken, it has been moved to plaso.containers. I need to dig deeper to the plaso code base. I will submit a fix for this issue. I am working on this now. We need to fix the jinja template as well.

Add text parser template for plaso

Is there already a definition file for the scaffolder?
Yes, plaso.

What is required for this scaffolder to work?
tests, formatter and parser templates.

What questions are needed?
adding them as I go through the template

Single or Multi line

log2timeline / l2tscaffolder Goto Github PK

l2tscaffolder's People

Contributors

Stargazers

Watchers

Forkers

l2tscaffolder's Issues

Recommend Projects

Recommend Topics

Recommend Org