Expected Behavior

After running the installer again and change/set only the config parameters to fit your needs, Peekaboo should start normally

Current Behavior

After unning the installer on a 1.6.2 installation, Peekaboo fails to start because of some missing config parameters and sections in the config and does not catch up the new internal defaults

Possible Solution

The installer does update the submodule itself

Steps to Reproduce

1. Have a working 1.6.2 installation
2. git pull
3. git checkout v1.7
4. run the installer
5. Try to restart the peekaboo daemon like service peekaboo restart

Context (Environment)

Updating the submodule manually via git submodule update --recursive --remote fixed it.
Maybe this could be done automatically in the script?

Detailed Description

example error message:

-- Unit peekaboo.service has begun starting up.
Apr 24 12:46:05 peekaboo.sittig.local peekaboo[22086]: Starting Peekaboo 1.6.2.
Apr 24 12:46:05 peekaboo.sittig.local peekaboo[22086]: 2019-04-24 12:46:05,480 - peekaboo.config - (MainThread) - CRITICAL - configuration option not found
Apr 24 12:46:05 peekaboo.sittig.local peekaboo[22086]: 2019-04-24 12:46:05,480 - peekaboo.config - (MainThread) - ERROR - No option 'user' in section: 'global'
Apr 24 12:46:05 peekaboo.sittig.local peekaboo[22086]: Traceback (most recent call last):
Apr 24 12:46:05 peekaboo.sittig.local peekaboo[22086]:   File "/opt/peekaboo/local/lib/python2.7/site-packages/peekaboo/config.py", line 80, in __parse
Apr 24 12:46:05 peekaboo.sittig.local peekaboo[22086]:     self.user = config.get('global', 'user')
Apr 24 12:46:05 peekaboo.sittig.local peekaboo[22086]:   File "/usr/lib/python2.7/ConfigParser.py", line 618, in get
Apr 24 12:46:05 peekaboo.sittig.local peekaboo[22086]:     raise NoOptionError(option, section)
Apr 24 12:46:05 peekaboo.sittig.local peekaboo[22086]: NoOptionError: No option 'user' in section: 'global'
Apr 24 12:46:05 peekaboo.sittig.local systemd[1]: peekaboo.service: Main process exited, code=exited, status=1/FAILURE
Apr 24 12:46:05 peekaboo.sittig.local systemd[1]: peekaboo.service: Failed with result 'exit-code'.
Apr 24 12:46:05 peekaboo.sittig.local systemd[1]: Failed to start Peekaboo Extended Email Attachment Behavior Observation Owl.

We should enhance the installer so it can install unreleased Peekaboo versions, e.g. directly from the git repo. The ansible playbook could have a flag for that. Maybe the PeekabooAV repo could be a submodule or the ansible playbook checks it out into a temporary directory itself.

Currently setting up a new instance of peekaboo.

Going through the "Check the components" section and tried to execute
su -c "cuckoo" peekaboo

But it seems that the executable cuckoo is not installed to /usr/local/bin/cuckoo and cuckoo could not be found. Executing bash su -c "/opt/cuckoo/bin/cuckoo" peekaboo is working though
Is this intented?
If so, I would update the documentation

Hi,

just tried to create a new test installation on a brand new Ubuntu 18.04 server installation but the PeekabooAV-install.sh fsileed with this error message. Any idea?

TASK [Work around flask werkzeug version mismatch] *****************************************************
fatal: [localhost]: FAILED! => {"changed": false, "cmd": "/opt/cuckoo/bin/pip2 install werkzeug==0.16.1", "msg": "stdout: Collecting werkzeug==0.16.1\n\n:stderr: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f864e20e450>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/werkzeug/\n Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f864e20e590>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/werkzeug/\n Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f864e20e690>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/werkzeug/\n Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f864e20e890>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/werkzeug/\n Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f864e20ea10>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /simple/werkzeug/\n Could not find a version that satisfies the requirement werkzeug==0.16.1 (from versions: )\nNo matching distribution found for werkzeug==0.16.1\n"}
to retry, use: --limit @/root/peekabooav-installer/PeekabooAV-install.retry

PLAY RECAP *********************************************************************************************
localhost : ok=18 changed=4 unreachable=0 failed=1

ERROR: 'ansible-playbook' failed. Please fix manually

Greetings

Frank

Currently we have a copy of the amavis plugin in the installer that needs to be kept in sync with the main instance in PeekabooAV repo. We should eliminate the local copy and take the plugin directly from the Peekaboo installation.

gcc is not installed.
Therefore building failes for some python modules.

Adding package gcc in "Install basic tools" solves this.

Hello there, I got an error (see below) when I try to install Peekaboo with the installer. I use a ubuntu-18.04.2-live-server-amd64 VM.

TASK [Force python version switch in Peekaboo virtualenv] *******************************************
fatal: [localhost]: FAILED! => {"msg": "The conditional check 'peekaboo_python.stat.lnk_target != python_exe' failed. The error was: error while evaluating conditional (peekaboo_python.stat.lnk_target != python_exe): 'dict object' has no attribute 'lnk_target'\n\nThe error appears to have been in '/root/peekabooav-installer/PeekabooAV-install.yml': line 392, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n - name: Force python version switch in Peekaboo virtualenv\n ^ here\n"}
to retry, use: --limit @/root/peekabooav-installer/PeekabooAV-install.retry

PLAY RECAP ******************************************************************************************
localhost : ok=34 changed=7 unreachable=0 failed=1

ERROR: 'ansible-playbook' failed. Please fix manually

Thanks for your help and quick response!

https://github.com/scVENUS/PeekabooAV-Installer/blame/master/README-postinstallation.md#L162

@JannisTriesToCode #58 24:46

Our file capabilities configuration of tcpdump currently leaves it open to all users on the system. We should lock that down.

Some document formats identify as ZIP archives when looked at by file/magic. This includes Apple numbers, pages and keys filetypes and some Microsoft Office XML format documents which are generated by third-party tools (seen with a PDF conversion tool). This causes AMaViS to unpack them and hand the individual content snippets as samples to Peekaboo which prevents us from doing a proper analysis. There is no direct way to prevent this. There is an option keep_decoded_original_maps which would apply to all ZIP archives and do an additional analysis on all of them as well.

But as with all of AMaViS, we can hook some functions creatively to introduce a blacklist. There we can also decide to prevent unpacking altogether or tell AMaViS to do the unpacking but additionally hand the original, unmodified file to the virus scanner.

Example code for e.g. 50-peekaboo.conf:

# do not unpack certain files even though by their filetype they are archives
my $non_decompose_filename_re = new_RE(
  qr'\.(numbers|key|pages)$',
);

sub do_7zip_filtered($$$;$) {
  my($part, $tempdir, $archiver, $testing_for_sfx) = @_;

  if (defined($part->name_declared) &&
      Amavis::Lookup::lookup(0, $part->name_declared,
                             $non_decompose_filename_re)) {
    Amavis::Util::do_log(4, "filter_decompose_parts: not unpacking %s (%s)",
                         $part->base_name, $part->name_declared);
    # report part being atomic
    return 0;
  }

  return Amavis::Unpackers::do_7zip($part, $tempdir, $archiver,
                                    $testing_for_sfx);
}

unshift @decoders, (
  ['zip', \&do_7zip_filtered, ['7za', '7z']],
);

Untested code for keeping of original:

# do unpack but keep certain files even though by their filetype they are archives
my $keep_original_filename_re = new_RE(
  qr'\.docx$',
);

sub do_7zip_filtered($$$;$) {
  my($part, $tempdir, $archiver, $testing_for_sfx) = @_;

  if (defined($part->name_declared) &&
      Amavis::Lookup::lookup(0, $part->name_declared,
                             $keep_original_filename_re)) {
    Amavis::Util::do_log(4, "filter_decompose_parts: unpacking but keeping %s (%s)",
                         $part->base_name, $part->name_declared);
    # report part as original needing to be kept
    Amavis::Unpackers::do_7zip($part, $tempdir, $archiver,
                                      $testing_for_sfx);
    return 2;
  }

  return Amavis::Unpackers::do_7zip($part, $tempdir, $archiver,
                                    $testing_for_sfx);
}

unshift @decoders, (
  ['zip', \&do_7zip_filtered, ['7za', '7z']],
);

Needs some testing and a bit of thinking-cap scratching, how to integrate and which of the options to choose for analysis.

apt-get dist-upgrade run from the script croaks when upgrading grub because it spawns a dialog asking for the device to reinstall grub to. Maybe we should add an option to the installer to run apt-get commands fully silent.

   peekaboo: Preparing to unpack .../22-grub2-common_2.02-2ubuntu8.9_amd64.deb ...
    peekaboo: Package configuration┌──────────────────────────┤ Configuring grub-pc ├──────────────────────────┐│││ The GRUB boot loader was previously installed to a disk that is no││ longer present, or whose unique identifier has changed for some reason.   ││ It is important to make sure that the installed GRUB core image stays in  ││ sync with GRUB modules and grub.cfg. Please check again to make sure││ that GRUB is written to the appropriate boot devices.││││ If you're unsure which drive is designated as boot drive by your BIOS,    ││ it is often a good idea to install GRUB to all of them.││││ Note: it is possible to install GRUB to partition boot records as well,   ││ and some appropriate partitions are offered here. However, this forces    ││ GRUB to use the blocklist mechanism, which makes it less reliable, and    ││ therefore is not recommended.││││<Ok>│││└───────────────────────────────────────────────────────────────────────────┘
    peekaboo: Preparing to unpack .../25-libldap-common_2.4.45+dfsg-1ubuntu1.1_all.deb ...

Originally posted by @michaelweiser in #22

I have created a YT-Tutorial about the installation of peekaboo. The motivation was to make it easier for my partner in a peekaboo related study project to set up the system. I'm pretty sure that there is a lot of potential for improvement as I took a very naive approach. Please let me know about errors and possible improvements.

Since switchting to setup.py based installation the installer has no way to detect version changes between installed and to-be-installed version. As long as /opt/peekaboo/bin/peekaboo exists, Peekaboo will not be updated. Code from playbook:

  tasks:
    - name: Install Peekaboo
      command: /opt/peekaboo/bin/python setup.py install
      args:
        chdir: PeekabooAV
        creates: /opt/peekaboo/bin/peekaboo
#      pip:
#        name: peekabooav
#        state: present

Hello, I've been trying for days to get an executable version with the Peekaboo AV installer. The requirements are exactly as described on your page, Ubuntu 18.04, installation as root.
There are a few copy errors during installation, but I was able to fix them manually, but I always fail at the last point:

TASK [Start Peekaboo] ***************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Unable to start service peekaboo: Job for peekaboo.service failed because a timeout was exceeded.\nSee "systemctl status peekaboo.service" and "journalctl -xe" for details.\n"}
to retry, use: --limit @/tmp/tmp.shh6Vbzxc3-PeekabooAV-Installer/peekabooav-installer/PeekabooAV-install.retry

PLAY RECAP **************************************************************************************************
localhost : ok=50 changed=13 unreachable=0 failed=1

Sep 03 12:30:23 mail24intra peekaboo[13660]: peekaboo.queuing - (Worker-1) - DEBUG - Worker 1: Ready
Sep 03 12:30:23 mail24intra peekaboo[13660]: peekaboo.queuing - (MainThread) - DEBUG - Create Worker 2
Sep 03 12:30:23 mail24intra peekaboo[13660]: peekaboo.queuing - (MainThread) - INFO - Created 3 Workers.
Sep 03 12:30:23 mail24intra peekaboo[13660]: peekaboo.queuing - (MainThread) - DEBUG - Disabling cluster dupl
Sep 03 12:30:23 mail24intra peekaboo[13660]: peekaboo.queuing - (Worker-2) - DEBUG - Worker 2: Ready
Sep 03 12:30:23 mail24intra peekaboo[13660]: peekaboo.toolbox.cuckoo - (MainThread) - DEBUG - Requesting cuck
Sep 03 12:30:23 mail24intra peekaboo[13660]: urllib3.connectionpool - (MainThread) - DEBUG - Starting new HTT
Sep 03 12:30:23 mail24intra cuckoo[12513]: 2019-09-03 12:30:23,510 [werkzeug] INFO: 127.0.0.1 - - [03/Sep/201
Sep 03 12:30:23 mail24intra peekaboo[13660]: urllib3.connectionpool - (MainThread) - DEBUG - http://127.0.0.1
Sep 03 12:30:23 mail24intra peekaboo[13660]: urllib3.connectionpool - (MainThread) - DEBUG - Starting new HTT
Sep 03 12:30:23 mail24intra cuckoo[12513]: 2019-09-03 12:30:23,514 [werkzeug] INFO: 127.0.0.1 - - [03/Sep/201
Sep 03 12:30:23 mail24intra peekaboo[13660]: urllib3.connectionpool - (MainThread) - DEBUG - http://127.0.0.1
Sep 03 12:30:23 mail24intra peekaboo[13660]: urllib3.connectionpool - (MainThread) - DEBUG - Starting new HTT
Sep 03 12:30:23 mail24intra cuckoo[12513]: 2019-09-03 12:30:23,517 [werkzeug] INFO: 127.0.0.1 - - [03/Sep/201
Sep 03 12:30:23 mail24intra peekaboo[13660]: urllib3.connectionpool - (MainThread) - DEBUG - http://127.0.0.1
Sep 03 12:30:23 mail24intra peekaboo[13660]: Traceback (most recent call last):
Sep 03 12:30:23 mail24intra peekaboo[13660]: File "/opt/peekaboo/bin/peekaboo", line 10, in
Sep 03 12:30:23 mail24intra peekaboo[13660]: sys.exit(run())
Sep 03 12:30:23 mail24intra peekaboo[13660]: File "/opt/peekaboo/local/lib/python2.7/site-packages/peekaboo
Sep 03 12:30:23 mail24intra peekaboo[13660]: config.cuckoo_poll_interval)
Sep 03 12:30:23 mail24intra peekaboo[13660]: File "/opt/peekaboo/local/lib/python2.7/site-packages/peekaboo
Sep 03 12:30:23 mail24intra peekaboo[13660]: self.reported = self.__status()["tasks"]["reported"]
Sep 03 12:30:23 mail24intra peekaboo[13660]: TypeError: 'NoneType' object has no attribute 'getitem'

peekaboo.service - Peekaboo Extended Email Attachment Behavior Observation Owl
Loaded: loaded (/etc/systemd/system/peekaboo.service; enabled; vendor preset: enabled)
Active: activating (start) since Tue 2019-09-03 12:33:24 CEST; 1min 14s ago
Main PID: 13683 (peekaboo)
Tasks: 4 (limit: 4674)
CGroup: /system.slice/peekaboo.service
└─13683 /opt/peekaboo/bin/python2 /opt/peekaboo/bin/peekaboo -c /opt/peekaboo/etc/peekaboo.conf -D

Sep 03 12:33:24 mail24intra peekaboo[13683]: urllib3.connectionpool - (MainThread) - DEBUG - Starting new HTT
Sep 03 12:33:24 mail24intra peekaboo[13683]: urllib3.connectionpool - (MainThread) - DEBUG - http://127.0.0.1
Sep 03 12:33:24 mail24intra peekaboo[13683]: Traceback (most recent call last):
Sep 03 12:33:24 mail24intra peekaboo[13683]: File "/opt/peekaboo/bin/peekaboo", line 10, in
Sep 03 12:33:24 mail24intra peekaboo[13683]: sys.exit(run())
Sep 03 12:33:24 mail24intra peekaboo[13683]: File "/opt/peekaboo/local/lib/python2.7/site-packages/peekaboo
Sep 03 12:33:24 mail24intra peekaboo[13683]: config.cuckoo_poll_interval)
Sep 03 12:33:24 mail24intra peekaboo[13683]: File "/opt/peekaboo/local/lib/python2.7/site-packages/peekaboo
Sep 03 12:33:24 mail24intra peekaboo[13683]: self.reported = self.__status()["tasks"]["reported"]
Sep 03 12:33:24 mail24intra peekaboo[13683]: TypeError: 'NoneType' object has no attribute 'getitem'
lines 1-18/18 (END)

Unfortunately I haven't found a solution yet. Do you know what is going wrong here? Of course it would be great for me if I could get a completely installed Virtualbox image from you, then the installation would already be finished :)

Based on comments in scVENUS/PeekabooAV#91 we should automatically generate an API token and set it in cuckoo.conf and peekaboo.conf.

git clone https://github.com/scvenus/peekabooav-installer
cd peekabooav-installer/
git clone https://github.com/scvenus/peekabooav PeekabooAV
./PeekabooAV-install.sh

The generated peekaboo.conf is still the one without the api_token attribute. This is probably due to the installer using the .conf from the PeekabooAV-Installer repo and not the one from the PeekabooAV repo.

This issue appears to be related:
#43

Update version tags, configurations and references to use PeekabooAV 2.0 by default.
Plus testing ;-)

I would like to make a pull request. But after going through the contributing guidelines, I don't know if this change is valuable enough

Clevero@255be80

Currently we install config files for amavis (50-peekaboo.conf) and Peekaboo itself that contain example values. This causes the final setup to be non-functional by default. We could use ansible templates and logic to at least customize some of those values, e.g. the local host's FQDN etc.

Peekaboo is deprecating use of python 2 (scVENUS/PeekabooAV#129). We should switch the default python version for new installs to python 3.

Our storage directory /var/lib/peekaboo contains highly sensitive data but is currently world-readable. Particularly .cuckoo/conf/cuckoo.conf contains the database password and must not be readable by anyone other than the Peekaboo (or Cuckoo) runtime user. But also in-flight or archives malware samples as well as analyses reports should be kept private to peekaboo.

There are hardcoded database passwords:
https://github.com/scVENUS/PeekabooAV-Installer/blob/master/mysql/mysql.txt
That's generally not a good idea.

A possible fix:
https://docs.ansible.com/ansible/2.6/plugins/lookup/password.html

The tarballs auto-generated by GitHub do not contain submodule content. Since they're not git repos either, git submodule commands can not be used either. We need to get the submodule content into the tarball.

Here is a description with the proper way to start cuckoo, api, web ui, processors, router:
cuckoosandbox/cuckoo#2025

We should adapt our unit files accordingly.

@michaelweiser what's your optinion on this?

Short question: What is the difference between those config directories?

I'm thinking about that /opt/peekaboo/ is the place for the config that is overwritten with every update and /opt/peekaboo/etc/ is the place where we can modify our config that is kept over updates?

As @JannisTriesToCode points out in his tutorial (#58) we forgot to update the documentation in #53 when we split out cuckoo with its own user. We should update the documentation.

peekaboo.conf contains passwords but is currently installed world-readable. It should only be accessible to the peebaboo user.

Commit 763476f changed the owner to peekaboo and allows it to read and write to it.

I want to suggest to change only the group to peekaboo and mode to 640.
This way peekaboo can read but not write to it and database password is secure from others.

        group: peekaboo
        mode: 0640

Using the latest version I am running into communication problems between cuckoo and peekaboo.

Feb 12 13:49:40 jannis-VirtualBox cuckoo[1338]:   File "/opt/cuckoo/local/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
Feb 12 13:49:40 jannis-VirtualBox cuckoo[1338]:     return self.view_functions[rule.endpoint](**req.view_args)
Feb 12 13:49:40 jannis-VirtualBox cuckoo[1338]:   File "/opt/cuckoo/local/lib/python2.7/site-packages/cuckoo/apps/api.py", line 634, in cuckoo_status
Feb 12 13:49:40 jannis-VirtualBox cuckoo[1338]:     return jsonify(response)
Feb 12 13:49:40 jannis-VirtualBox cuckoo[1338]:   File "/opt/cuckoo/local/lib/python2.7/site-packages/flask/json.py", line 251, in jsonify
Feb 12 13:49:40 jannis-VirtualBox cuckoo[1338]:     if current_app.config['JSONIFY_PRETTYPRINT_REGULAR'] and not request.is_xhr:
Feb 12 13:49:40 jannis-VirtualBox cuckoo[1338]:   File "/opt/cuckoo/local/lib/python2.7/site-packages/werkzeug/local.py", line 347, in __getattr__
Feb 12 13:49:40 jannis-VirtualBox cuckoo[1338]:     return getattr(self._get_current_object(), name)
Feb 12 13:49:40 jannis-VirtualBox cuckoo[1338]: AttributeError: 'Request' object has no attribute 'is_xhr'
Feb 12 13:49:40 jannis-VirtualBox cuckoo[1338]: 2020-02-12 13:49:40,367 [werkzeug] INFO: 127.0.0.1 - - [12/Feb/2020 13:49:40] "GET /cuckoo/status HTTP/1.1" 500 -

Feb 12 13:51:30 jannis-VirtualBox peekaboo[7085]: urllib3.connectionpool - (MainThread) - DEBUG - http://127.0.0.1:8090 "GET /cuckoo/status HTTP/1.1" 500 290
Feb 12 13:51:30 jannis-VirtualBox peekaboo[7085]: peekaboo.toolbox.cuckoo - (MainThread) - ERROR - Invalid JSON in response when getting http://127.0.0.1:8090/cuckoo/status: Expecting value: line 1 column 1 (char 0)
Feb 12 13:51:30 jannis-VirtualBox peekaboo[7085]: peekaboo.toolbox.cuckoo - (MainThread) - CRITICAL - Connection to Cuckoo REST API failed

It could be related to the latest update of a dependency of flask. Link

https://github.com/scVENUS/PeekabooAV-Installer/blob/cfad3c136a22214877d7391cfadf54b92be07bd4/utils/peekabooStatus.sh

peekabooStatus.sh doesn't work with current version, neither Peekaboo nor database.

Documentation has to be added.