famedly / barad-dur Goto Github PK
View Code? Open in Web Editor NEWmatrix phone-home stats collector
matrix phone-home stats collector
In GitLab by @dnelson2023 on Feb 23, 2023, 24:29
I've deployed Barad-dur in a kubernetes environment using image: registry.gitlab.com/famedly/infra/services/barad-dur:latest.
The pod appears to be running and in the ready state, but it hasn't initialized the db table(s). It looks like it failed in setup_logging(). I'm hoping you can provide some info on how to fix this.
The following is logged:
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: IndeterminateOffset', src/main.rs:23:22
stack backtrace:
0: 0x7f5716902560 - std::backtrace_rs::backtrace::libunwind::trace::h8c197fa9a679d134
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
1: 0x7f5716902560 - std::backtrace_rs::backtrace::trace_unsynchronized::h9ee19d58b6d5934a
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
2: 0x7f5716902560 - std::sys_common::backtrace::_print_fmt::h4badc450600fc417
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/sys_common/backtrace.rs:65:5
3: 0x7f5716902560 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::had334ddb529a2169
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/sys_common/backtrace.rs:44:22
4: 0x7f571693ee1e - core::fmt::write::h1aa7694f03e44db2
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/fmt/mod.rs:1209:17
5: 0x7f57168fcd15 - std::io::Write::write_fmt::h61b2bdc565be41b5
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/io/mod.rs:1682:15
6: 0x7f5716902325 - std::sys_common::backtrace::_print::h4ec69798b72ff254
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/sys_common/backtrace.rs:47:5
7: 0x7f5716902325 - std::sys_common::backtrace::print::h0e6c02048dec3c77
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/sys_common/backtrace.rs:34:9
8: 0x7f5716903bdf - std::panicking::default_hook::{{closure}}::hcdb7e705dc37ea6e
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/panicking.rs:267:22
9: 0x7f571690391a - std::panicking::default_hook::he03a933a0f01790f
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/panicking.rs:286:9
10: 0x7f57169042d8 - std::panicking::rust_panic_with_hook::he26b680bfd953008
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/panicking.rs:688:13
11: 0x7f5716904077 - std::panicking::begin_panic_handler::{{closure}}::h559120d2dd1c6180
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/panicking.rs:579:13
12: 0x7f5716902a0c - std::sys_common::backtrace::__rust_end_short_backtrace::h36db621fc93b005a
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/sys_common/backtrace.rs:137:18
13: 0x7f5716903d92 - rust_begin_unwind
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/panicking.rs:575:5
14: 0x7f57165d1cc3 - core::panicking::panic_fmt::he7679b415d25c5f4
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/panicking.rs:65:14
15: 0x7f57165d1f83 - core::result::unwrap_failed::hb71caff146724b6b
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/result.rs:1791:5
16: 0x7f57166575e0 - barad_dur::setup_logging::{{closure}}::h0df4cdb1401c1e14
17: 0x7f571669830d - <fern::log_impl::Dispatch as log::Log>::log::h5b45ca66e1c7bbb4
18: 0x7f5716897ebf - log::__private_api_log::hdb9ae2420b64e492
19: 0x7f571660212f - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h0b6d7ca9358ddcb2
20: 0x7f57166151a8 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::he71c60bdd6da9319
21: 0x7f57165ea241 - tokio::runtime::task::core::Core<T,S>::poll::h50d165a00360f808
22: 0x7f5716638967 - tokio::runtime::task::harness::Harness<T,S>::poll::h09a29a767476b9d7
23: 0x7f571686b9c6 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::hc0f0ebb7679c7246
24: 0x7f571686b5b0 - tokio::runtime::scheduler::multi_thread::worker::Context::run::h87323f8ea004a904
25: 0x7f571686e2d9 - tokio::macros::scoped_tls::ScopedKey<T>::set::h13371b3ca5c3ad02
26: 0x7f571686acc1 - tokio::runtime::scheduler::multi_thread::worker::run::h8b3956965431363a
27: 0x7f5716885732 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h9c10fae310dfcaf6
28: 0x7f57168819ac - tokio::runtime::task::core::Core<T,S>::poll::h54a37ace3227557a
29: 0x7f571687fb2f - tokio::runtime::task::harness::Harness<T,S>::poll::ha43acb01d8765877
30: 0x7f5716872869 - tokio::runtime::blocking::pool::Inner::run::h2ad2814d0694f87e
31: 0x7f5716872ef2 - std::sys_common::backtrace::__rust_begin_short_backtrace::hf5dd45ba19f3f8fb
32: 0x7f5716878c65 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h93cdc31f609e70dc
33: 0x7f5716906af3 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hed812350c5aef7a8
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/alloc/src/boxed.rs:1987:9
34: 0x7f5716906af3 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hc7df8e435a658960
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/alloc/src/boxed.rs:1987:9
35: 0x7f5716906af3 - std::sys::unix::thread::Thread::new::thread_start::h575491a8a17dbb33
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/sys/unix/thread.rs:108:17
We'll likely need some indices on the homeserver
and local_timestamp
columns to get those queries to be more performant.
In GitLab by @nikzen on Apr 3, 2023, 10:50
Merges larodar-main-patch-21630 -> main
In GitLab by @jcgruenhage on Sep 2, 2021, 10:49
There's multiple issues with the current Dockerfile:
docker built .
will not work, as the release binary is copied in instead of being built in the containerSomething like https://gitlab.com/famedly/company/backend/services/CRM/-/blob/main/Dockerfile should work well
Currently, the only way to get the aggregated stats is to get them directly from the database. A better option could be to expose some /get_aggregated_stats
endpoint to avoid unnecessary coupling among services
In GitLab by @jcgruenhage on Feb 15, 2021, 13:41
To help with operations, this tool should automatically do migrations when upgrading to a newer version with a new database schema. sqlx
has some built-in support for this, which https://gitlab.com/famedly/services/user-directory/-/tree/main/migrations uses, so that can be used as an inspiration.
In GitLab by @jcgruenhage on Oct 26, 2021, 13:25
Merges jcgruenhage/axum -> main
Fixes #6 and #7, as well as supporting offline compilation (not requiring a database for compilation) and replacing actix with axum.
In GitLab by @jdreichmann on Apr 15, 2021, 15:00
Merges transcaffeine/package-docker -> main
In GitLab by @Shekhinah on Mar 9, 2021, 16:54
Merges shekhinah/more-helpful-config-sample -> main
As part of the work for auditable containers (https://github.com/famedly/product-management/issues/1446), we should use cargo auditable
in all Dockerfiles to build our rust binaries, as that will embed info about the dependency tree in the binary itself.
In practice, this just means going through the project's Dockerfiles and replacing cargo build --release
with cargo auditable build --release
.
In GitLab by @jcgruenhage on Feb 15, 2021, 13:38
https://github.com/matrix-org/panopticon has a separate script in python, that handles aggregation. We want to handle this in the same tool, so we should write a future that handles this instead. For this, we need a timer loop, which spawns this future once a day (or do we want to spawn it more often?) to do the daily compaction.
In GitLab by @jcgruenhage on Feb 15, 2021, 13:27
Workflow for this is probably reading a dump of panopticon into temporary tables and then moving the data over to the correct tables. Should happen in a separate SQL script.
After internal discussion, we will remove the dependabot configuration of version updates in favour of configuration in the github UI.
The axum version used should be updated to a recent version. This will likely require some refactoring.
In GitLab by @jcgruenhage on Feb 15, 2021, 13:33
This should implement an HTTP endpoint, which takes in a JSON endpoint like https://github.com/matrix-org/panopticon does, and stores it into the database.
For this http endpoint, we need a simple actix-web server. https://gitlab.com/famedly/services/organisation-directory/ uses actix-web for http, but it's important not to take over too much, as the organisation directory does not use sqlx
, but diesel instead, so the database interaction will be different.
In GitLab by @jcgruenhage on Feb 15, 2021, 13:29
This should support phone home stats for different versions of synapse, and as far as other servers implement it, phone home stats of those other homeservers too. This is a future feature though, so it can wait a bit. For the first release, just a single request from the most recent synapse release is sufficient.
The sync action template should be set up for this repository. See the parent issue for more information.
In GitLab by @Shekhinah on Mar 9, 2021, 13:35
Merges shekhinah/cmd-config-argument -> main
In GitLab by @jcgruenhage on Oct 28, 2021, 10:06
Merges jcgruenhage/import-fixes -> main
I've tried out the import scripts and looked at them again in more details, and found a few things that I wasn't quite happy with. After fixing those up, I checked whether the aggregation worked, and no, while it did aggregate data, the data was off in comparison to what I expected. After checking again and modifying the query, I was able to get that into the right shape, although not without some support from an experienced SQL magician. Last but not least, I thought to myself: If I'm already doing all that stuff, I might as well refactor the code a bit, update deps, update to the newly stabilized 2021 edition of rust and extend documentation.
In GitLab by @Shekhinah on Mar 9, 2021, 10:45
Merges shekhinah/ci -> main
In GitLab by @Shekhinah on Mar 9, 2021, 19:19
Merges main -> shekhinah/code-review
In GitLab by @jcgruenhage on Aug 23, 2021, 09:48
In GitLab by @Shekhinah on Feb 22, 2021, 09:43
Merges develop -> main
Rebased the commits and implemented suggested changes.
In GitLab by @dominikrimpf on Dec 18, 2022, 20:50
Following errors appeared with cargo audit
:
Fetching advisory database from `https://github.com/RustSec/advisory-db.git`
Loaded 474 security advisories (from /Users/dominik/.cargo/advisory-db)
Updating crates.io index
warning: couldn't update crates.io index: registry: SecureTransport error: (null); class=Net (12)
Scanning Cargo.lock for vulnerabilities (177 crate dependencies)
Crate: chrono
Version: 0.4.19
Title: Potential segfault in `localtime_r` invocations
Date: 2020-11-10
ID: RUSTSEC-2020-0159
URL: https://rustsec.org/advisories/RUSTSEC-2020-0159
Solution: Upgrade to >=0.4.20
Dependency tree:
chrono 0.4.19
├── sqlx-core 0.5.9
│ ├── sqlx-macros 0.5.9
│ │ └── sqlx 0.5.9
│ │ └── barad-dur 0.1.0
│ └── sqlx 0.5.9
└── barad-dur 0.1.0
Crate: regex
Version: 1.5.4
Title: Regexes with large repetitions on empty sub-expressions take a very long time to parse
Date: 2022-03-08
ID: RUSTSEC-2022-0013
URL: https://rustsec.org/advisories/RUSTSEC-2022-0013
Solution: Upgrade to >=1.5.5
Dependency tree:
regex 1.5.4
├── serde-hjson 0.9.1
│ └── config 0.11.0
│ └── barad-dur 0.1.0
└── axum 0.2.8
└── barad-dur 0.1.0
Crate: time
Version: 0.1.44
Title: Potential segfault in the time crate
Date: 2020-11-18
ID: RUSTSEC-2020-0071
URL: https://rustsec.org/advisories/RUSTSEC-2020-0071
Solution: Upgrade to >=0.2.23
Dependency tree:
time 0.1.44
└── chrono 0.4.19
├── sqlx-core 0.5.9
│ ├── sqlx-macros 0.5.9
│ │ └── sqlx 0.5.9
│ │ └── barad-dur 0.1.0
│ └── sqlx 0.5.9
└── barad-dur 0.1.0
Crate: tokio
Version: 1.12.0
Title: Data race when sending and receiving after closing a `oneshot` channel
Date: 2021-11-16
ID: RUSTSEC-2021-0124
URL: https://rustsec.org/advisories/RUSTSEC-2021-0124
Solution: Upgrade to >=1.8.4, <1.9.0 OR >=1.13.1
Dependency tree:
tokio 1.12.0
├── tower 0.4.10
│ ├── barad-dur 0.1.0
│ └── axum 0.2.8
│ └── barad-dur 0.1.0
├── tokio-util 0.6.8
│ ├── tower 0.4.10
│ └── axum 0.2.8
├── tokio-stream 0.1.7
│ └── sqlx-core 0.5.9
│ ├── sqlx-macros 0.5.9
│ │ └── sqlx 0.5.9
│ │ └── barad-dur 0.1.0
│ └── sqlx 0.5.9
├── tokio-rustls 0.22.0
│ └── sqlx-rt 0.5.9
│ ├── sqlx-macros 0.5.9
│ └── sqlx-core 0.5.9
├── sqlx-rt 0.5.9
├── hyper 0.14.14
│ ├── barad-dur 0.1.0
│ └── axum 0.2.8
├── barad-dur 0.1.0
└── axum 0.2.8
Crate: tower-http
Version: 0.1.1
Title: Improper validation of Windows paths could lead to directory traversal attack
Date: 2022-01-21
ID: RUSTSEC-2022-0043
URL: https://rustsec.org/advisories/RUSTSEC-2022-0043
Solution: Upgrade to >=0.2.1 OR >=0.1.3, <0.2.0
Dependency tree:
tower-http 0.1.1
└── axum 0.2.8
└── barad-dur 0.1.0
Could you bump your dependency versions to fix these issues?
In GitLab by @jcgruenhage on Aug 23, 2021, 24:27
Merges jcgruenhage/additional-fields -> main
To avoid loosing data on synapse updates which add new metrics, we shouldn't have #[serde(deny_unknown_fields)]
in there.
Reported by @reivilibre
In GitLab by @lrsksr on Apr 19, 2023, 13:41
Merges lrsksr/chore/release-v0.1.2 -> main
In GitLab by @Shekhinah on Mar 1, 2021, 14:26
Merges shekhinah/aggregate-stats-reports -> main
I've changed again the table format in the migrations file. As this project isn't in use I think that's still legal.
We should add functionality to generate an aggregated stats report based on the server_context
field.
Additionally, we should change the aggregated stats endpoint to allow for specifying which server context you want to fetch an aggregated report for.
The stats aggregation statements currently run on a timer, once every 86400 seconds. This is not ideal as there will be scheduling drift, and it's unclear when during the day the aggregation will run. We should implement properly configurable scheduling.
In GitLab by @Shekhinah on Mar 9, 2021, 16:41
Merges shekhinah/better-testing -> main
In GitLab by @Shekhinah on Feb 25, 2021, 17:07
Merges shekhinah/collect-stat-reports -> main
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.