helium314 / local-nlp-backend Goto Github PK

View Code? Open in Web Editor NEW

This project forked from n76/dejavu

64.0 64.0 4.0 3.02 MB

Yet another network location backend for the UnifiedNLP/microG project

License: GNU General Public License v3.0

Java 7.35% Kotlin 92.65%

local-nlp-backend's People

Contributors

Stargazers

Watchers

Forkers

bitrate16 nlpn00b sorunome riqq

local-nlp-backend's Issues

import data error

Hey :-)

Previously, I've used Déjà Vu NLP -> I disabled it -> installed "Local NLP Backend" -> import data -> selected a csv file from opencellid -> "keep local emitters unchanged" -> error message: "error: file format unknown". I also tried it with the original, compressed .gz-file - to no avail.

Did I do anything wrong?

cheers

Allow GPS access needs to be set manually

When trying to activate the module in microg app, it seems like microg just reloads and the local-nlp-backend is still marked as deactivated. So the user just gets the impression of a faulty app.
To make it work I needed to go to the android settings of the app, and manually give it the location permission "Allow all the time". This is on android 13. But I remember, that it was the same on android 11.

link to microG 0.2.10 (a.k.a. "this backend does not work")

Hi,
Issue: this backend is not working on my phone, Samsung Galaxy sm j530-f (stock rom, android 7.0).

Helium314 shared he's using microG gmsCore, version 0.2.10, I have experienced the same problems he cites with newer versions: stations are registered, but I'm unable to make the NLP work as a provider. So, can sb share a link to microG gmsCore 0.2.10 zip file.
Alternatively, please share a link to a microG version that works properly. (Tried with MinMicroG version 2.12.0, it does not work)

I also tried to make this backend work just with UnifiedNLP, by installing it using magisk or xposed (using intika's or rawi's versions), but none of those allows me to pass all the UnifiedNLP or gmsCore "Self-Check" Will microG gmsCore 0.2.10, together with any of those magisk or xposed modules, allow me to pass "Self-Check"?

Otherwise, is there anybody out there who can give any clue to get this backend functional? I installed MicroG 2.12.0, UnifiedNLP, and then the backend. I did not remove any Google stock app.

Too many requests?

Noticed that the app was doing requests to frequently. Location indicator shows up every 10-30 seconds while seated and not moving, so I guess scanned networks are barely changing. I'm using LocalNLPBackend in MicroG with location permission allowed always, Android 13, LineageOS4microg 20.0, "Use cell towers" enabled, "Use wifi locations" enabled, active mode off, active mode GPS timout 10.

This is a sample from the logcat grepping the app id:

05-17 17:41:55.221  4023  4023 D LocationFuser: Received no location from helium314.localbackend/org.fitchfamily.android.dejavu.BackendService
05-17 17:41:55.223  1699  2042 I WifiScanRequestProxy: Scan request from helium314.localbackend throttled
05-17 17:42:05.220  4023  4023 D LocationFuser: Received no location from helium314.localbackend/org.fitchfamily.android.dejavu.BackendService
05-17 17:42:05.222  1699  2042 I WifiScanRequestProxy: Scan request from helium314.localbackend throttled
05-17 17:42:15.220  4023  4023 D LocationFuser: Received no location from helium314.localbackend/org.fitchfamily.android.dejavu.BackendService
05-17 17:42:15.227  1699  2042 I WifiScanRequestProxy: Scan request from helium314.localbackend throttled
05-17 17:42:25.220  4023  4023 D LocationFuser: Received no location from helium314.localbackend/org.fitchfamily.android.dejavu.BackendService
05-17 17:42:25.222  1699  2042 I WifiScanRequestProxy: Scan request from helium314.localbackend throttled
05-17 17:42:35.226  4023  4023 D LocationFuser: Received no location from helium314.localbackend/org.fitchfamily.android.dejavu.BackendService
05-17 17:42:35.230  1699  2042 I WifiScanRequestProxy: Scan request from helium314.localbackend throttled
05-17 17:42:45.220  4023  4023 D LocationFuser: Received no location from helium314.localbackend/org.fitchfamily.android.dejavu.BackendService
05-17 17:42:45.238  1699  2042 I WifiScanRequestProxy: Scan request from helium314.localbackend throttled
05-17 17:42:55.221  1699  2042 I WifiScanRequestProxy: Scan request from helium314.localbackend throttled
05-17 17:42:55.222  4023  4023 D LocationFuser: Received no location from helium314.localbackend/org.fitchfamily.android.dejavu.BackendService
05-17 17:43:05.221  4023  4023 D LocationFuser: Received no location from helium314.localbackend/org.fitchfamily.android.dejavu.BackendService
05-17 17:43:05.224  1699  2042 I WifiScanRequestProxy: Scan request from helium314.localbackend throttled
05-17 17:43:15.220  4023  4023 D LocationFuser: Received no location from helium314.localbackend/org.fitchfamily.android.dejavu.BackendService
05-17 17:43:15.222  1699  2042 I WifiScanRequestProxy: Scan request from helium314.localbackend throttled
05-17 17:43:25.223  1699  2042 I WifiScanRequestProxy: Scan request from helium314.localbackend throttled
05-17 17:43:25.223  4023  4023 D LocationFuser: Received no location from helium314.localbackend/org.fitchfamily.android.dejavu.BackendService
05-17 17:43:35.221  4023  4023 D LocationFuser: Received no location from helium314.localbackend/org.fitchfamily.android.dejavu.BackendService
05-17 17:43:35.224  1699  2042 I WifiScanRequestProxy: Scan request from helium314.localbackend throttled
05-17 17:43:45.228  1699  2042 I WifiScanRequestProxy: Scan request from helium314.localbackend throttled
05-17 17:43:45.228  4023  4023 D LocationFuser: Received no location from helium314.localbackend/org.fitchfamily.android.dejavu.BackendService
05-17 17:43:55.221  4023  4023 D LocationFuser: Received no location from helium314.localbackend/org.fitchfamily.android.dejavu.BackendService
05-17 17:43:55.233  1699  2042 I WifiScanRequestProxy: Scan request from helium314.localbackend throttled
05-17 17:44:05.221  4023  4023 D LocationFuser: Received no location from helium314.localbackend/org.fitchfamily.android.dejavu.BackendService
05-17 17:44:05.223  1699  2042 I WifiScanRequestProxy: Scan request from helium314.localbackend throttled
05-17 17:44:15.221  1699  2042 I WifiScanRequestProxy: Scan request from helium314.localbackend throttled
05-17 17:44:15.223  4023  4023 D LocationFuser: Received no location from helium314.localbackend/org.fitchfamily.android.dejavu.BackendService
05-17 17:44:49.704  4023  4023 D LocationFuser: Received no location from helium314.localbackend/org.fitchfamily.android.dejavu.BackendService
05-17 17:44:49.705  4023  4023 D LocationFuser: Received no location from helium314.localbackend/org.fitchfamily.android.dejavu.BackendService
05-17 17:44:49.706  4023  4023 D LocationFuser: Received no location from helium314.localbackend/org.fitchfamily.android.dejavu.BackendService
05-17 17:44:49.706  1699  2042 I WifiScanRequestProxy: Scan request from helium314.localbackend throttled
05-17 17:44:55.220  4023  4023 D LocationFuser: Received no location from helium314.localbackend/org.fitchfamily.android.dejavu.BackendService
05-17 17:44:55.222  1699  2042 I WifiScanRequestProxy: Scan request from helium314.localbackend throttled
05-17 17:45:05.219  4023  4023 D LocationFuser: Received no location from helium314.localbackend/org.fitchfamily.android.dejavu.BackendService
05-17 17:45:05.221  1699  2042 I WifiScanRequestProxy: Scan request from helium314.localbackend throttled
05-17 17:45:15.220  4023  4023 D LocationFuser: Received no location from helium314.localbackend/org.fitchfamily.android.dejavu.BackendService
05-17 17:45:15.222  1699  2042 I WifiScanRequestProxy: Scan request from helium314.localbackend throttled
05-17 17:45:25.219  4023  4023 D LocationFuser: Received no location from helium314.localbackend/org.fitchfamily.android.dejavu.BackendService
05-17 17:45:25.222  1699  2042 I WifiScanRequestProxy: Scan request from helium314.localbackend throttled
05-17 17:45:35.220  4023  4023 D LocationFuser: Received no location from helium314.localbackend/org.fitchfamily.android.dejavu.BackendService
05-17 17:45:35.221  1699  2042 I WifiScanRequestProxy: Scan request from helium314.localbackend throttled

Is this something expected?

Cannot export data: No enum constant org.fitchfamily.android.dejavu.EmitterType

version: 1.2.11
installed from: f-droid
os: lineage microg 20 (2023-07-04)

full stacktrace from logcat:

04-22 13:41:06.603  5149  5165 I LocalNLP Settings: exportToFile - error
04-22 13:41:06.603  5149  5165 I LocalNLP Settings: java.lang.IllegalArgumentException: No enum constant org.fitchfamily.android.dejavu.EmitterType.
04-22 13:41:06.603  5149  5165 I LocalNLP Settings: 	at java.lang.Enum.valueOf(Enum.java:259)
04-22 13:41:06.603  5149  5165 I LocalNLP Settings: 	at org.fitchfamily.android.dejavu.EmitterType.valueOf(RfCharacteristics.kt:0)
04-22 13:41:06.603  5149  5165 I LocalNLP Settings: 	at org.fitchfamily.android.dejavu.DatabaseKt.toRfEmitter(Database.kt:489)
04-22 13:41:06.603  5149  5165 I LocalNLP Settings: 	at org.fitchfamily.android.dejavu.DatabaseKt.toRfEmitter$default(Database.kt:486)
04-22 13:41:06.603  5149  5165 I LocalNLP Settings: 	at org.fitchfamily.android.dejavu.Database$getAll$1.invoke(Database.kt:423)
04-22 13:41:06.603  5149  5165 I LocalNLP Settings: 	at org.fitchfamily.android.dejavu.Database$getAll$1.invoke(Database.kt:423)
04-22 13:41:06.603  5149  5165 I LocalNLP Settings: 	at org.fitchfamily.android.dejavu.DatabaseKt$toSequence$1.invoke(Database.kt:476)
04-22 13:41:06.603  5149  5165 I LocalNLP Settings: 	at kotlin.sequences.GeneratorSequence$iterator$1.calcNext(Sequences.kt:591)
04-22 13:41:06.603  5149  5165 I LocalNLP Settings: 	at kotlin.sequences.GeneratorSequence$iterator$1.hasNext(Sequences.kt:609)
04-22 13:41:06.603  5149  5165 I LocalNLP Settings: 	at org.fitchfamily.android.dejavu.SettingsActivity$exportToFile$2.invokeSuspend(SettingsActivity.kt:1295)
04-22 13:41:06.603  5149  5165 I LocalNLP Settings: 	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
04-22 13:41:06.603  5149  5165 I LocalNLP Settings: 	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
04-22 13:41:06.603  5149  5165 I LocalNLP Settings: 	at kotlinx.coroutines.internal.LimitedDispatcher.run(LimitedDispatcher.kt:42)
04-22 13:41:06.603  5149  5165 I LocalNLP Settings: 	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:95)
04-22 13:41:06.603  5149  5165 I LocalNLP Settings: 	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:570)
04-22 13:41:06.603  5149  5165 I LocalNLP Settings: 	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
04-22 13:41:06.603  5149  5165 I LocalNLP Settings: 	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:677)
04-22 13:41:06.603  5149  5165 I LocalNLP Settings: 	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:664)

Location scanning seems stuck on in 1.2.0-beta.3

Love the update to Deja Vu.

I noticed that my location indicator was stuck "on" while using Local NLP Backend. It was not microG using the location either, but rather Local NLP Backend itself. Not sure what it was doing, but the location indicator was on for many many minutes. I don't know if this is intended behavior. No GPS app was running either. Phone was otherwise idle.

I had to kill the app to make it stop attempting to acquire location. I also noticed that afterwards when trying to manually scan from the NLP settings, it didn't work until I toggled Local NLP Backend off and on in the microG settings.

[1.2.5]"Show Nearby Emitters" triggers crash

With Local NLP not being initialized yet. Crashes happen regardless if Wi-Fi Scanning is enabled under Location or not.

This is on A13 / LOS20.

I'm not seeing anything super obvious in logcat, except this fairly cryptic (for me at least) set of messages that tend to repeat:

E WifiHAL : get_wifi_interface_info: QCA_WLAN_VENDOR_ATTR_LL_STATS_IFACE_INFO_TS_DUTY_CYCLE not found
E WifiHAL : nl80211: requestResponse->nl_recvmsgs failed: -5
E WifiVendorHal: getWifiLinkLayerStats_1_6_Internal(l.1199) failed {.code = ERROR_INVALID_ARGS, .description = }

can't import from previous release

if you install over the pre-alpha release it crashes. So I tried to do a clean install and use the importer to import the rf.db from the previous release. This give this error :

Error importing database : Can't downgrade database from version 5 to 4

Only WLAN2 entries in the database.

With a new installation without importing anything, there are only WLAN2 entries in the database even after a few days.
The exported database then looks like this:

database v4
rfID,rfType,latitude,longitude,radius_ns,radius_ew,note
WLAN2/.....
WLAN2/.....
WLAN2/.....
...
WLAN2/.....
WLAN2/.....

[Feature] Upload scan results to MLS

Hi,

It would be nice to be able to upload the results to MLS (kind of like TowerCollector).
MLS api seems straightforward and android provides the API to gather required parameters.
This api seems to be the one used by the app.

Anyway, could be a nice addition ;).

Thanks

Déjà Vu database import: 'Error: file format unknown'

Hi!

I had been using the Déjà Vu NLP backend until now and wished to import its database into Local NLP. The description of the 'Import data' option in the interface of Local NLP suggests that this is supported. However, the import fails with the message 'Error: file format unknown'.

Steps to reproduce

Copy /data/data/org.fitchfamily.android.dejavu/databases/rf.db to a location accessible without root privileges.
Open the configuration interface of Local NLP backend
Select 'Import data'
Select the file rf.db copied at step 1.
Pick any option on how to handle emitters already existing in the local database, it makes no difference.
Observe error message.

Is this import not supported? If it is, does anyone has ny idea as to the cause of the problem, or a possible solution?

microG remove NLP module support since 0.2.28.231657

Upstream microG remove NLP module support since 0.2.28.231657, and only use builtin module. So we can't use this project with microG.
Upstream still doesn't provide new module way. But they said, "No modules anymore, they became unreliable in recent Android versions due to 'energy savings'"
If upstream don't want to support external module because of background control of newer androids, the only way may be try to merge this project into microG project.

Opencellid database successfully imported but not working

Hello, I am using a rooted curtana (Xiaomi Redmi Note 9s) device with Android 14 CrDroid rom, microg v0.2.27.223616
and my only backend is LocalNlp v1.2.11.

The problem is that the opencellid data do not seem to work.

More specifically, I have imported a CSV file database from opencellid for my country (Greece) successfully, but I did not get location fix without satellites for a long time moving around in the capital of Greece, so today I did some tests at home.
I deleted all data from localnlp and imported only the CSV file freshly downloaded from opencellid, disabled the active mode and went for the "show nearby emitters" option. There was one emitter found, marked as unknown, so I searched for it on the opencellid site and I found it.

Then I enabled active mode, and made sure I get a satellite location fix. After that the unknown emitter was registered with a specific location and I could get a location while being in my house without satellites fix.

So, may there be some problem on the way the app is reading the data from opencellid csv file?

[Feature] Manual training

I wanted to ask if manual training could be something you could consider to add.

It happens that I handle devices which very low GPS detection and in zones were GPS services are not available.

Since this work attaching a GPS location to a phone cell of WiFi AP I was thinking about the possibility of doing this manually. Perhaps, maybe, through a map choosing your current location and with access to a table or log to remove it in the case I was wrong at any time.

And without microG?

If I understand it correctly, this replaces the google services that tend to provide location data within seconds, but needs microG.
I have a completely G-Free phone, are you aware of any usable alternatives?

Scanning failed ...

LineageOS 18 (android 11), microg installed

SatStat shows the reception of cells, but
Show nearby emitters -> "Scanning failed, maybe the backend service is disabled"
At the same time My Location app reports:
Local NLP Backend:
Some Latitude, some Longitude
Accuracy: 371m (wich is strange with the cell tower in sight ...)
Of course I imported the Mozialla Database befrorehand and the reported location is next to perfect (within 20m)
For testing puroses I switched off WiFi scanning, to make sure only to use cell towers.

Cannot import MLS

1.2.4 from f-droid:
I downloaded Mozillas full cell exports csv file from here: https://location.services.mozilla.com/downloads
The import skips every entry.

Crash : java.lang.NullPointerException at BackendService.backgroundProcessing

Version : 1.2.11 F-Droid
Android 14 / LineageOS 21 on Samsung A52s

FATAL EXCEPTION: DefaultDispatcher-worker-1
Process: helium314.localbackend, PID: 29126
java.lang.NullPointerException
at org.fitchfamily.android.dejavu.BackendService.backgroundProcessing(BackendService.kt:754)
at org.fitchfamily.android.dejavu.BackendService.access$backgroundProcessing(BackendService.kt:50)
at org.fitchfamily.android.dejavu.BackendService$queueForProcessing$1.invokeSuspend(BackendService.kt:724)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.internal.LimitedDispatcher.run(LimitedDispatcher.kt:42)
at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:95)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:570)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:677)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:664)
Suppressed: kotlinx.coroutines.DiagnosticCoroutineContextException: [StandaloneCoroutine{Cancelling}@2a57992, Dispatchers.IO]

[1.2.6-debug] Crash: database is locked

Got this one while interacting with the configuration screen (IIRC).

While something is probably not entirely right, I may not be able to reproduce it, it seemed some kind of freak accident.

FATAL EXCEPTION: DefaultDispatcher-worker-6
Process: helium314.localbackend.debug, PID: 2335
android.database.sqlite.SQLiteDatabaseLockedException: database is locked (code 5 SQLITE_BUSY): , while compiling: PRAGMA journal_mode
at android.database.sqlite.SQLiteConnection.nativePrepareStatement(Native Method)
at android.database.sqlite.SQLiteConnection.acquirePreparedStatement(SQLiteConnection.java:1068)
at android.database.sqlite.SQLiteConnection.executeForString(SQLiteConnection.java:811)
at android.database.sqlite.SQLiteConnection.setJournalMode(SQLiteConnection.java:419)
at android.database.sqlite.SQLiteConnection.setJournalFromConfiguration(SQLiteConnection.java:339)
at android.database.sqlite.SQLiteConnection.open(SQLiteConnection.java:268)
at android.database.sqlite.SQLiteConnection.open(SQLiteConnection.java:205)
at android.database.sqlite.SQLiteConnectionPool.openConnectionLocked(SQLiteConnectionPool.java:512)
at android.database.sqlite.SQLiteConnectionPool.open(SQLiteConnectionPool.java:210)
at android.database.sqlite.SQLiteConnectionPool.open(SQLiteConnectionPool.java:202)
at android.database.sqlite.SQLiteDatabase.openInner(SQLiteDatabase.java:1085)
at android.database.sqlite.SQLiteDatabase.open(SQLiteDatabase.java:1065)
at android.database.sqlite.SQLiteDatabase.openDatabase(SQLiteDatabase.java:929)
at android.database.sqlite.SQLiteDatabase.openDatabase(SQLiteDatabase.java:918)
at android.database.sqlite.SQLiteOpenHelper.getDatabaseLocked(SQLiteOpenHelper.java:373)
at android.database.sqlite.SQLiteOpenHelper.getWritableDatabase(SQLiteOpenHelper.java:316)
at org.fitchfamily.android.dejavu.Database.getDatabase(Database.kt:44)
at org.fitchfamily.android.dejavu.Database.query(Database.kt:81)
at org.fitchfamily.android.dejavu.Database.query$default(Database.kt:70)
at org.fitchfamily.android.dejavu.Database.getEmitters(Database.kt:420)
at org.fitchfamily.android.dejavu.Cache.loadIds(Cache.kt:114)
at org.fitchfamily.android.dejavu.BackendService.backgroundProcessing(BackendService.kt:733)
at org.fitchfamily.android.dejavu.BackendService.access$backgroundProcessing(BackendService.kt:50)
at org.fitchfamily.android.dejavu.BackendService$queueForProcessing$1.invokeSuspend(BackendService.kt:710)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.internal.LimitedDispatcher.run(LimitedDispatcher.kt:42)
at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:95)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:570)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:677)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:664)
Suppressed: kotlinx.coroutines.DiagnosticCoroutineContextException: [StandaloneCoroutine{Cancelling}@dce322f, Dispatchers.IO]