Activity types detection doesn't work out of the box. about locokit HOT 8 CLOSED

Oh, I should add: keep an eye on my new LocoKit2 repo. I'm rebuilding LocoKit from the ground up, to get rid of almost a decade of cruft, and make use of latest technologies, cleaner algorithms, etc.

I've only got as far as rebuilding the core recording functionality so far. But the new Kalman filter is already a big win. Can see screenshots of the old Kalman versus the new Kalman in this support forum post. The difference is sometimes quite dramatic, and a big step up from both raw location data and from the old naive Kalman filter.

I haven't got as far as porting across timeline item processing, or activity type classification to LocoKit2 yet. So it's not a drop in replacement yet. But over the new few months those should fall into place (I don't have any significant changes I want to make to either of those subsystems, so it's largely just going to be copy and paste).

from locokit.

Hey, thanks for such a detailed response. Yes, I'm using the core-ml branch.
Ok, going to train BD0 core-types model, thanks for the advice.
Btw new Kalman filter looks awesome.

from locokit.

Do I understand right that classifying isn't done immediately when recording items?

Yep!

Until recently I did have TimelineRecorder immediately classifying some samples, if it would help to distinguish whether to start a new TimelineItem or not (eg it's a moving -> moving transition, but at slow speed, and classifiedType changed, so it could plausibly be something like walking -> running).

But I stopped doing that because it turned out that running the CoreML classifier in the background was causing excessive energy drain, even when it was only one sample and only infrequently. My assumption is that it must've caused a high performance core to temporarily spin up, while the rest of the recording process could be done on high efficiency cores.

Now I'm trying to figure out right way to get the activityType for specific dateRange and I wonder is it the right flow to achieve it

I'm just sifting through the Arc Timeline / Arc Mini code now, but I think what I'm doing is just touching timelineItem.classifierResults, which either loads the cached classifier results, or runs a classifier over the samples in the item, then returns those results.

When a classifier is run over the samples, if the sample's classifiedType changes, it might or might not mark the sample as hasChanges, thus ready to be saved/updated in the db. I think I might've taken out that hasChanges = true though, to avoid some unnecessary db updates in some cases. Not sure about that... still sifting through the code.

But either way, yeah, whenever you run the classifier over the samples in an item, it'll update the sample.classifiedType of each.

TimelineItem.swift

    public var classifierResults: ClassifierResults? {
        if let cached = _classifierResults { return cached }

        guard let results = classifier?.classify(self, timeout: 30) else { return nil }

        // don't cache if it's incomplete
        if results.moreComing { return results }

        _classifierResults = results
        return results
    }

It actually looks like reclassifySamples() in TimelineSegment is doing an unnecessary duplication of that setting of sample.classifiedType. Or... no, it looks like maybe it's doing a more efficient version of it, by only running a classifier over the samples with nil confirmedType and nil classifiedType.

Ok, so, yeah, TimelineSegment is attempting to only fill in missing classifications. While hitting timelineItem.classifierResults looks like it probably reclassifies all the samples, regardless of existing classifiedType value.

I think I primarily use timelineItem.classifierResults for the Edit views in Arc / Mini, so that I can get full results for showing in a list in the UI.

I recently came up with tracking the hashes of already updated segments to not process them again, and calling .save on all TimelineItems inside of updated segment, but not sure if it's how it all should be working.

Yeah I'm not sure what the best approach will be. Might depend a lot on your app architecture and/or UI.

Ah, just found the hasChanges = true, so that's still there:

    internal override var _classifiedType: ActivityTypeName? {
        didSet { if oldValue != _classifiedType { hasChanges = true } }
    }

Ok so that means that as long as the classifier has been run over the samples, you'll get that value persisted to db if you call save() on the samples.

Looks like I'm actually doing that immediately/implicitly for confirmedType:

    public override var confirmedType: ActivityTypeName? {
        didSet { if oldValue != confirmedType { hasChanges = true; save() } }
    }

Which is probably the change I'm vaguely remembering - I was probably previously doing an immediate save() when classifiedType changed too. Which I'm betting was triggering too many db updates under some conditions.

from locokit.

For TimelineSegment.startUpdating(), looks like I'm triggering that on and off based on app going into background/foreground.

From Arc Timeline's timeline view controller init():

        let activeObserver = when(UIApplication.didBecomeActiveNotification) { [weak self] _ in
            self?.segment?.startUpdating()
            self?.workoutsSegment?.startUpdating()
        }
        observers.append(activeObserver)

        let backgroundObserver = when(UIApplication.didEnterBackgroundNotification) { [weak self] _ in
            self?.segment?.stopUpdating()
            self?.workoutsSegment?.stopUpdating()
        }
        observers.append(backgroundObserver)

So looks like that'll probably be triggering some sample.classifiedType updating too. Though I still prefer the explicit call to timelineItem.classifierResults, just because that's the most conclusive / certain. Once that's done, the values should all be there. Where as with a TimelineSegment, the classifiedType values are only going to flow in when ... well, it's less explicit, and more difficult to reason about.

from locokit.

Thanks again, it was really helpful insights, now classifying became more or less clear for me.

from locokit.

Hi @SherhaMM!

Step one is make sure you're using the core-ml branch. But sounds like you've already got that far.

The main branch uses the old custom activity type models, as well as server side models. But that system is deprecated, and I'll be turning off the server side models API in a few days, as EOL.

The core-ml branch uses all Core ML models, built on-device. As well as an optional BD0 model, as you saw. In my own apps on the App Store, I use a BD0 model which is a copy of my own CD0 model (global scale model), but slimmed down to only contain the core activity types (walking, running, cycling, car, airplane). That BD0 acts as a fallback, for when the user doesn't have any custom built models of their own.

The only option that I found is to manually train ml models by providing confirmedType for samples.

Yep, that's what you'll need to do. You also will need to run the model updates background task , which you can see code for in Arc Mini as well (coreMLModelUpdates, managed in TaskManager.swift). Arc Mini schedules that task to run at highest frequency of I think once every hour, but it won't do anything unless there's new data changes that justify a model update.

When that task runs, it will rebuild any pending CD0, CD1, and CD2 models (which are the global model, state/country sized models, and neighbourhood sized models respectively). Those updates will typically take less than a second each, until the user has built up a large amount of data on their device.

Once you have enough data of your own, and a well fleshed out CD0 of your own, you can use that as a BD0 to ship in your app, same as I do in Arc Timeline. Although I did find it best to rebuild it with only the core types included, otherwise new users get confused/annoyed by seemingly random classifier results like chairlift, skiing, hiking, etc.

The server side models sound appealing, as a way to get the users started quickly and accurately, without any model training. But in practice the Core ML models start to produce reliable results with extremely small amounts of data. So even just one or two days of user corrections/confirmations (ie UI that results in confirmedType getting set) can be enough to start getting results that look good to the users.

So in practice they're better off with purely Core ML models, no server side models, due to Core ML's higher accuracy and fast training time with limited data. And if you can build up a roughly useful BD0 from your own data, to bundle into the app, you can also bootstrap new users with semi-sensible classifier results of their first few days of use, until their own models pick up the slack.

from locokit.

Thanks! I'll close this issue now, but feel free to start a new one, or reopen this one, if you have any questions 😄

from locokit.

@sobri909 Hi, it's me again with detecting activity types .
I trained models and decided to include all sizes in the final app, to see how it all working (BTW I'm using TimelineRecorder).

• Do I understand right that classifying isn't done immediately when recording items?

Now I'm trying to figure out right way to get the activityType for specific dateRange and I wonder is it the right flow to achieve it:

1. Get TimelineSegment using TimelineStore.segment(for dateRange)
2. As TimelineSegment doesn't have the activityType, call TimelineSegment startUpdates() to reclassify the Items.
3. Wait until segment will be updated.
4. Execute some business logic using updated activityTypes of TimelineItems in the segment.

The problem with this approach is that I have to process every Segment every time before being able to get their activityType because TimelineSegment.startUpdates() isn't saved to the db.
I recently came up with tracking the hashes of already updated segments to not process them again, and calling .save on all TimelineItems inside of updated segment, but not sure if it's how it all should be working.

Thanks in advance.

from locokit.