Consistency issue: % phones that cannot support advertising mode about theheraldproject.github.io HOT 13 CLOSED

Hi there! I think I've sorted this all out.

I've created a general Bluetooth stats page, relevant to Herald and other protocols, that talks about the issues, provides a summary, and links to source material: https://vmware.github.io/herald/efficacy/statistics

I've also created a page that summarises Herald specific performance statistics and links to the relevant test results: https://vmware.github.io/herald/efficacy/herald

I've also amended the pages you mention to make them consistent, and link to the above stats pages.

I've also added more detail to the latest report, showing contact events, detections, payload exchanges from real tests. For example: https://vmware.github.io/herald/results/herald-2020-11-28

I think that makes all the info hang together nicer. Please let me know what you think. If there are any other suggestions in future, please open another issue! Thanks for taking the time to give feedback, it's very much appreciated!

from theheraldproject.github.io.

Also, it's not clear whether you're including devices that do not support BLE at all, which also could be the cause of the much higher 35% no-advertiser number.

from theheraldproject.github.io.

Ah yes I should be more consistent. Thanks for bringing this up. To clarify, it's ~35% of Android phones in the UK, which equates to ~14% of all UK phones. If you have a read of the paper it clears this up: https://vmware.github.io/herald/efficacy/paper

I'll clear up the website so it's using the same stat everywhere, and points to the section in the paper that has the explanation. Thanks for pointing it out!

from theheraldproject.github.io.

Oh and I forgot to mention - it's based on 'active use' data from the BBC, geofenced to the UK.

from theheraldproject.github.io.

Related to #5

from theheraldproject.github.io.

from theheraldproject.github.io.

Although the Mermaid pie charts aren't showing up nicely... I'll try and fix that.

from theheraldproject.github.io.

Image URLs fixed. Enjoy!

from theheraldproject.github.io.

Thanks, that's significantly better!

One thing to note is that using BBC App install data is biased and not representative of the population, and you should talk to them about compensating for their demographics so that it looks more like "person living in the UK" than "person who watches BBC" 😄

The data source you're using from AltBeacon is poor quality data:

• Several models are listed multiple times; eg: Pixel C is listed four times with two different Android versions, where each both say yes and no for BLE advertisement support. Nexus 9, Nexus 5, SM-G900F, Sony F5121 have similar issues.

• Several other models appear with differences in formatting / spacing / captialisation, or things like Samsung AOSP on Galaxy Nexus, Samsung Galaxy Nexus.

• The list itself is old, where the newest Google phone listed is the Pixel 2.

• AltBeacon itself was still discovering vendors with incorrect isMultipleAdvertisementSupported values, which also skews data their users provided before commits like: AltBeacon/android-beacon-library@f3d4337

Figure W2 is based on number of Android models (this should be called out as "models" rather than devices actually in use), and those duplicated entries will skew the numbers. Because the AltBeacon data is old, it will significantly skew the Android numbers towards old devices.

What's needed there is to clean up the AltBeacon data, and then figure out what to do about the models that aren't in the table, and publish that. It may also be "reasonable" to presume most devices on Android 6 or later that support BLE support advertisements properly.

Figure W3 is suggests 14% of devices do not support advertising -- but then Figure W1 shows that 9% of devices do not support BLE. As far as I can tell, Herald still requires BLE to work, which means it isn't able to target the entire 14%, only the subset that do support BLE (5%, based on your numbers). That should be amended to show devices that don't support BLE as a separate slice of that pie.

But as I mention above, the data from AltBeacon isn't good, mainly as a result of it not being an Android hardware feature code and OEMs breaking things (then again, even when it is a feature code, they can still break things).

The OFCOM data for "phones with BLE support" (72.53%) in the latest report shows an even higher amount of devices that don't support BLE -- but I don't know how to reconcile that part.

Also, please don't use seasons in dates, because they mean different things depending on which hemisphere the reader is in, and marginalises people in the other hemisphere to the author. Specify months or quarters instead. 🙃

from theheraldproject.github.io.

Oh also, consider excluding non-phone devices (like tablets and set-top boxes). This again is going to change around the BBC data, which is likely to bias towards iPlayer users (and thus, video playback, on devices with larger screens).

from theheraldproject.github.io.

Yes indeed. I've pored over the OFCOM data too, especially for the efficacy paper. It's hard finding accurate data on this. Herald itself does log whether a phone supports advertising or not. So app developers could, if they chose, track this in their own install base to get an accurate number.

Great feedback.

I believe the 14% is not inclusive of the 9%, but I shall double check and make it clear on the stats page.

from theheraldproject.github.io.

I have confirmed that the 14% is not inclusive of the 9%. So it's "~14% of phone users in the UK have a phone that does not support advertising"

from theheraldproject.github.io.

Thanks!

There's still an issue with Figure W2, which is based on number of Android models, that isn't called out as "models" rather than devices actually in use.

I'm also unclear as to whether has excluded non-phone Android devices, such as tablets and set-top boxes, which are unlikely to be used for any sort of Bluetooth contact tracing app, but are secondary devices that may be used for a BBC app. Some of these non-phone devices also appear in the AltBeacon list.

Calling the advertising support data used in this calculation merely "a little old" is an exceptionally charitable characterisation. Given the egregious data problems, the documentation still doesn't make clear how this was reconciled. Even if one cannot get access to the same (BBC) data you used, it should be possible to use another data source to plug those numbers through the same process, to determine what fraction of devices do or don't support advertising.

There is also a statement about new low end devices "possibly" not supporting full Bluetooth functionality, there is no data given to support this claim. As a counter-point, these low-end devices are likely to have other problems with their Bluetooth stack, such as being unable to support multiple connections, to which GATT-based apps like Herald is are not immune: we've seen interference on low-end devices impacting continuous blood-glucose monitors and other Bluetooth devices.

At present, Herald's website makes an exceptionally bold claim about the reach of other contact tracing solutions. While the changes made thus far are a significant improvement, there's still a gap in these numbers, and these options come with significant trade-offs that are not fully elided in the technical documentation.

from theheraldproject.github.io.