Comments (13)
Hi there! I think I've sorted this all out.
I've created a general Bluetooth stats page, relevant to Herald and other protocols, that talks about the issues, provides a summary, and links to source material: https://vmware.github.io/herald/efficacy/statistics
I've also created a page that summarises Herald specific performance statistics and links to the relevant test results: https://vmware.github.io/herald/efficacy/herald
I've also amended the pages you mention to make them consistent, and link to the above stats pages.
I've also added more detail to the latest report, showing contact events, detections, payload exchanges from real tests. For example: https://vmware.github.io/herald/results/herald-2020-11-28
I think that makes all the info hang together nicer. Please let me know what you think. If there are any other suggestions in future, please open another issue! Thanks for taking the time to give feedback, it's very much appreciated!
from theheraldproject.github.io.
Also, it's not clear whether you're including devices that do not support BLE at all, which also could be the cause of the much higher 35% no-advertiser number.
from theheraldproject.github.io.
Ah yes I should be more consistent. Thanks for bringing this up. To clarify, it's ~35% of Android phones in the UK, which equates to ~14% of all UK phones. If you have a read of the paper it clears this up: https://vmware.github.io/herald/efficacy/paper
I'll clear up the website so it's using the same stat everywhere, and points to the section in the paper that has the explanation. Thanks for pointing it out!
from theheraldproject.github.io.
Oh and I forgot to mention - it's based on 'active use' data from the BBC, geofenced to the UK.
from theheraldproject.github.io.
Related to #5
from theheraldproject.github.io.
from theheraldproject.github.io.
Although the Mermaid pie charts aren't showing up nicely... I'll try and fix that.
from theheraldproject.github.io.
Image URLs fixed. Enjoy!
from theheraldproject.github.io.
Thanks, that's significantly better!
One thing to note is that using BBC App install data is biased and not representative of the population, and you should talk to them about compensating for their demographics so that it looks more like "person living in the UK" than "person who watches BBC" 😄
The data source you're using from AltBeacon is poor quality data:
-
Several models are listed multiple times; eg: Pixel C is listed four times with two different Android versions, where each both say
yes
andno
for BLE advertisement support. Nexus 9, Nexus 5, SM-G900F, Sony F5121 have similar issues. -
Several other models appear with differences in formatting / spacing / captialisation, or things like
Samsung AOSP on Galaxy Nexus
,Samsung Galaxy Nexus
. -
The list itself is old, where the newest Google phone listed is the Pixel 2.
-
AltBeacon itself was still discovering vendors with incorrect
isMultipleAdvertisementSupported
values, which also skews data their users provided before commits like: AltBeacon/android-beacon-library@f3d4337
Figure W2
is based on number of Android models (this should be called out as "models" rather than devices actually in use), and those duplicated entries will skew the numbers. Because the AltBeacon data is old, it will significantly skew the Android numbers towards old devices.
What's needed there is to clean up the AltBeacon data, and then figure out what to do about the models that aren't in the table, and publish that. It may also be "reasonable" to presume most devices on Android 6 or later that support BLE support advertisements properly.
Figure W3
is suggests 14% of devices do not support advertising -- but then Figure W1
shows that 9% of devices do not support BLE. As far as I can tell, Herald still requires BLE to work, which means it isn't able to target the entire 14%, only the subset that do support BLE (5%, based on your numbers). That should be amended to show devices that don't support BLE as a separate slice of that pie.
But as I mention above, the data from AltBeacon isn't good, mainly as a result of it not being an Android hardware feature code and OEMs breaking things (then again, even when it is a feature code, they can still break things).
The OFCOM data for "phones with BLE support" (72.53%) in the latest report shows an even higher amount of devices that don't support BLE -- but I don't know how to reconcile that part.
Also, please don't use seasons in dates, because they mean different things depending on which hemisphere the reader is in, and marginalises people in the other hemisphere to the author. Specify months or quarters instead. 🙃
from theheraldproject.github.io.
Oh also, consider excluding non-phone devices (like tablets and set-top boxes). This again is going to change around the BBC data, which is likely to bias towards iPlayer users (and thus, video playback, on devices with larger screens).
from theheraldproject.github.io.
Yes indeed. I've pored over the OFCOM data too, especially for the efficacy paper. It's hard finding accurate data on this. Herald itself does log whether a phone supports advertising or not. So app developers could, if they chose, track this in their own install base to get an accurate number.
Great feedback.
I believe the 14% is not inclusive of the 9%, but I shall double check and make it clear on the stats page.
from theheraldproject.github.io.
I have confirmed that the 14% is not inclusive of the 9%. So it's "~14% of phone users in the UK have a phone that does not support advertising"
from theheraldproject.github.io.
Thanks!
There's still an issue with Figure W2
, which is based on number of Android models, that isn't called out as "models" rather than devices actually in use.
I'm also unclear as to whether has excluded non-phone Android devices, such as tablets and set-top boxes, which are unlikely to be used for any sort of Bluetooth contact tracing app, but are secondary devices that may be used for a BBC app. Some of these non-phone devices also appear in the AltBeacon list.
Calling the advertising support data used in this calculation merely "a little old" is an exceptionally charitable characterisation. Given the egregious data problems, the documentation still doesn't make clear how this was reconciled. Even if one cannot get access to the same (BBC) data you used, it should be possible to use another data source to plug those numbers through the same process, to determine what fraction of devices do or don't support advertising.
There is also a statement about new low end devices "possibly" not supporting full Bluetooth functionality, there is no data given to support this claim. As a counter-point, these low-end devices are likely to have other problems with their Bluetooth stack, such as being unable to support multiple connections, to which GATT-based apps like Herald is are not immune: we've seen interference on low-end devices impacting continuous blood-glucose monitors and other Bluetooth devices.
At present, Herald's website makes an exceptionally bold claim about the reach of other contact tracing solutions. While the changes made thus far are a significant improvement, there's still a gap in these numbers, and these options come with significant trade-offs that are not fully elided in the technical documentation.
from theheraldproject.github.io.
Related Issues (20)
- Add code rendering widget/include for Jekyll to website
- Upload getting started tutorials for iOS and Android
- Update Archi diagrams for Exposure API
- Add blog post about phone self calibration
- Add opentrace svg logo to project
- Add use case documentation
- Update PGP key for release signing for 2022-2023 HOT 1
- Add release announcement for v2.2.0
- Multiple instances of SensorArray?
- Update API docs for v2.0
- Add contributors and code style guide to website
- Create Security and Privacy guides on Website
- v2.0 release announcement
- LFPH project governance - Project Herald call for committers - July 2021 HOT 7
- Add compare URLs and links for each release
- Add references to LFPH on the website
- Add mailing list URL to the community page
- OpenTrace announcement
- Need fix for jekyll deploy script
- Fix first two links on the OpenTrace page
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from theheraldproject.github.io.