dragosrotaru / ppeforfree Goto Github PK
View Code? Open in Web Editor NEWCollective sensemaking for mutual aid groups manufacturing PPE during COVID.
Home Page: https://ppeforfree.org
License: GNU General Public License v3.0
Collective sensemaking for mutual aid groups manufacturing PPE during COVID.
Home Page: https://ppeforfree.org
License: GNU General Public License v3.0
Pre-requisites
See #9
We need a page on the website with a high-level view of all the Facebook posts and links being shared globally. This is the community MegaPhone or Signal Amplifier.
This could be as simple as a Reddit style News Feed. But we can also visualize where original ideas are coming from? What group leads the community by being the first to post fresh information. ETC.
Our website right now is very cluttered and confusing when we drop in the directory right away.
See #5
See #6
We will use this data to make a news aggregator and to keep an eye out for more data for coalition-building purposes.
I started a script in scripts/facebook-group-posts-scraper using this library:
https://github.com/kevinzg/facebook-scraper
It works ok, but it doesn't work with 100% consistency, you will have to troubleshoot and maybe edit the script.
Database will be MongoDB
Schema
type Post = {
id: UUID,
createdAt: TimeStamp,
text: string,
link: URL,
likes: number,
shares: number,
comments: number,
groupID: UUID,
scrapedAt: TimeStamp,
scrapeID: UUID,
}
Random Lib I found: https://github.com/ParvJain/Facebook-Group-Scraper (please look through)
OSCMS has a roster on google sheets I have gone through and grabbed every Facebook group (and page) id from (187 total). Roster:
The data is available in data/facebook-group-ids-unclean.txt
Not all ID's are for FB groups though, so we need to pre-process them. Salty_Steve wrote a script to do that in Python but it doesn't work for all. Maybe it just needs rate-limiting implemented. See scripts/facebook-group-id-validator.
fix scripts/facebook-group-id-validator and product a clean file of group IDs called data/facebook-group-ids.txt. This is critical, please manually check your work. We need clean data.
We should have a "help" page that offers multiple ways to engage:
we need to clearly explain how these different types of website visitors can get engaged.
Like Page as in Facebook Pages: https://www.facebook.com/OSCMSspringfield
https://twitter.com/ppeforfree
probably in the header as an icon, in which case make the GitHub link an icon too
See #8
How do we process the groupa? IDK, up to you. Let's get creative. The purpose is to enable a community map and directory. Work closely with DataViz.
Write a scraper for all the Facebook group urls on the HTML of this page.
See #7
How do we process the posts? IDK, up to you. Let's get creative. The purpose is to enable a megaphone feature or signal amplifier on the community. We've got to track links being shared (normalized of course, without query parameters) and track measure the virality of the content. We will want to display it on a community board that in an engaging way, contextualized with info about where it came from (maybe). Who posted it first? etc.
This can get really interesting. Can we track the propagation of information through the community? How do we display the info? What sorting features would we want? Work closely with the DataViz contributor.
A graph (as in graph theory) where vertices represent Groups. The radius of a vertex represents the size of the group, and edge lengths represent the number of members these groups have in common (closer means more members in common).
Programmatically Download CSV using this link here:
https://docs.google.com/spreadsheets/u/3/d/1JH5uL3WW6PwvwFRe4wqXkheK0-jcGYqaPmb9J3Dr6Ac/export?format=csv&id=1JH5uL3WW6PwvwFRe4wqXkheK0-jcGYqaPmb9J3Dr6Ac&gid=0
files like tsconfig.json and package.json inherit from their parent folder. This makes it hard to understand whats going on. Its better for projects to not be sub-nested. But deployment needs to be updated as well if that is the case.
Describe the bug
Directory table creates horizontal scroll
To Reproduce
Steps to reproduce the behavior:
Additional context
May not be implemented yet
The Prusa site runs on a GraphQL Endpoint. We want to scrape user IDs (46k) and Full Group Details (A few hundred). Don't publish this data, We are going to scrape the user ids so we can send them 1 message within the Prusa platform asking if they know of any local initiatives. Then we can process any links they reply to us with. Then if they are responsive, we provide them with a link to our site and thank them.
Make Internal Documentation Accessible from the website
We have our first translated copy of the manifesto being worked on in French, We need to support this on the site.
See #5
See #6
We need data on all the Facebook groups in the community.
The data available on public FB groups (not including content like posts, pics, events, etc) I have found by manually going through 2 FB group pages includes:
Note: I compiled this by manually going through 2 FB group pages, please go through a few more pages yourself to see if some groups have more, less or differing public data available and we will update our schema
We will not get any other information about individuals other than their facebook id. This data is needed because we want to see how connected groups are (how many individuals they have in common) and we want to reach out to those individuals that are in a shit ton of groups! Very useful for coalition-building
I started a script in scripts/facebook-group-posts-scraper using this library:
https://github.com/kevinzg/facebook-scraper
It works well! But! We NEED to collect the timestamp on all the posts. It doesnt work with 100% consistency, you will have to troubleshoot. We will use this data to make a news aggregator and to keep an eye out for more data for coalition-building purposes.
Database will be MongoDB
Schema
type Group = {
id: UUID,
name: string,
foundedOn: TimeStamp,
public: boolean,
description: string,
memberCount: number,
adminCount: number,
moderatorCount: number,
memberCountIncreaseWeekly: number,
postCountIncreaseMonthly: number,
postCountIncreaseDaily: number,
memberList: UUID[],
adminList: UUID[],
moderatorList: UUID[],
pageList: UUID[],
scrapedAt: TimeStamp,
scrapeID: UUID,
}
Random lib I found: https://github.com/ParvJain/Facebook-Group-Scraper (please look through)
Here is something to research:
Using PushShift (see example API call below) to track down discussions and automate outreach.
https://reddit-api.readthedocs.io/en/latest/#comments-search
Facebook Groups Scraper: description is being cut off because of "show more" button on FB. example groups: 208780533550355, 809393332878811
See #10
We need a page on the website with a high-level view of all the Facebook Groups.
Ideas for Views:
As it stands:
Scraping will be phased out by:
- manually
- Named Entity Recognition (NLP) from Title / Description / Posts
- image metadata (probably scrubbed by facebook?)
- members
- pages
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.