sinar / popit-scripts Goto Github PK
View Code? Open in Web Editor NEWScirpts to manage import and sync data to http://popit.api.sinarproject.org
License: BSD 3-Clause "New" or "Revised" License
popit-scripts's People
Contributors
Stargazers
Watchers
popit-scripts's Issues
Question about Inconsistent use of role in Posts under Fifth Cabinet of Najib Razak
Inconsistent use of role in Posts:
- Are Deputy Ministers given "Minister" role; or "Deputy Minister" role?
- Ministers under PMO are given "Ministers" role; shouldn't it be "Minister"?
- Most role are null; is this correct
[
{
"post_id": "55b734577070042c250b332e",
"label": "Deputy Minister in the Prime Minister's Department",
"role": null
},
{
"post_id": "55b749d87070042c250c412d",
"label": "Deputy Minister of Agriculture and Agro-Based Industry",
"role": null
},
{
"post_id": "55b75b65ed2606a46a89c503",
"label": "Deputy Minister of Communications and Multimedia",
"role": null
},
{
"post_id": "55b7373866d0d137274dee07",
"label": "Deputy Minister of Defence",
"role": null
},
{
"post_id": "55b75819222b329d5ce5f94d",
"label": "Deputy Minister of Domestic Trade, Co-operatives and Consumerism",
"role": null
},
{
"post_id": "55b75a35222b329d5ce62cbd",
"label": "Deputy Minister of Education",
"role": null
},
{
"post_id": "55b7456366d0d137274e56f5",
"label": "Deputy Minister of Energy, Green Technology and Water",
"role": null
},
{
"post_id": "55b754becf2b56b826ef3e0f",
"label": "Deputy Minister of Federal Territories",
"role": null
},
{
"post_id": "55b7348e69b3d487259489da",
"label": "Deputy Minister of Finance",
"role": null
},
{
"post_id": "55b72f2469b3d48725943407",
"label": "Deputy Minister of Foreign Affairs",
"role": null
},
{
"post_id": "55b7367c7070042c250b6daa",
"label": "Deputy Minister of Health",
"role": null
},
{
"post_id": "55b755a7a19ec9ce5a550355",
"label": "Deputy Minister of Higher Education",
"role": null
},
{
"post_id": "55b735257070042c250b62dd",
"label": "Deputy Minister of Home Affairs",
"role": null
},
{
"post_id": "55b75324c4705304262e17bc",
"label": "Deputy Minister of Human Resources",
"role": null
},
{
"post_id": "55b744f9c4705304262dc62d",
"label": "Deputy Minister of International Trade and industry",
"role": null
},
{
"post_id": "55b7595eed2606a46a89c4f7",
"label": "Deputy Minister of Natural Resources and Environment",
"role": null
},
{
"post_id": "55b745d8c4705304262dc638",
"label": "Deputy Minister of Plantation Industries and Commodities",
"role": null
},
{
"post_id": "55b74c3566d0d137274e9370",
"label": "Deputy Minister of Rural and Regional Development",
"role": null
},
{
"post_id": "55b74f2b69b3d48725954700",
"label": "Deputy Minister of Science, Technology and Innovation",
"role": null
},
{
"post_id": "55b7370269b3d48725949685",
"label": "Deputy Minister of Tourism and Culture",
"role": null
},
{
"post_id": "55b7360c7070042c250b6a0e",
"label": "Deputy Minister of Transport",
"role": null
},
{
"post_id": "55b7575c027b3e0d5d28f54d",
"label": "Deputy Minister of Urban Wellbeing, Housing and Local Government",
"role": null
},
{
"post_id": "55b74fc47070042c250c7862",
"label": "Deputy Minister of Women, Family and Community Development",
"role": null
},
{
"post_id": "55b7523466d0d137274ebf0d",
"label": "Deputy Minister of Works",
"role": null
},
{
"post_id": "55b7564d1527fde65b033d34",
"label": "Deputy Minister of Youth and Sports",
"role": null
},
{
"post_id": "55b7287a9a3fd0026f667a66",
"label": "Deputy Prime Minister",
"role": "Minister"
},
{
"post_id": "55b746c369b3d48725950a7b",
"label": "Minister of Agriculture and Agro-Based Industry",
"role": null
},
{
"post_id": "55b75aed1527fde65b035d85",
"label": "Minister of Communications and Multimedia",
"role": null
},
{
"post_id": "55b731b969b3d4872594537b",
"label": "Minister of Defense",
"role": null
},
{
"post_id": "55b757ac222b329d5ce5f8c9",
"label": "Minister of Domestic Trade, Co-operatives and Consumerism",
"role": null
},
{
"post_id": "55b759a7ed2606a46a89c500",
"label": "Minister of Education",
"role": null
},
{
"post_id": "55b733087070042c250b2fa3",
"label": "Minister of Energy, Green Technology and Water",
"role": null
},
{
"post_id": "55b7546d69b3d48725956b9b",
"label": "Minister of Federal Territories",
"role": null
},
{
"post_id": "55b74b2169b3d48725952e00",
"label": "Minister of Finance",
"role": null
},
{
"post_id": "55b72ee269b3d4872594202a",
"label": "Minister of Finance II",
"role": null
},
{
"post_id": "55b72f267070042c250afb3d",
"label": "Minister of Foreign Affairs",
"role": null
},
{
"post_id": "55b72e1669b3d48725942014",
"label": "Minister of Health",
"role": null
},
{
"post_id": "55b7553bcf2b56b826ef3e13",
"label": "Minister of Higher Education",
"role": null
},
{
"post_id": "55b74ad169b3d48725952df5",
"label": "Minister of Home Affairs",
"role": null
},
{
"post_id": "55b7527366d0d137274ebf17",
"label": "Minister of Human Resources",
"role": null
},
{
"post_id": "55b7321369b3d487259479b5",
"label": "Minister of International Trade and Industry",
"role": null
},
{
"post_id": "55b7329669b3d4872594869c",
"label": "Minister of International Trade and Industry II",
"role": null
},
{
"post_id": "55b758e9222b329d5ce6228c",
"label": "Minister of Natural Resources and Environment",
"role": null
},
{
"post_id": "55b7460dc4705304262dc639",
"label": "Minister of Plantation Industries and Commodities",
"role": null
},
{
"post_id": "55b74bf2cf2b56b826eef112",
"label": "Minister of Rural and Regional Development",
"role": null
},
{
"post_id": "55b74e9bcf2b56b826ef1e5b",
"label": "Minister of Science, Technology and Innovation",
"role": null
},
{
"post_id": "55b7315e69b3d48725945370",
"label": "Minister of Tourism and Culture",
"role": null
},
{
"post_id": "55b72a492f71d383700765a4",
"label": "Minister of Transport",
"role": null
},
{
"post_id": "55b756e91527fde65b033d35",
"label": "Minister of Urban Wellbeing, Housing and Local Government",
"role": null
},
{
"post_id": "55b74f78cf2b56b826ef1e71",
"label": "Minister of Women, Family and Community Development",
"role": null
},
{
"post_id": "55b750ac69b3d487259560e2",
"label": "Minister of Works",
"role": null
},
{
"post_id": "55b75601222b329d5ce5f26a",
"label": "Minister of Youth and Sports",
"role": null
},
{
"post_id": "55b72903869c13f57048ca7e",
"label": "Ministers in the Prime Minister's Department",
"role": "Ministers"
},
{
"post_id": "55b7285a916cff2b6d55ae92",
"label": "Prime Minister",
"role": "Minister"
}
]
AQL to extract "Fifth Cabinet of Najib Razak"; with post_id added for convenience:
FOR o IN organizations
FILTER o.data.classification == "cabinet"
FILTER LIKE(LOWER(o.data.name), "%fifth%")
COLLECT posts = o.data.posts
FOR p IN posts
// FILTER p.role == null
// FILTER p.role != null
// FILTER p.label == null
// FILTER p.label != null
SORT p.label
RETURN {
post_id: p.id,
label: p.label,
role: p.role
}
Import Constituency Areas into Popit NG
MP Wong Chen still listed as Parliament Candidate
MP Wong Chen has only Posts "Parliament Candidate" attached to him. The error already exist in the old MySociety PopIt DB.
Data below; along with Membership ID for your convenience:
[
{
"mid": "55b2ffdc1965c6b14cace682",
"mrange": [
"2013",
null
],
"mrole": "Parliamentary Candidate"
},
{
"mid": "55b3000c1965c6b14cace992",
"mrange": [
"2013",
null
],
"mrole": "Parliamentary Candidate"
},
{
"mid": "55b31d5d4a4cab6f4ca15313",
"mrange": [
"2013",
null
],
"mrole": "Parliamentary Candidate"
},
{
"mid": "55b31d60e81cea364b06d62a",
"mrange": [
"2013",
null
],
"mrole": "Parliamentary Candidate"
}
]
AQL used:
FOR p IN persons
FILTER LIKE(LOWER(p.data.name),'%wong chen%')
COLLECT m = p.data.memberships[*]
FOR n IN m
RETURN {
mid: n.id,
mrange: [n.start_date, n.end_date],
mrole: n.role
}
Fix inconsistent email type values in contact-details
Update existing posts with proper area codes and roles
Membership to CSV script that flattens and organizers some key fields
CLI importer for Organization Memberships
Steps
- Import CSV
- If no Membership ID - Import
- If no Person ID - search and ask if id is correct
- if no match, add new person, add id
- if no post_id add post
Fix twitter import in contacts
Twitter contact details in MySociety Popit are stored as urls with category twitter.
For import into Sinar Popit follow recommendations of discussions on Popolo-spec issue: https://github.com/popolo-project/popolo-spec
- twitter user id to be stored as an identifier
- twitter contact_details to be stored as type url, label Twitter and as full url https://twitter.com/username
Exporter into Onodo format
Put Ministers/Deputies of Fifth Cabinet of Najib, including missing Ministries Org
For Fifth Cabinet of Najib Razak:
Missing Orgs for Ministries; only 3 listed, the rest needs to be added
http://api.popit.sinarproject.org/en/search/organizations?q=classification:government
Below; needed to link Ministry OpenSpending budgets; are all the Ministries (including existing 3 above) and the posts_id of the Ministers/Deputies/members embedded for convenience:
return [
{
"name": "Ministry of Communications and Multimedia",
"members": [{
"post_id": "55b75b65ed2606a46a89c503",
"post_name": "Deputy Minister of Communications and Multimedia"
}, {
"post_id": "55b75aed1527fde65b035d85",
"post_name": "Minister of Communications and Multimedia"
}]
},
{
"name": "Ministry of Education",
"members": [{
"post_id": "55b75a35222b329d5ce62cbd",
"post_name": "Deputy Minister of Education"
}, {
"post_id": "55b759a7ed2606a46a89c500",
"post_name": "Minister of Education"
}]
},
{
"name": "Ministry of Natural Resources and Environment",
"members": [{
"post_id": "55b7595eed2606a46a89c4f7",
"post_name": "Deputy Minister of Natural Resources and Environment"
}, {
"post_id": "55b758e9222b329d5ce6228c",
"post_name": "Minister of Natural Resources and Environment"
}]
},
{
"name": "Ministry of Domestic Trade, Co-operatives and Consumerism",
"members": [{
"post_id": "55b75819222b329d5ce5f94d",
"post_name": "Deputy Minister of Domestic Trade, Co-operatives and Consumerism"
}, {
"post_id": "55b757ac222b329d5ce5f8c9",
"post_name": "Minister of Domestic Trade, Co-operatives and Consumerism"
}]
},
{
"name": "Ministry of Urban Wellbeing, Housing and Local Government",
"members": [{
"post_id": "55b7575c027b3e0d5d28f54d",
"post_name": "Deputy Minister of Urban Wellbeing, Housing and Local Government"
}, {
"post_id": "55b756e91527fde65b033d35",
"post_name": "Minister of Urban Wellbeing, Housing and Local Government"
}]
},
{
"name": "Ministry of Youth and Sports",
"members": [{
"post_id": "55b7564d1527fde65b033d34",
"post_name": "Deputy Minister of Youth and Sports"
}, {
"post_id": "55b75601222b329d5ce5f26a",
"post_name": "Minister of Youth and Sports"
}]
},
{
"name": "Ministry of Higher Education",
"members": [{
"post_id": "55b755a7a19ec9ce5a550355",
"post_name": "Deputy Minister of Higher Education"
}, {
"post_id": "55b7553bcf2b56b826ef3e13",
"post_name": "Minister of Higher Education"
}]
},
{
"name": "Ministry of Federal Territories",
"members": [{
"post_id": "55b754becf2b56b826ef3e0f",
"post_name": "Deputy Minister of Federal Territories"
}, {
"post_id": "55b7546d69b3d48725956b9b",
"post_name": "Minister of Federal Territories"
}]
},
{
"name": "Ministry of Human Resources",
"members": [{
"post_id": "55b75324c4705304262e17bc",
"post_name": "Deputy Minister of Human Resources"
}, {
"post_id": "55b7527366d0d137274ebf17",
"post_name": "Minister of Human Resources"
}]
},
{
"name": "Ministry of Works",
"members": [{
"post_id": "55b7523466d0d137274ebf0d",
"post_name": "Deputy Minister of Works"
}, {
"post_id": "55b750ac69b3d487259560e2",
"post_name": "Minister of Works"
}]
},
{
"name": "Ministry of Women, Family and Community Development",
"members": [{
"post_id": "55b74fc47070042c250c7862",
"post_name": "Deputy Minister of Women, Family and Community Development"
}, {
"post_id": "55b74f78cf2b56b826ef1e71",
"post_name": "Minister of Women, Family and Community Development"
}]
},
{
"name": "Ministry of Science, Technology and Innovation",
"members": [{
"post_id": "55b74f2b69b3d48725954700",
"post_name": "Deputy Minister of Science, Technology and Innovation"
}, {
"post_id": "55b74e9bcf2b56b826ef1e5b",
"post_name": "Minister of Science, Technology and Innovation"
}]
},
{
"name": "Ministry of Rural and Regional Development",
"members": [{
"post_id": "55b74c3566d0d137274e9370",
"post_name": "Deputy Minister of Rural and Regional Development"
}, {
"post_id": "55b74bf2cf2b56b826eef112",
"post_name": "Minister of Rural and Regional Development"
}]
},
{
"name": "Ministry of Finance",
"members": [{
"post_id": "55b72ee269b3d4872594202a",
"post_name": "Minister of Finance II"
}, {
"post_id": "55b74b2169b3d48725952e00",
"post_name": "Minister of Finance"
}]
},
{
"name": "Ministry of Home Affairs",
"members": [{
"post_id": "55b735257070042c250b62dd",
"post_name": "Deputy Minister of Home Affairs"
}, {
"post_id": "55b74ad169b3d48725952df5",
"post_name": "Minister of Home Affairs"
}]
},
{
"name": "Ministry of Agriculture and Agro-Based Industry",
"members": [{
"post_id": "55b749d87070042c250c412d",
"post_name": "Deputy Minister of Agriculture and Agro-Based Industry"
}, {
"post_id": "55b746c369b3d48725950a7b",
"post_name": "Minister of Agriculture and Agro-Based Industry"
}]
},
{
"name": "Ministry of Plantation Industries and Commodities",
"members": [{
"post_id": "55b7460dc4705304262dc639",
"post_name": "Minister of Plantation Industries and Commodities"
}, {
"post_id": "55b745d8c4705304262dc638",
"post_name": "Deputy Minister of Plantation Industries and Commodities"
}]
},
{
"name": "Ministry of Energy, Green Technology and Water",
"members": [{
"post_id": "55b7456366d0d137274e56f5",
"post_name": "Deputy Minister of Energy, Green Technology and Water"
}, {
"post_id": "55b733087070042c250b2fa3",
"post_name": "Minister of Energy, Green Technology and Water"
}]
},
{
"name": "Ministry of International Trade and Industry",
"members": [{
"post_id": "55b7321369b3d487259479b5",
"post_name": "Minister of International Trade and Industry"
}, {
"post_id": "55b7329669b3d4872594869c",
"post_name": "Minister of International Trade and Industry II"
}]
},
{
"name": "Ministry of Tourism and Culture",
"members": [{
"post_id": "55b7370269b3d48725949685",
"post_name": "Deputy Minister of Tourism and Culture"
}, {
"post_id": "55b7315e69b3d48725945370",
"post_name": "Minister of Tourism and Culture"
}]
},
{
"name": "Ministry of Foreign Affairs",
"members": [{
"post_id": "55b72f267070042c250afb3d",
"post_name": "Minister of Foreign Affairs"
}, {
"post_id": "55b72f2469b3d48725943407",
"post_name": "Deputy Minister of Foreign Affairs"
}]
},
{
"name": "Ministry of Defense",
"members": [{
"post_id": "55b7373866d0d137274dee07",
"post_name": "Deputy Minister of Defence"
}, {
"post_id": "55b731b969b3d4872594537b",
"post_name": "Minister of Defence"
}]
},
{
"name": "Ministry of Health",
"members": [{
"post_id": "55b7367c7070042c250b6daa",
"post_name": "Deputy Minister of Health"
}, {
"post_id": "55b72e1669b3d48725942014",
"post_name": "Minister of Health"
}]
},
{
"name": "Ministry of Transport",
"members": [{
"post_id": "55b7360c7070042c250b6a0e",
"post_name": "Deputy Minister of Transport"
}, {
"post_id": "55b72a492f71d383700765a4",
"post_name": "Minister of Transport"
}]
},
{
"name": "Prime Minister's Department",
"members": [{
"post_id": "55b734577070042c250b332e",
"post_name": "Deputy Minister in the Prime Minister's Department"
}, {
"post_id": "55b72903869c13f57048ca7e",
"post_name": "Ministers in the Prime Minister's Department"
}]
}
]
NOTE: If above form is not convenient for import; do let me know the preferred format.
Cabinet Ministers missing start date
The following Cabinet Ministers are missing their start date. Membership ID included for your convenience.
reported by michael leow Sinar/popit_ng#157
[
{
"mid": "55b7534cc4705304262e17c3",
"myname": "Ismail Bin Abd. Muttalib",
"myportfolio": "Deputy Minister of Human Resources",
"myrange": [
null,
null
]
},
{
"mid": "55b72d2d7070042c250af543",
"myname": "Joseph Entulu Belaun",
"myportfolio": "Ministers in the Prime Minister's Department",
"myrange": [
null,
null
]
}
]
AQL
FOR p IN posts
FILTER LIKE(LOWER(p.data.label),'%minister%')
FOR m IN memberships
FILTER m.data.post.id == p._key
FILTER m.data.start_date == null
SORT p.data.label
RETURN {
mid: m._key,
myname: m.data.person.name,
myportfolio: p.data.label,
myrange: [ m.data.start_date, m.data.end_date ]
}
Import 6th Cabinet of Najib Razak
Fields for List of Ministers
When generating lists of ministers, the following user requirements are asked for.
-
gender
-
state of origin (because there are state leaders that are sensitive about whether their state "has a minister" in the cabinet)
-
is the minister
a) an MP, or
b) Senator appointed by the State, or
c) Senator appointed by the PM, or
d) none of the above. -
is the Minister also an ADUN? Which constituency? (to show potential conflict of interest as a lawmaker and policymaker at the Federal level)
Requested by Adriene Leong, Bar Council
Add on behalf of id's for Dewan Rakyat
Update and fix existing MP memberships
Generate list of current MP memberships.
- 2 MPs have passed away
- Fix errors, missing MPs
Import Parliamentary Areas 1-222
Script to sync with data from Everypolitician.org
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.