sinar / popit-scripts Goto Github PK
View Code? Open in Web Editor NEWScirpts to manage import and sync data to http://popit.api.sinarproject.org
License: BSD 3-Clause "New" or "Revised" License
Scirpts to manage import and sync data to http://popit.api.sinarproject.org
License: BSD 3-Clause "New" or "Revised" License
Inconsistent use of role in Posts:
[
{
"post_id": "55b734577070042c250b332e",
"label": "Deputy Minister in the Prime Minister's Department",
"role": null
},
{
"post_id": "55b749d87070042c250c412d",
"label": "Deputy Minister of Agriculture and Agro-Based Industry",
"role": null
},
{
"post_id": "55b75b65ed2606a46a89c503",
"label": "Deputy Minister of Communications and Multimedia",
"role": null
},
{
"post_id": "55b7373866d0d137274dee07",
"label": "Deputy Minister of Defence",
"role": null
},
{
"post_id": "55b75819222b329d5ce5f94d",
"label": "Deputy Minister of Domestic Trade, Co-operatives and Consumerism",
"role": null
},
{
"post_id": "55b75a35222b329d5ce62cbd",
"label": "Deputy Minister of Education",
"role": null
},
{
"post_id": "55b7456366d0d137274e56f5",
"label": "Deputy Minister of Energy, Green Technology and Water",
"role": null
},
{
"post_id": "55b754becf2b56b826ef3e0f",
"label": "Deputy Minister of Federal Territories",
"role": null
},
{
"post_id": "55b7348e69b3d487259489da",
"label": "Deputy Minister of Finance",
"role": null
},
{
"post_id": "55b72f2469b3d48725943407",
"label": "Deputy Minister of Foreign Affairs",
"role": null
},
{
"post_id": "55b7367c7070042c250b6daa",
"label": "Deputy Minister of Health",
"role": null
},
{
"post_id": "55b755a7a19ec9ce5a550355",
"label": "Deputy Minister of Higher Education",
"role": null
},
{
"post_id": "55b735257070042c250b62dd",
"label": "Deputy Minister of Home Affairs",
"role": null
},
{
"post_id": "55b75324c4705304262e17bc",
"label": "Deputy Minister of Human Resources",
"role": null
},
{
"post_id": "55b744f9c4705304262dc62d",
"label": "Deputy Minister of International Trade and industry",
"role": null
},
{
"post_id": "55b7595eed2606a46a89c4f7",
"label": "Deputy Minister of Natural Resources and Environment",
"role": null
},
{
"post_id": "55b745d8c4705304262dc638",
"label": "Deputy Minister of Plantation Industries and Commodities",
"role": null
},
{
"post_id": "55b74c3566d0d137274e9370",
"label": "Deputy Minister of Rural and Regional Development",
"role": null
},
{
"post_id": "55b74f2b69b3d48725954700",
"label": "Deputy Minister of Science, Technology and Innovation",
"role": null
},
{
"post_id": "55b7370269b3d48725949685",
"label": "Deputy Minister of Tourism and Culture",
"role": null
},
{
"post_id": "55b7360c7070042c250b6a0e",
"label": "Deputy Minister of Transport",
"role": null
},
{
"post_id": "55b7575c027b3e0d5d28f54d",
"label": "Deputy Minister of Urban Wellbeing, Housing and Local Government",
"role": null
},
{
"post_id": "55b74fc47070042c250c7862",
"label": "Deputy Minister of Women, Family and Community Development",
"role": null
},
{
"post_id": "55b7523466d0d137274ebf0d",
"label": "Deputy Minister of Works",
"role": null
},
{
"post_id": "55b7564d1527fde65b033d34",
"label": "Deputy Minister of Youth and Sports",
"role": null
},
{
"post_id": "55b7287a9a3fd0026f667a66",
"label": "Deputy Prime Minister",
"role": "Minister"
},
{
"post_id": "55b746c369b3d48725950a7b",
"label": "Minister of Agriculture and Agro-Based Industry",
"role": null
},
{
"post_id": "55b75aed1527fde65b035d85",
"label": "Minister of Communications and Multimedia",
"role": null
},
{
"post_id": "55b731b969b3d4872594537b",
"label": "Minister of Defense",
"role": null
},
{
"post_id": "55b757ac222b329d5ce5f8c9",
"label": "Minister of Domestic Trade, Co-operatives and Consumerism",
"role": null
},
{
"post_id": "55b759a7ed2606a46a89c500",
"label": "Minister of Education",
"role": null
},
{
"post_id": "55b733087070042c250b2fa3",
"label": "Minister of Energy, Green Technology and Water",
"role": null
},
{
"post_id": "55b7546d69b3d48725956b9b",
"label": "Minister of Federal Territories",
"role": null
},
{
"post_id": "55b74b2169b3d48725952e00",
"label": "Minister of Finance",
"role": null
},
{
"post_id": "55b72ee269b3d4872594202a",
"label": "Minister of Finance II",
"role": null
},
{
"post_id": "55b72f267070042c250afb3d",
"label": "Minister of Foreign Affairs",
"role": null
},
{
"post_id": "55b72e1669b3d48725942014",
"label": "Minister of Health",
"role": null
},
{
"post_id": "55b7553bcf2b56b826ef3e13",
"label": "Minister of Higher Education",
"role": null
},
{
"post_id": "55b74ad169b3d48725952df5",
"label": "Minister of Home Affairs",
"role": null
},
{
"post_id": "55b7527366d0d137274ebf17",
"label": "Minister of Human Resources",
"role": null
},
{
"post_id": "55b7321369b3d487259479b5",
"label": "Minister of International Trade and Industry",
"role": null
},
{
"post_id": "55b7329669b3d4872594869c",
"label": "Minister of International Trade and Industry II",
"role": null
},
{
"post_id": "55b758e9222b329d5ce6228c",
"label": "Minister of Natural Resources and Environment",
"role": null
},
{
"post_id": "55b7460dc4705304262dc639",
"label": "Minister of Plantation Industries and Commodities",
"role": null
},
{
"post_id": "55b74bf2cf2b56b826eef112",
"label": "Minister of Rural and Regional Development",
"role": null
},
{
"post_id": "55b74e9bcf2b56b826ef1e5b",
"label": "Minister of Science, Technology and Innovation",
"role": null
},
{
"post_id": "55b7315e69b3d48725945370",
"label": "Minister of Tourism and Culture",
"role": null
},
{
"post_id": "55b72a492f71d383700765a4",
"label": "Minister of Transport",
"role": null
},
{
"post_id": "55b756e91527fde65b033d35",
"label": "Minister of Urban Wellbeing, Housing and Local Government",
"role": null
},
{
"post_id": "55b74f78cf2b56b826ef1e71",
"label": "Minister of Women, Family and Community Development",
"role": null
},
{
"post_id": "55b750ac69b3d487259560e2",
"label": "Minister of Works",
"role": null
},
{
"post_id": "55b75601222b329d5ce5f26a",
"label": "Minister of Youth and Sports",
"role": null
},
{
"post_id": "55b72903869c13f57048ca7e",
"label": "Ministers in the Prime Minister's Department",
"role": "Ministers"
},
{
"post_id": "55b7285a916cff2b6d55ae92",
"label": "Prime Minister",
"role": "Minister"
}
]
AQL to extract "Fifth Cabinet of Najib Razak"; with post_id added for convenience:
FOR o IN organizations
FILTER o.data.classification == "cabinet"
FILTER LIKE(LOWER(o.data.name), "%fifth%")
COLLECT posts = o.data.posts
FOR p IN posts
// FILTER p.role == null
// FILTER p.role != null
// FILTER p.label == null
// FILTER p.label != null
SORT p.label
RETURN {
post_id: p.id,
label: p.label,
role: p.role
}
MP Wong Chen has only Posts "Parliament Candidate" attached to him. The error already exist in the old MySociety PopIt DB.
Data below; along with Membership ID for your convenience:
[
{
"mid": "55b2ffdc1965c6b14cace682",
"mrange": [
"2013",
null
],
"mrole": "Parliamentary Candidate"
},
{
"mid": "55b3000c1965c6b14cace992",
"mrange": [
"2013",
null
],
"mrole": "Parliamentary Candidate"
},
{
"mid": "55b31d5d4a4cab6f4ca15313",
"mrange": [
"2013",
null
],
"mrole": "Parliamentary Candidate"
},
{
"mid": "55b31d60e81cea364b06d62a",
"mrange": [
"2013",
null
],
"mrole": "Parliamentary Candidate"
}
]
AQL used:
FOR p IN persons
FILTER LIKE(LOWER(p.data.name),'%wong chen%')
COLLECT m = p.data.memberships[*]
FOR n IN m
RETURN {
mid: n.id,
mrange: [n.start_date, n.end_date],
mrole: n.role
}
Steps
Twitter contact details in MySociety Popit are stored as urls with category twitter.
For import into Sinar Popit follow recommendations of discussions on Popolo-spec issue: https://github.com/popolo-project/popolo-spec
For Fifth Cabinet of Najib Razak:
Missing Orgs for Ministries; only 3 listed, the rest needs to be added
http://api.popit.sinarproject.org/en/search/organizations?q=classification:government
Below; needed to link Ministry OpenSpending budgets; are all the Ministries (including existing 3 above) and the posts_id of the Ministers/Deputies/members embedded for convenience:
return [
{
"name": "Ministry of Communications and Multimedia",
"members": [{
"post_id": "55b75b65ed2606a46a89c503",
"post_name": "Deputy Minister of Communications and Multimedia"
}, {
"post_id": "55b75aed1527fde65b035d85",
"post_name": "Minister of Communications and Multimedia"
}]
},
{
"name": "Ministry of Education",
"members": [{
"post_id": "55b75a35222b329d5ce62cbd",
"post_name": "Deputy Minister of Education"
}, {
"post_id": "55b759a7ed2606a46a89c500",
"post_name": "Minister of Education"
}]
},
{
"name": "Ministry of Natural Resources and Environment",
"members": [{
"post_id": "55b7595eed2606a46a89c4f7",
"post_name": "Deputy Minister of Natural Resources and Environment"
}, {
"post_id": "55b758e9222b329d5ce6228c",
"post_name": "Minister of Natural Resources and Environment"
}]
},
{
"name": "Ministry of Domestic Trade, Co-operatives and Consumerism",
"members": [{
"post_id": "55b75819222b329d5ce5f94d",
"post_name": "Deputy Minister of Domestic Trade, Co-operatives and Consumerism"
}, {
"post_id": "55b757ac222b329d5ce5f8c9",
"post_name": "Minister of Domestic Trade, Co-operatives and Consumerism"
}]
},
{
"name": "Ministry of Urban Wellbeing, Housing and Local Government",
"members": [{
"post_id": "55b7575c027b3e0d5d28f54d",
"post_name": "Deputy Minister of Urban Wellbeing, Housing and Local Government"
}, {
"post_id": "55b756e91527fde65b033d35",
"post_name": "Minister of Urban Wellbeing, Housing and Local Government"
}]
},
{
"name": "Ministry of Youth and Sports",
"members": [{
"post_id": "55b7564d1527fde65b033d34",
"post_name": "Deputy Minister of Youth and Sports"
}, {
"post_id": "55b75601222b329d5ce5f26a",
"post_name": "Minister of Youth and Sports"
}]
},
{
"name": "Ministry of Higher Education",
"members": [{
"post_id": "55b755a7a19ec9ce5a550355",
"post_name": "Deputy Minister of Higher Education"
}, {
"post_id": "55b7553bcf2b56b826ef3e13",
"post_name": "Minister of Higher Education"
}]
},
{
"name": "Ministry of Federal Territories",
"members": [{
"post_id": "55b754becf2b56b826ef3e0f",
"post_name": "Deputy Minister of Federal Territories"
}, {
"post_id": "55b7546d69b3d48725956b9b",
"post_name": "Minister of Federal Territories"
}]
},
{
"name": "Ministry of Human Resources",
"members": [{
"post_id": "55b75324c4705304262e17bc",
"post_name": "Deputy Minister of Human Resources"
}, {
"post_id": "55b7527366d0d137274ebf17",
"post_name": "Minister of Human Resources"
}]
},
{
"name": "Ministry of Works",
"members": [{
"post_id": "55b7523466d0d137274ebf0d",
"post_name": "Deputy Minister of Works"
}, {
"post_id": "55b750ac69b3d487259560e2",
"post_name": "Minister of Works"
}]
},
{
"name": "Ministry of Women, Family and Community Development",
"members": [{
"post_id": "55b74fc47070042c250c7862",
"post_name": "Deputy Minister of Women, Family and Community Development"
}, {
"post_id": "55b74f78cf2b56b826ef1e71",
"post_name": "Minister of Women, Family and Community Development"
}]
},
{
"name": "Ministry of Science, Technology and Innovation",
"members": [{
"post_id": "55b74f2b69b3d48725954700",
"post_name": "Deputy Minister of Science, Technology and Innovation"
}, {
"post_id": "55b74e9bcf2b56b826ef1e5b",
"post_name": "Minister of Science, Technology and Innovation"
}]
},
{
"name": "Ministry of Rural and Regional Development",
"members": [{
"post_id": "55b74c3566d0d137274e9370",
"post_name": "Deputy Minister of Rural and Regional Development"
}, {
"post_id": "55b74bf2cf2b56b826eef112",
"post_name": "Minister of Rural and Regional Development"
}]
},
{
"name": "Ministry of Finance",
"members": [{
"post_id": "55b72ee269b3d4872594202a",
"post_name": "Minister of Finance II"
}, {
"post_id": "55b74b2169b3d48725952e00",
"post_name": "Minister of Finance"
}]
},
{
"name": "Ministry of Home Affairs",
"members": [{
"post_id": "55b735257070042c250b62dd",
"post_name": "Deputy Minister of Home Affairs"
}, {
"post_id": "55b74ad169b3d48725952df5",
"post_name": "Minister of Home Affairs"
}]
},
{
"name": "Ministry of Agriculture and Agro-Based Industry",
"members": [{
"post_id": "55b749d87070042c250c412d",
"post_name": "Deputy Minister of Agriculture and Agro-Based Industry"
}, {
"post_id": "55b746c369b3d48725950a7b",
"post_name": "Minister of Agriculture and Agro-Based Industry"
}]
},
{
"name": "Ministry of Plantation Industries and Commodities",
"members": [{
"post_id": "55b7460dc4705304262dc639",
"post_name": "Minister of Plantation Industries and Commodities"
}, {
"post_id": "55b745d8c4705304262dc638",
"post_name": "Deputy Minister of Plantation Industries and Commodities"
}]
},
{
"name": "Ministry of Energy, Green Technology and Water",
"members": [{
"post_id": "55b7456366d0d137274e56f5",
"post_name": "Deputy Minister of Energy, Green Technology and Water"
}, {
"post_id": "55b733087070042c250b2fa3",
"post_name": "Minister of Energy, Green Technology and Water"
}]
},
{
"name": "Ministry of International Trade and Industry",
"members": [{
"post_id": "55b7321369b3d487259479b5",
"post_name": "Minister of International Trade and Industry"
}, {
"post_id": "55b7329669b3d4872594869c",
"post_name": "Minister of International Trade and Industry II"
}]
},
{
"name": "Ministry of Tourism and Culture",
"members": [{
"post_id": "55b7370269b3d48725949685",
"post_name": "Deputy Minister of Tourism and Culture"
}, {
"post_id": "55b7315e69b3d48725945370",
"post_name": "Minister of Tourism and Culture"
}]
},
{
"name": "Ministry of Foreign Affairs",
"members": [{
"post_id": "55b72f267070042c250afb3d",
"post_name": "Minister of Foreign Affairs"
}, {
"post_id": "55b72f2469b3d48725943407",
"post_name": "Deputy Minister of Foreign Affairs"
}]
},
{
"name": "Ministry of Defense",
"members": [{
"post_id": "55b7373866d0d137274dee07",
"post_name": "Deputy Minister of Defence"
}, {
"post_id": "55b731b969b3d4872594537b",
"post_name": "Minister of Defence"
}]
},
{
"name": "Ministry of Health",
"members": [{
"post_id": "55b7367c7070042c250b6daa",
"post_name": "Deputy Minister of Health"
}, {
"post_id": "55b72e1669b3d48725942014",
"post_name": "Minister of Health"
}]
},
{
"name": "Ministry of Transport",
"members": [{
"post_id": "55b7360c7070042c250b6a0e",
"post_name": "Deputy Minister of Transport"
}, {
"post_id": "55b72a492f71d383700765a4",
"post_name": "Minister of Transport"
}]
},
{
"name": "Prime Minister's Department",
"members": [{
"post_id": "55b734577070042c250b332e",
"post_name": "Deputy Minister in the Prime Minister's Department"
}, {
"post_id": "55b72903869c13f57048ca7e",
"post_name": "Ministers in the Prime Minister's Department"
}]
}
]
NOTE: If above form is not convenient for import; do let me know the preferred format.
The following Cabinet Ministers are missing their start date. Membership ID included for your convenience.
reported by michael leow Sinar/popit_ng#157
[
{
"mid": "55b7534cc4705304262e17c3",
"myname": "Ismail Bin Abd. Muttalib",
"myportfolio": "Deputy Minister of Human Resources",
"myrange": [
null,
null
]
},
{
"mid": "55b72d2d7070042c250af543",
"myname": "Joseph Entulu Belaun",
"myportfolio": "Ministers in the Prime Minister's Department",
"myrange": [
null,
null
]
}
]
AQL
FOR p IN posts
FILTER LIKE(LOWER(p.data.label),'%minister%')
FOR m IN memberships
FILTER m.data.post.id == p._key
FILTER m.data.start_date == null
SORT p.data.label
RETURN {
mid: m._key,
myname: m.data.person.name,
myportfolio: p.data.label,
myrange: [ m.data.start_date, m.data.end_date ]
}
When generating lists of ministers, the following user requirements are asked for.
gender
state of origin (because there are state leaders that are sensitive about whether their state "has a minister" in the cabinet)
is the minister
a) an MP, or
b) Senator appointed by the State, or
c) Senator appointed by the PM, or
d) none of the above.
is the Minister also an ADUN? Which constituency? (to show potential conflict of interest as a lawmaker and policymaker at the Federal level)
Requested by Adriene Leong, Bar Council
Generate list of current MP memberships.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.