cit-ee / exploring-msa-meetup-characteristics Goto Github PK

View Code? Open in Web Editor NEW

0.0 2.0 0.0 12.25 MB

Exploring the MSAs through the lens of social events by exploiting data obtained from Meetup API

Python 5.50% Jupyter Notebook 94.50%

meetup-api census-data msa social-events pandas geocoder-library

exploring-msa-meetup-characteristics's People

Watchers

exploring-msa-meetup-characteristics's Issues

KeyError: Looks for group topics

It's not working. It seems there's an assumption in meetup_scraper.py that group.topics.names are being collected. Could you test with the four fields we decided on? These are: id, time, group.category.id, yes_rsvp. Once it passes the test, please upload the api_specs.py you used.

Building meetup location - meetup event bridge. Please wait..
Traceback (most recent call last):
  File "src\data\make_dataset.py", line 218, in <module>
    props_for[args.endpoint][args.query]['fields'], args.chkpnt_freq, args.resume)
  File "src\data\make_dataset.py", line 159, in build_meetup_events_data
    meetup_events_df.columns.tolist())
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\10-31-17\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 137, in get_df_from_nested_dicts
    flattened_dict_ls.append(_flatten_nested_data(nested_dict))
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\10-31-17\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 63, in _flatten_nested_data
    for topic in data['group']['topics']:
KeyError: 'topics'

config dir doesn't have the files

I think these two files were supposed to be in the config directory? I don't see them.

api_specs.py
resources.py

Events endpoint fails after working for a while

Failed after getting events data for over 300 locations.

Building meetup location - meetup event bridge. Please wait..
Traceback (most recent call last):
  File "src\data\make_dataset.py", line 204, in <module>
    props_for[args.endpoint][args.query]['fields'], args.chkpnt_freq, args.resume)
  File "src\data\make_dataset.py", line 146, in build_meetup_events_data
    events_data = fetch_paginated_data(events_url, None)
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 77, in fetch_paginated_data
    data = fetch_paginated_data(next_url, data)
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 77, in fetch_paginated_data
    data = fetch_paginated_data(next_url, data)
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 77, in fetch_paginated_data
    data = fetch_paginated_data(next_url, data)
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 77, in fetch_paginated_data
    data = fetch_paginated_data(next_url, data)
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 77, in fetch_paginated_data
    data = fetch_paginated_data(next_url, data)
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 77, in fetch_paginated_data
    data = fetch_paginated_data(next_url, data)
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 77, in fetch_paginated_data
    data = fetch_paginated_data(next_url, data)
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 77, in fetch_paginated_data
    data = fetch_paginated_data(next_url, data)
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 77, in fetch_paginated_data
    data = fetch_paginated_data(next_url, data)
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 77, in fetch_paginated_data
    data = fetch_paginated_data(next_url, data)
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 77, in fetch_paginated_data
    data = fetch_paginated_data(next_url, data)
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 77, in fetch_paginated_data
    data = fetch_paginated_data(next_url, data)
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 77, in fetch_paginated_data
    data = fetch_paginated_data(next_url, data)
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 77, in fetch_paginated_data
    data = fetch_paginated_data(next_url, data)
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 77, in fetch_paginated_data
    data = fetch_paginated_data(next_url, data)
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 77, in fetch_paginated_data
    data = fetch_paginated_data(next_url, data)
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 77, in fetch_paginated_data
    data = fetch_paginated_data(next_url, data)
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 77, in fetch_paginated_data
    data = fetch_paginated_data(next_url, data)
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 77, in fetch_paginated_data
    data = fetch_paginated_data(next_url, data)
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 77, in fetch_paginated_data
    data = fetch_paginated_data(next_url, data)
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 77, in fetch_paginated_data
    data = fetch_paginated_data(next_url, data)
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 77, in fetch_paginated_data
    data = fetch_paginated_data(next_url, data)
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 77, in fetch_paginated_data
    data = fetch_paginated_data(next_url, data)
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 77, in fetch_paginated_data
    data = fetch_paginated_data(next_url, data)
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 77, in fetch_paginated_data
    data = fetch_paginated_data(next_url, data)
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 77, in fetch_paginated_data
    data = fetch_paginated_data(next_url, data)
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 77, in fetch_paginated_data
    data = fetch_paginated_data(next_url, data)
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 77, in fetch_paginated_data
    data = fetch_paginated_data(next_url, data)
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 77, in fetch_paginated_data
    data = fetch_paginated_data(next_url, data)
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 77, in fetch_paginated_data
    data = fetch_paginated_data(next_url, data)
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 77, in fetch_paginated_data
    data = fetch_paginated_data(next_url, data)
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 77, in fetch_paginated_data
    data = fetch_paginated_data(next_url, data)
  File "C:\Users\mehedia\Dropbox\1_Data\CIT.ee\Meetup.com\Exploring-MSA-Meetup-Characteristics\src\data\helper_utils\meetup_scraper.py", line 64, in fetch_paginated_data
    payload, res_headers = _make_api_call(url)
TypeError: 'NoneType' object is not iterable

Relevant fields to focus on from the /find/groups endpoint

You can find the full list of properties available at the /find/groups endpoint. For my explorations I have chosen the following fields:

{
    'only': [ 'id', 'name', 'status', 'link', 'created', 'lat', 'lon', 'who', 'category.name'  ],
    'fields': [ 'past_event_count' ],
    'radius': <default>
}

Does this need to change? If so please indicate what the new structure should be!

Scrape /find/locations endpoints using city names from Cities_and_Towns.csv as query parameter

This endpoint returns data for the following fields -

city
country
localized_country_name
state
name_string
zip
lat
lon

An example:

{
"0": {
"city": "Dunbar",
"country": "us",
"localized_country_name": "USA",
"state": "WV",
"name_string": "Dunbar, West Virginia, USA",
"zip": "25064",
"lat": 38.37,
"lon": -81.74
}

There is provision for returning a subset of these fields. For now I am restricting the scraping to just lat, lon to serve our purposes. But are there any other fields that seem interesting enough to be pulled along side the ones mentioned above?

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.