Click here too see graph visualizations of StackExchange.

See also: TagOverflow.

I wrote scripts generating a map of topics from StackExchange sites (e.g. StackOverflow), in form of a graph of tags. Started as an entry for StackExchange visualization competition at Kaggle.

If you like pictures, visit wiki for this GitHub project. However, if you want to read the documentation - read below.

To do:

• interactive d3js graphs
• plots for Area51
• automated plots

Current state:

• with queries from SE Data Explorer (but it works for any other csv tables for any other tags, as long as it is in the same form)
• with API scrapers to get tags from beta sites and to make a map of the StackExchange network
• further development moved to TagOverflow - an interactive tag visualization in d3.js

==============================

data.stackexchange.com -> csv -> oetable2graphml.py -> graphml -> gephi -> pdf

• To get data, use a data.SE query to obtain table of tag co-occurrences; my other queries

• Run oetable2graphml.py to convert it to graphml file (requires NetworkX), e.g.

python oetable2graphml.py input.csv output.graphml

• Use Gephi to import graphml file and process it to your taste.
  E.g. (on Gephi 0.8.1 beta):

• Overview tab:
• Ranking -> Nodes -> Size -> weight -> Min:15, Max:30, Spline:3 -> Run
  (optimal options may vary)
• Layout -> ARF -> Run
  OR: Layout -> Force Atlas -> Run; Layout -> Fruchterman Reingold -> Run
  (and you may like to experiment with parameters or other methods)
• Layout -> Noverlap -> Run
• Statistics -> Modularity -> Run
• Partition -> Nodes -> Refresh -> Modularity Class -> Apply
  (and optionally choosing colors to your taste)
• Font size: 26pt, Node size, Show node labels
• Layout -> Label Adjust -> Run
• Preview tab:
• Nodes -> Border Color: #A0A0A0
• Node Labels -> Show Labels: True, Font: 4pt
• Edges -> Opacity: 40.0
• Refresh; Export

First, obtain tag bundles with SE API, e.g. se-api-py, e.g. doing:

x = se.fetch("questions", site="biology", filter="!nR5-WLw0-5")  # filter says that we ask only for the 'tags' field
t = [y['tags'] for y in x]

You need to have list of list with tags per post, e.g.

t = [["plants", "flowers"], ["plants", "carnivorous", "big-list"], ["carnivorous", "fish", "piranha"]]

Then process it e.g. in that way:

import tag_bundle_processing as tbp
bun = tbp.Bundle(t) 
# or: bun = tbp.Bundle(json_path="data.json")

bun.filter_elements(first_n=32)  # takes only 32 most frequent tags
bun.calculate_pair_weights(self, func=oe_ratio, threshold=1.5)
bun.export2graphml("path/to/file.graphml")

And then proceed use Gephi as for mature SE sites.