From @danfowler on April 30, 2016 11:31

• Link to http://data.okfn.org/data
• https://data.gov.uk/blog/data-packages-bulk-downloads

Copied from original issue: frictionlessdata/frictionlessdata.io#228

From @danfowler on April 30, 2016 11:20

• point to tutorials on creation

Copied from original issue: frictionlessdata/frictionlessdata.io#226

• Introduction to Open Knowledge International
• Introduction to Frictionless Data and Data Packages
  • In-depth intro to JSON Table Schema
    • Highlight missingValue, constraints, enum (as controlled vocab)
• Current Libraries/Integrations
  • Python Pandas
  • R
  • CKAN
• Current Partners
  • Dataship
  • DM4T
  • rOpenSci
  • WPRDC
• Call To Action
  • Create a Pilot with Us
  • Join Mailing List

WIP: https://docs.google.com/presentation/d/1thmb3N2s_TD-UxugZTC26yXzxP-miS9r2QMI2E9DFJM/pub?start=false&loop=false&delayms=3000

From @pwalsh on March 30, 2015 9:4

Some ideas for the current roadmap page (a brain dump): http://data.okfn.org/roadmap

• Relationship between tools/specs/community on this page, and the Tools and Standards pages proper
  • Maybe need a new object type in Jekyll that describes standard <> tooling <> community, and this can be used across site
• Tooling would be useful to group by language, and or indicate lang with icons. Tools that have CLIs should also be indicated clearly (due to potential for language agnostic use)
• Document a clear way to add tooling (user story: I just wrote a JTS validator in php - how do I add it to the list?)

Copied from original issue: frictionlessdata/frictionlessdata.io#168

From @rgrp on May 2, 2016 8:31

• Tools
  • General architecture - what components are there to a full "Data Package" stack
  • Pointers to specific languages e.g. Python, R, ...
  • How to start on a platform integration (what type of platform: publishing, consuming etc)
• Specs
  • Where to find the main specs
  • Where to raise issues
• Test suite

Copied from original issue: frictionlessdata/frictionlessdata.io#234

See: frictionlessdata/frictionlessdata.io#221

Handbook theme should link to IP policy as well as terms of use ... (and that should then be pulled through to here)

I'm booting this issue as an ongoing place to discuss some goals for v2 of the theme.

• Really nice styling for HTML tables for the inevitable multitude of simple examples of tabular data.
• There will be a fair amount of code examples, so it would be nice to have a really nice styling for code snippets.

suggestions from #10 (also see coments):

• Really concerned about "invisibility" of side menu and lack of top menu
  • For handbook this style made sense as more book like but we will need a more normal top menu here I think.
• bootstrap "boxes" stuff for e.g. /tools/ and some of /guides/ - see old http://data.okfn.org/tools/ for example
  • support fancy buttons

From @danfowler on April 30, 2016 11:11

• http://datapackagist.okfnlabs.org/

Copied from original issue: frictionlessdata/frictionlessdata.io#224

From @bnvk on May 10, 2015 15:48

Starting at this page it starts jumping into implementation details (tabular, geospatial...) kind of quickly without offering users a "quick" way to get involved immediately while in the process of understanding what they're doing bigger picture

Copied from original issue: frictionlessdata/frictionlessdata.io#182

A typical question when explaining the concept of Data Packages to people who already have some familiarity with "containerization", "metadata", and schema languages is how Data Packages compare, especially when considering the work involved in supporting an additional standard.

A comparison table listing the features supported by different formats seems like a good place to point people. In addition, it could make clear potential ways in which two standards can complement each other.

Potential comparisons to make:

• CSVW: https://www.w3.org/2013/csvw/wiki/Main_Page
• HDF5/netCDF4: https://www.hdfgroup.org/
• csv-schema: http://digital-preservation.github.io/csv-schema/csv-schema-1.0.html
• OAI-ORE: https://www.openarchives.org/ore/
• Research Objects: http://www.researchobject.org/
• BagIt: https://en.wikipedia.org/wiki/BagIt
• SQL: https://en.wikipedia.org/wiki/SQL
• XML: https://www.w3.org/XML/
• Serialization formats: Pickle (https://docs.python.org/3/library/pickle.html), Feather (https://blog.rstudio.org/2016/03/29/feather/)

Related: frictionlessdata/frictionlessdata.io#274

• Import most of the /doc/ stuff from data.okfn.org - but put refactor structure to have directory approach e.g. /doc/publish/tabular/ rather than /doc/publish-tabular/
• Fix up the bootstrap row stuff in /doc/ ... now in #10
• Merge guides with doc stuff

From @danfowler on May 6, 2016 11:9

Epics:

• Validate the data - #251
• Describe the data - describe the columns, provide general metadata - #254
• Import the data (into my tool)
• Share the data
  • Email
  • Post online
  • Publish to Wordpress, Google Drive etc

• Enrich / Merge data (join two datasets together on a single field)
• Present the data (on my website)
  • Super Simple Data Catalog
• Publish the data to a catalog / repository (to a particular platform)
  • Publish to CKAN, DSpace, etc

• Example: Data dependencies and automatically installing a lot on your machine

Where

List under http://frictionlessdata.io/user-stories/

Copied from original issue: frictionlessdata/frictionlessdata.io#245

From @rgrp on June 17, 2016 11:46

Split out from #206

• Move data.okfn.org/tools/view to viewer.frictionlessdata.io
  • implement redirect
• Move data.okfn.org/tools/validate to validate.frictionlessdata.io
  • implement redirect
• Redirect data.okfn.org/tools -> fd.io/tools
• Redirect data.okfn.org/create -> datapackagist.okfnlabs.org

Copied from original issue: frictionlessdata/frictionlessdata.io#261

From @rgrp on April 30, 2016 20:10

Overview of workflow for creating a Data Package (and associated tools and tutorials)

Rough flowchart:

• Type of data - Tabular / Geo / Any
• Are you using a library or tool or are you doing by hand
  • ? Python / R / ... =>
  • Doing it by hand => ...

This could start as a simple diagram ... - would go on /guides/publish/ page

Copied from original issue: frictionlessdata/frictionlessdata.io#229

This site uses a copy of the open data handbook. It should use a unique icon.
I'm not sure if this icon from http://data.okfn.org/data is appropriate

@danfowler I think it provides a good overview, just some comments:

• Storage libraries names are wrong: pip install jsontableschema_sql -> pip install jsontableschema-sql. Same for BigQuery.
• Badly formatted link: For more information on which fields are supported, see [the full Data Package standard][dp-spec].
• I was surprised to see how the guide jumped straight from checking the metadata to store the data on an external backend. My first though was how do I see the actual data on the Data Package object? I think it's worth having a section between "Reading basic metadata" and "Loading into an SQL database" named "Reading the data" or something which shows an example like:

print [e['name'] for e in dp.resources[0].data if int(e['atomic number']) < 10]
[u'Hydrogen', u'Helium', u'Lithium', u'Beryllium', u'Boron', u'Carbon', u'Nitrogen', u'Oxygen', u'Fluorine']

And also mentions any other form of accessing the data directly on python (like lazily with iter)

From @danfowler on April 30, 2016 11:20

This can be really simple and largely point into other tutorials e.g.

• About Tabular Data Package ...
• Python option with python lib #221
• "Raw" options - example python scripts showing how simple this is (inline with gist-it)
• datapak examples (?)

Copied from original issue: frictionlessdata/frictionlessdata.io#227

From @rgrp on December 1, 2013 20:28

• general instructions - http://data.okfn.org/publish
• tabular data #87 - http://data.okfn.org/doc/publish-tabular
• geo data #90 - http://data.okfn.org/doc/publish-geo
• any kinds of data #123 - http://data.okfn.org/doc/publish-any

Common:

• Putting your data packages online - http://data.okfn.org/doc/publish-online
• Creating DataPackage.json
• FAQs re standard conventions etc - #154

Copied from original issue: frictionlessdata/frictionlessdata.io#91

• Migrate current data.okfn.org/tools with minor refactoring
• Short summary about why tools are essential (?)
• Key online services that you an use (e.g. create, view, validate, validate data)
• Guides - using Data Packages with ...
• Tool catalog - long list. Need to add:
  • Language or Platform
  • Organisation

https://github.com/iburadempa/jts_erd/
https://github.com/iburadempa/pg_jts/

From @rgrp on May 6, 2016 11:59

Already have basics here: http://frictionlessdata.io/guides/json-table-schema

May want to add there or have separate mini-tutorials (if that is focused just on what JTS is ...)

Things to add:

• Inferring the schema - tools
• Use for validation
• Profiles / registry (if we have it)

Copied from original issue: frictionlessdata/frictionlessdata.io#248

From @rgrp on May 2, 2016 8:58

• Data management pipeline and relationship to Frictionless Data tooling
• User story overview and relation to components
• Data Package Tool Stack

High level user story + components view

https://docs.google.com/drawings/d/1uZhzgc43j8tQuN3MD5IoRdKUT192fvjDefNq85HtSU4/edit

See: https://discuss.okfn.org/t/data-package-libraries-overview/957

https://docs.google.com/drawings/d/1td7wmG54wbxCpS7kb5clPLyqESwHHTBvuxlIIRd-x5k/edit

https://docs.google.com/drawings/d/1SZxJ00WEp3iK5fqljc-yumalRePmwdaaH98_8uJYlbs/edit

https://docs.google.com/drawings/d/1W0s91bQGS-bmGOLm519mMq9zDJvRhP71pwuJtkflRws/edit

Ideas from Elsewhere

"Lego bricks"

For describing how specs or tools fit together e.g.

                Tabular Data Package
    |                                   |
---------   Data Descriptor ---------   |
Data Package | JSON Table Schema | CSV

From: https://coreos.com/

Copied from original issue: frictionlessdata/frictionlessdata.io#235

From @rgrp on January 12, 2014 12:17

a la http://package.json.nodejitsu.com/

Copied from original issue: frictionlessdata/frictionlessdata.io#94

Planning issue for describing the overall information architecture of Frictionless Data Site(s)

Note: use "fd[.io]" as shorthand for "frictionlessdata[.io]

Sites and subsites

• fd.io/ - main site, most information lives here. Detailed description below.
• specs.fd.io - specifications site - see frictionlessdata/frictionlessdata.io#242
• profiles.fd.io - profiles registry
• tools.fd.io ?? stuff like validate etc - may just have validate.fd.io

fd.io

/
# all general documentation
# thought about putting this under other items but think we want to centralise
/doc/

/data-packages/
  # maybe /standards/ - summary and link elsewhere
  /specs/
  /profiles/

/tools/

# roadmap and developer stuff - maybe merge with community
/develop/

# community stuff
/community/

# get help ...
/help/

/about/
/blog/ -> labs (or blog.okfn.org one day)

/

• Find out more
• Data Packages
• Tools
• Benefits
• Testimonials
• Get Started
• Get help

/data-packages/

• publish
• find
• use

/doc/

# index - want this to be good but maybe we can autogenerate ...
/

# guides to the specs
/data-package/
/tabular-data-package/
/geo-data-package/

# actually get started
/publish/
/publish/tabular/
/publish/geo/

# "patterns" and general howtos go here
/publish/faq/
/publish/faq/.../

# move these under /ref/ (for reference) ??
/csv/
/geojson/

/data-packages/

• what are they - simple little diagram ...
• publish, find, use - sort of data.okfn.org front page
• mainly link elsewhere ...

/tools/

Big box at top: "Have something to add: edit this page on github or submit an issue here."

Nice listing of tools. Not a table but we do want a table of contents.

Structure options:

• Function (e.g. validate, model etc)
• Platform / language

Right now suggest we structure as follows:

## Top: Set of key functions with link to best tool

## By Function
  - items in here link to below

// might just drop this section and people can look in sections below

## Integrations

## Libraries

## Services

/community/

Get involved, find people, develop the project

(other possible names: /contribute/ or /develop/)

• forum
• report issues
• chat channel
• twitter (labs)
• github (the organization) or perhaps github.com/frictionless

/help/ (or maybe /support/)

Community Support -> forum

Paid Professional Support

Professional level training and support as well as custom development is available from Open Knowledge Services team. Please see http://services.okfn.org/

/search/

/search/ - put a search box on site in navbar if possible (or sidebar)

Google custom search. Search this site, plus specs + labs blog (specific category ...?)

/about/

About page. Partners etc.

The frictionlessdata tab in the top-right doesn't render correctly in Safari. I was able to fix it by disabling the transform property. I believe it's caused by the perspective transform overriding the zindex

• move /standards/ => /data-packages/
  • Why? Because all the standards / spec stuff is about data package and this is much conceptually accurate plus better link naming for SEO
• redo page to be like front page but about data packages (??)
  • later we should replace the image and not reuse front page image
• Draw some decent diagrams - see #8 for diagram ideas
  • Current diagram is pretty poor
  • Suggest at least one lego diagram + the general diagram about separation of concerns (though not sure that is super clear and maybe should be in benefits section here or elsewhere)
• Subpages
  • specs
  • profiles

Notes

• Make JTS more prominent e.g. /data-packages/json-table-schema/

Asides:

• think about about where specs / standards go - see frictionlessdata/frictionlessdata.io#242

From @rgrp on November 24, 2013 17:40

Basics are easy here.

Real question is whether there is a recommendation on data types. Main options afaict:

• geojson (+ topojson)
• shapefiles
• csv (with lon / lat columns or similar ...)
• spatialite / sqlite style

Copied from original issue: frictionlessdata/frictionlessdata.io#90

• Enrich
• Model it
• Validate data ( data continuous integration)
• ...

Some images in img directory have a source vector e.g. http://frictionlessdata.io/img/data-package.png and we should record this where possible.

Suggest we create README.md in img and add info on sources.

Other questions:

• Should we always cache a version of the image in img even where we could direct embed from google drawings?

/cc @danfowler for comments.

From @rgrp on April 6, 2013 15:40

• R example
• (Maybe we do in other languages too later ...)

What does it demonstrate: quickly getting data and using it together

Required data

• House price data - http://data.okfn.org/data/house-prices-uk
• UK Long interest rate - http://data.okfn.org/data/bond-yields-uk-10y
• UK GDP - http://data.okfn.org/data/gdp-uk

Do we need this quarterly?

Why this analysis

• People are interested in house prices - why do they go up and down
• Classic explanation is other economic variables e.g. demand (GDP, population and interest rate (mortgages)), supply (housing stock). Here will just use demand variables and see what we find.

Copied from original issue: frictionlessdata/frictionlessdata.io#32

There are official tools and integrations that partners support and they should be tagged as such.

Main item for this frictionlessdata/frictionlessdata.io#235

Here is very specific new stuff for this site.

"Lego bricks"

For describing how specs or tools fit together e.g.

                Tabular Data Package
    |                                   |
---------   Data Descriptor ---------   |
Data Package | JSON Table Schema | CSV

From: https://coreos.com/

Read Information Architect overview first: #2

Bigger Stuff

In approximate priority order:

• /doc/- #4
  • First tutorial - on Python (Dan) - frictionlessdata/frictionlessdata.io#221
• /tools/ #5 Tools page: revert to the old approach
• /user-stories/ #7
• /data-packages/(currently /standards/) #9

Probably not this iteration:

• / - improvements to front page

Misc

• Hide "donate", "share" buttons on all pages (not really needed for this type of site) (may as well leave these ...)
• Show "edit this page" button by default (in _config.yml) and then turn off where needed
• Update the README to something sensible
• Handbook theme should link to IP policy as well as terms of use ... (and that should then be pulled through to here)
• Create sections to break up sidebar ToC a bit (??) - not a priority as will probably reflect overall architectute

From @rgrp on May 18, 2016 11:1

• Validating CSV - maybe covered by #240
• A Schema for CSV -> JSON Table Schema
  • Inferring it automatically
• Piping CSV between tools ...

Copied from original issue: frictionlessdata/frictionlessdata.io#256

• Really concerned about "invisibility" of side menu and lack of top menu
  • For handbook this style made sense as more book like but we will need a more normal top menu here I think.
• bootstrap "boxes" stuff for e.g. /tools/ and some of /guides/ - see old http://data.okfn.org/tools/ for example
  • support fancy buttons

From @rgrp on May 17, 2016 8:49

Specifications page is now http://frictionlessdata.io/data-packages/

We need to spec out this page and then implement.

Thoughts

• Better diagrams - see slides in recent deck from CSVConf
  • Tabular Data Package - extraction in progress here
  • Tooling
  • Extendable metadata
• Summarize principles and advantages
• Link to summary of specs and full specs (already have this)
• Link to FAQs / patterns / tooling / how to use

Copied from original issue: frictionlessdata/frictionlessdata.io#255

From @rgrp on April 6, 2013 15:39

More of a blog post than something on the site (publish on e.g. okfnlabs.org). Think this is pretty useful.

Walk through of turning a google spreadsheet into a data package

• publish to the web
• get public CSV url for relevant sheet
• create datapackage.json (cf #28) and publish it somewhere
  • this is an important one - should we make this part of data.okfn.org itself (cf #52 - community catalog). Simplest option would be gist or even pasting the json into a google doc and getting raw text url for that (seems quite clunky!)

cf #30 (google spreadsheet export tool)

Cost: 2-3h

Copied from original issue: frictionlessdata/frictionlessdata.io#31

From @danfowler on April 30, 2016 11:17

AKA: introduction to Data Package profiles

Focus on tabular data. Would have to hand-create JTS as there is currently no user interface. Link to tutorial on validating tabular data.

Copied from original issue: frictionlessdata/frictionlessdata.io#225

https://github.com/theodi/csvlint
https://github.com/theodi/csvlint.rb

From @rgrp on May 6, 2016 11:33

• Layout and style
  • Examples to look at
• Bounties

Cf the #234 (developer guide)

See also https://www.digitalocean.com/community/get-paid-to-write

Copied from original issue: frictionlessdata/frictionlessdata.io#247

From @rgrp on May 1, 2016 18:31

• Git and Github (assume you are familiar)
• Pushing your data ...
• Benefits
  • Versioning
  • Diffs
  • Pull requests
• Continuous Data Integration
• Gotchas - github.com/xxx/yyy !=> github.com/xxx/yyy/datapackage.json etc ...

Copied from original issue: frictionlessdata/frictionlessdata.io#233

From @danfowler on May 1, 2016 9:18

As a Data Manager I want my data publishers to create data packages for me according to a specific metadata and/or data schema so that the data I get is high quality and consistent.

As a Data Manager I want to customize/build my own data package creator like datapackagist so that I can support the previous user story.

Copied from original issue: frictionlessdata/frictionlessdata.io#232

In accordance with version 0.7.0 of datapackage-py.

From @rgrp on October 12, 2015 17:17

For a long time we've discussed "continuous data integration" and doing this with Data Packages. This is a discussion space for these ideas.

What do we mean by "Continuous Data Integration"?

Continuous Integration with code means automated running of tests (and sometimes deployments) whenever new code is pushed.

For data, this would mean automatically validating (testing) data and metadata on each new contribution and running any deployment tests.

Automatically validating a Data Package data against its schema on each commit (assuming the data was stored in git)

Sub parts

• Demo example - https://github.com/frictionlessdata/ex-continuous-data-integration
• Proper Library - frictionlessdata/example-continuous-data-integration#2
• Section in Tutorial - #240

Initial Implementation

We could start out piggy-backing on existing code CI services such as Travis: instead of running code test we run data validate tests as a code test.

By hand version

• create a demo repo - probably clone a core data package dataset
• implement a tiny test script in scripts called datatest.py or similar. Use goodtables to test the data
• turn on travis ci and run the test script
  • bonus points for somehow showing pretty output from goodtables

Standardize:

• make a simple library: goodtables-ci or similar ;-) - frictionlessdata/example-continuous-data-integration#2

Copied from original issue: frictionlessdata/frictionlessdata.io#198

From @rgrp on April 30, 2016 20:12

Supports: "Creating a Data Package" workflow #229

• Assume you have your data
• What is JSON
• Structure of a datapackage.json - see the guides, specs etc
• Tools to support you (online validators, dpm etc)

Copied from original issue: frictionlessdata/frictionlessdata.io#230