Code Monkey home page Code Monkey logo

metalicious's Introduction

README

Metalicious is an open-source, web-based data dictionary that is designed to capture and display metadata from databases, tables, and fields for platforms with one or more databases. Users may customize and deploy Metalicious as an internal or public data dictionary website.

This project is a generic release of the City of Chicago's data dictionary.

Installation

Copy the repository or download a compressed folder and unpack the directory to the folder you wish to host Metalicious.

Importing Database

Using the command line, run the following script:

mysql -u db-username -p -h yourlocaldatabase < database/Metalicious_DB.sql

Metalicious will create a new database called "datadictionary". Ensure no database with the same name exists as Metalicious will overwrite pre-existing data.

Creating Users

There is not a way to enter new users through the website in the current version. Adding new users will require access to the MySQL server and database. Use the following script to add new users:

INSERT INTO Users
	(User_Name, Password, First_Name, Last_Name)
VALUES
	('_username-to-be-added_', '_password_', '_user-first-name_', '_user-last-name_');

Where the _underscored_ arguments will be inserted into the user account.

Creating Business Functions

"Business Functions" is used to create categories to help users navigate to data dictionary entries. Currently, Metalicious does not support adding business functions through the website. Instead, new functions can be added through the following commands:

INSERT INTO Business_Functions
	(Business_Function_Name, Business_Function_Description)
VALUES
	('_business-function-to-be-added_', '_business-function-description_');

The _underscored_ entries may be modified to fit your needs.

Deploying Website

Move the contents of /web to the directory you wish to deploy Metalicious.

After moving the contents, configure /include/dbconnopen.php to the appropriate MySQL database and login information:

$cnnCDD = mysqli_connect("_yourlocaldatabase_", "_db-username_", "_db-password_")

The deployment of Metalicious assumes it will be installed in the root directory of the server (e.g., DOCUMENT_ROOT). To change this behavior, modify the file location.php

Editing Contact Form

Metalicious includes a contact form for users to contact a central location. You can edit the contact information in ajax/contact_send.php

$to = "\"_email-name_\" <[email protected]>";
$subject = "Metalicious Data Dictionary: Online Comment / Question / Suggestion";

Adding Data

Adding a new system or revising an existing system

  1. Log into website Username: beginning of e+mail address (i.e. jsmith – do not include @gmail.com) Password: same as username
  2. Visit example.com/database_info.php, where example.com is the directory hosting Metalicious. Click "Create New DB" Note: If a database has been created, you may open an existing database to create a new database
  3. Enter all relevant information on the system
  4. Select “Create New Database” once all information has been entered Note: This will create the database, but it will not yet appear on the website
  5. Activate the database Click on your name at the top of the screen and select “Admin” from the drop down Click on the database and then select “Activate Revision” The system will now appear on the website. Note: The database will need to be activated in order for it to be complete and appear in the website as well as for you to add the business function.
  6. Adding the Business Function Use the “Search” feature to open up the new database At the end of the database detail table, select the business function where this database belongs Follow step 7 to update the revision
  7. Making Revisions to a database Click “Change Info” at the end of the database detail
  • Click “Save Revisions” Click “Database Revisions” and click “Load this revision” next to the appropriate dated revision.

Importing a table

  1. Steps to take to get file ready for importing
  • See examples of the table file derived from the City of Chicago Congregate Dining Meal Application
  • Copy all relevant fields (system, table, column/field, type, length, value range, description, examples, comments) with no header row and paste into a separate excel file and save as CSV (comma delimited) file.
  • The System name in the file must match exactly how the system is named in the website. For example, if the system is titled “Attendance System” in the website, the spreadsheet must have “Attendance System” listed under the system column.
  1. Log into website
  2. Select “import” from the drop down list under your name
  3. Select the “browse” button under the second option - Choose your Tables.csv file and then click on the submit button
  4. Add the appropriate variable CSV file
  5. Select “submit”

Importing a variable

  1. Steps to take to get file ready for importing
  • See examples of the variable file derived from the City of Chicago Congregate Dining Meal Application
  • Copy all relevant fields (system, table, column/field, type, length, value range, description, examples, comments) with no header row and paste into a separate excel file and save as CSV (comma delimited) file.
  • The System name in the file must match exactly how the system is named in the website. For example, if the system is titled “Attendance System” in the website, the spreadsheet must have “Attendance System” listed under the system column.
  1. Log into website
  2. Select “import” from the drop down list under your name
  3. Select the “browse” button under the second option - Choose your Variables.csv file and then click on the submit button
  4. Add the appropriate variable CSV file
  5. Select “submit”

Requirements

Metalicious has not gone through broad deployment testing. Below are the specifications which this project was built. We appreciate feedback on other platforms which Metalicious does and does not operate.

System Requirements

Severs hosting this project requires. This project was developed with the following specifications:

  • MySQL Community Server 5.5.25a
  • PHP 5.4.5

License & Support

Metalicious is released under an MIT-style license. Portions of the design of the website uses Font Awesome in the design, which is licensed under CC BY 3.0. Other licensed components of this project are compatible with an MIT license. The complete license is contained in the LICENSE file at the root of this distribution. By using this software in any fashion, you are agreeing to be bound by the terms of this license. You must not remove this notice, or any other, from this software.

The City of Chicago and its partners do not assume liability or support of Metalicious.

Acknowledgements

Metalicious was developed with support from the MacArthur Foundation and Chapin Hall at the University of Chicago.

metalicious's People

Contributors

kfogel avatar kwade27 avatar tomschenkjr avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

metalicious's Issues

Business function pages should list databases

As a curious citizen, I want to see a table of relevant databases on every business function page - 311, budget, housing, and so on - so that I can browse the city's database offerings.

screen shot 2013-10-21 at 3 42 30 pm

Right now, the only way to find things is search or by looking at the Recent Items on the home page.

Add list of all available data sources

Add some feature that allows for listing all data sources available without having to search or drill down by Business Function. I could imagine people getting frustrated because they were not sure what search term to use or what business function covered whatever they were trying to find. Also, there is a certain transparency value to a complete list, with which people can do as they like, even if it ends up being really long.

Add API functionality

Such a system will need to provide database information, schema, and associated descriptions. An API functionality will add machine-readable capabilities to the portal to enable third-party access and avoid the inevitable screen scraper (please put down the Python script).

Often, APIs could be used to create separate applications that provide different functionality, such as: an application that allows you to "upvote" fields to be the publish wishes to see on to the data portal, help integrate the data portal and data dictionary (see #19), integration with other tools (e.g., enterprise asset management), or simply redisplay the information.

Improve the search functionality to improve filtering

While the data dictionary was designed to be driven primarily through a search box, we need to improve advanced search filtering.

The most basic aspects of the functionality would be to add filtering for 'database', 'table', or 'field' results.

Allow attachments

Could be used to provide Entity-Relationship Diagrams and other useful documentation.

Tying metalicious databases to data portal datasets

The databases in the the dictionary are the sources for all/most of the datasets on a city's data portal.

To get complete transparency, and to allow cities to keep track of what ETL scripts they have floating around that get data from these systems and put it on their portal, it would be excellent to be able to associate databases in metalicious with datasets on a data portal.

You could keep this pretty lightweight - just associate a list of Socrata dataset id's with a given database in metalicious, and get fetch any additional data you might want about these datasets - their name, number of downloads, when they were last updated, whatever - from the Socrata Open Data API.

This would be a similar approach to how we built the project repository for the open gov hack night website - by only entering in Github repos and fetching all other data from the Github API. The code for that is here.

Importing databases using extracts from DB reverse-engineering programs

The process of uploading a new database into the platform is rather complicated, especially if one hopes to document a large universe of databases. While the platform does a good job of displaying information, burdensome import procedures Meanwhile, a number of popular software programs will reverse-engineer a database schemas and provides a good first-step.

It would be ideal if one could (1) reverse-engineer databases quickly using software; (2) export that reverse-engineered schema; (3) upload that extract to ; (4) have parse and extract the relevant information.

We will begin to incorporate compatibility with Oracle Data Modeler and bulk-importing data obtained from the Data Modeler's "Export to CSV" option. Oracle Data Modeler is free and compatible for Oracle, MySQL, and Access databases. For City of Chicago operations, this will cover a large portion of the city portfolio.

Other platforms are also good candidates for export-to-upload functionality, such as ERwin or SQuirreL.

Port to Ruby ?

I think there might be more developer support in Chicago if this were written in Ruby. What do other contributors (existing and potential) think? Anybody reading this who would like to contribute and would prefer to work in Ruby?

Metalicious_DB.sql: Procedure Definer

The procedures defined in the Metalicious_DB.sql file are expecting a user citydata@hyperion.chapinhall.org to be defined in the database. Would be nice to define this myself during installation in a different fashion than using find/replace on the file to change the user/host.

Unable to login

Unable to login.

Apache logs shows the following notice:

PHP Notice: Trying to get property of non-object in /var/www/metalicious/login.php on line 19...

Business function pages should list databases in sortable tables

The business function pages on metalicious do a great job listing the databases available within each corner of the city. But the tool overall is still oriented around search.

  • In order to increase the browse-ability of the site, it would be awesome to have those databases in table so you could sort them by views, number of tables, department, installation year, and so on.
  • The default sort could be set to views, so you get the most relevant stuff first.
  • To give people context about what these databases actually do, you could display the database description as a column in this table or a tooltip that would appear when you hover over the row.
  • Datatables is a great library that makes all of this easy to do, and you can skin it with Bootstrap styles.

Deploy import compatibility with outside database reverse-engineering platforms

Importing data into the data dictionary is not ideal, with some substantial manual work.

Future iterations should be compatible with programs that reverse-engineer database schemas and export that information. One example is Oracle Data Modeler, which will reverse engineer Oracle and MySQL databases then can export to CSV. These should be able to be bulk-imported into the platform.

No CLA

Forgot to upload the CLA to the CONTRIBUTING file. Will upload ASAP.

ETL scripts

ETL scripts are programs that take data from a database/spreadsheet/data source, maybe transform it a bit, and upload it to a data portal for public consumption.

The City of Chicago has hundreds of these running all the time. That's how the data portal stays up to data - for the most part, people aren't manually transferring data.

Since this is City code, shouldn't it be open source? (Possible security issues here, but just spitballing.) You could imagine a repo that would have all the ETL scripts, and a little JSON file tying each ETL script to it's data source on metalicious and its dataset on the portal.

This repo would help with ETL management. But there's more to it: if the ETL scripts were then linked to from metalicious, the data dictionary would provide complete transparency: here's what databases we have, here's where we make them public, and here's the code that does that. I imagine this would be most useful for other cities looking to start open data programs.

Login Issue

Website will not accept login attempts. Generates error:

The user specified as a definer ('citydata'@'hyperion.chapinhall.org') does not exist

Bootstrap icons not appearing

Bootstrap icons do not appear when Metalicious is deployed to a subdirectory. It does work when deployed to the root directory.

Document building and zoning data systems

It would be extremely useful to understand the internals of City systems that touch on buildings and zoning. Specifically, whatever systems are powering the following websites:

There are lots of fields exposed across these systems that are not available in any of the relevant open datasets - building footprints, permits, violations, and the like - and it would be great to understand the data / software universe they come from.

Create documentation in ReadTheDocs

The instructions in the README.md file are becoming rather long. There is a need to move to a more appropriate, structured instruction method. Preference is to use ReadTheDocs to document installation, configuration, and documentation instructions.

Complete contributing.md documentation

Some items still to be included:

  • Development process explanation
  • Description of the use of issue tracking, labels, and milestones for projects
  • Coding standards

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.