Code Monkey home page Code Monkey logo

portal.js.bak's Introduction

🌀 Portal.JS
A gateway to your data

🌀 Portal is the data presentation framework. Portal can be used to showcase a single dataset or build a full-scale data catalog/portal. Portal is built in Javascript and React on top of the popular Next.js framework.

Status

Portal is currently focused on presenting a single (Frictionless) dataset for viewing and exploration.

Install

Git clone then:

yarn install

Usage

In this directory:

export PORTAL_DATASET_PATH=/path/to/my/dataset
yarn dev

And you will get a nice dataset page at http://localhost:3000

Design Notes

Portal.js is a React and NextJS based framework for building dataset/resources pages and catalogs. It consists of:

  • React components for data portal functionality e.g. data tables, graphs, dataset pages etc
  • Tooling to load data (based on Frictionless)
  • Template sites you can reuse using create-next-app
    • Single dataset micro-site
    • Github backed catalog
    • CKAN backed catalog
    • ...
  • Local development environment
  • Deployment integration with DataHub.io

In summary, technically PortalJS is: NextJS + data specific react components + data loading glue (mostly using frictionless-js).

portal.js.bak's People

Contributors

abhishekgahlot avatar anuveyatsu avatar cotts avatar dependabot[bot] avatar lauragift21 avatar risenw avatar rufuspollock avatar steveoni avatar tavareshansen avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

portal.js.bak's Issues

Add Frictionless views in dataset page

Views on the Frictionless object are rendered on the page ...

Acceptance

  • Views specified in a datapackage is correctly compiled and into a frictionless view spec Done See datopian/portal-experiment#5
  • Views can be displayed in the dataset page using the appropriate view engine

Screen Shot 2021-03-01 at 1 08 05 PM

Tasks

Analysis

How it works

Frictionless Data View (https://specs.frictionlessdata.io/views/) in a Data Package. E.g. this is a simple one:

{
  "id": "Graph",
  "type": "Graph",
  "state": {
    "graphType": "lines",
    "group": "Date",
    "series": [ "VIXClose" ]
  }
}

Convert the view to spec for the charting library.

  • First, prepare the view ...
    • Inline resources onto the view compile data inline onto resource in attribute _values
  • Then convert the view to relevant spec ... using https://github.com/frictionlessdata/datapackage-render-js
  • Display the view using the appropriate library based on spec. (Start with Plotly for now)
graph LR

view --> compiled[Compiled View]
compiled --datapackage-render-js--> plotly[Plotly Spec]
compiled --> vega[Vega Spec]

Algorithm for inlining the data ... - see https://github.com/datopian/frontend-showcase-js/blob/d8918ef6172d394e4bacff4fb8acbd4102fa0ebf/src/index.jsx#L67-L85

  • for each resource make sure we inlined all the data for that resource onto a _values attribute on the resource
  • if resources is specified on view use that other otherwise use the first resource in the resource array
  • replace the item in the resources array on the view with the resource object in the resources array

How it starts

{
  "id": "Graph",
  "type": "Graph",
  "state": {
    "graphType": "lines",
    "group": "Date",
    "series": [ "VIXClose" ]
  }
}

Then becomes

{
  // where the data comes from ...
  "resources": [0],
  "id": "Graph",
  "type": "Graph",
  "state": {
    "graphType": "lines",
    "group": "Date",
    "series": [ "VIXClose" ]
  }
}

Then becomes ...

{
  // where the data comes from ...
  "resources": [
    {
      name: ...
      _values: ...
    }
  ],
  "id": "Graph",
  "type": "Graph",
  "state": {
    "graphType": "lines",
    "group": "Date",
    "series": [ "VIXClose" ]
  }
}

https://github.com/datopian/datapackage-views-js/blob/bb08d2032e/__fixtures__/chart.fixtures.js

What do i want in a library ...

Given a Frictionless Dataset with a Frictionless View like ...

{
  "id": "Graph",
  "type": "Graph",
  "state": {
    "graphType": "lines",
    "group": "Date",
    "series": [ "VIXClose" ]
  }
}

Use the library like this

dpview = ...

out = dpview(...)

PlotlyView

Minimal Home page

Tasks

  • Simple mockup of how would it look like with following components:
    • Navigation bar (already implemented)
    • Hero component:
      • Search box
      • Intro text
      • Main image
    • Recent datasets
  • Implement

npm ERR! code E404

Hello Portal.js Developers!

I am interested in making a beautiful front-end for my CKAN-powered data catalogue!

When I try npm init portal-app my-data-portal I get a 404 error (see below).

(base) bryan@my-computer:~/Projects/ckan$ npm init portal-app my-data-portal

npm ERR! code E404
npm ERR! 404 Not Found - GET https://registry.npmjs.org/create-portal-app - Not found
npm ERR! 404 
npm ERR! 404  'create-portal-app@latest' is not in the npm registry.
npm ERR! 404 You should bug the author to publish it (or use the name yourself!)
npm ERR! 404 
npm ERR! 404 Note that you can also install from a
npm ERR! 404 tarball, folder, http url, or git url.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/bryan/.npm/_logs/2021-01-22T14_24_04_727Z-debug.log
Install for create-portal-app@latest failed with code 1

I am very new to NPM and Javascript, so perhaps I am just doing something wrong. Any assistance would be appreciated!

Thank you!

Bryan P

Add portal show CLI feature

The portal cli can automatically display dataset page with information when we run the command:

# PATH defaultso to current directory
portal [show [PATH]]

Acceptance

Screen Shot 2021-03-03 at 4 34 26 PM

Screen Shot 2021-03-03 at 4 36 21 PM

screen-recording-2021-03-03-at-42229-pm_SSWxjhCm.mov

Tasks

  • Have the portal nextjs app use an ENV variable to locate the file
  • Add CLI app folder
  • Create CLI file
    • that clones the portal repo
    • installs portal
    • exports the dataset folder
    • starts the portal app

Analysis

Starter, hacky approach

A starter, a shell script like ... this

cd /Users/rgrp/src/datopian/sandbox/portal
export DATASET=...
npm run dev
portal PATH

How it works:

cd ~/.portal/
git clone https://github.com/datopian/portal.js/
cd portal.js/packages/portal-app
# or yarn install ...
npm install
export DATASET=PATH
next dev

# watch and reload next ... (perhaps a hack by writing something into next folder to trigger reload)
# or use https://github.com/hashicorp/next-remote-watch

GraphQL Experiment - Apollo + Next.js

Research how to use apollo as the primary data source for Next.js apps.

Acceptance Criteria

  • We have a demo app showing how to work with Apollo and Next.js with local data and external data sources.

Task

  • Set up a demo project with Apollo and Next.js
  • Convert external rest api into graphql using apollo data source

Setup integration tests using cypress

Acceptance criteria

  • I can run yarn e2e and it'd run integration tests
  • We have tests for all existing pages
  • ? Run via Travis or similar CI

Tasks

  • Install and setup Cypress
  • Write tests
  • Setup CI

Refactoring Code as of July 2020

There are a few things we could do to refactor the codebase and set up a proper style guide.

Estimation

4d

Acceptance

  • We have code linting and prettifying which is checked automatically as part of CI
  • We have shared components that are reused
  • Typescript set up with proper types
  • No map using index as key
  • Create custom hooks/Context API, and replace initializeApollo with it

Tasks

  • atm no shared components. We can define shared components like input, forms, buttons, etc.
  • There's no gql schema management system. Right now, it has a gql query defined in each component file, which should be managed in one place.
  • Why don't we use lazy hooks method like useLazyQuery as well as useQuery?
  • It seems like they're trying to use typescript, but no types defined. Actually, it's not typescript.
  • I can see a bunch of duplicated code, which we can refactor by using well-structured data.
    e.g. /components/dataset/About.tsx => We can define table columns/rows.
  • Implement code linting e.g. use eslint?
  • Do not use array index as key when rendering list
  • Create and use custom hooks/Context API for managing 3rd party libraries. e.g. apollo-client initializer defined in lib

Analysis

Update display to imitate datahub.io

We want a data page like the ones on datahub.io (See sample below), including the design.

Acceptance

  • Show key metadata
  • Display sample data / data itself using react table
  • Show a sample of the README

Screen Shot 2021-02-25 at 10 11 40 AM

Screen Shot 2021-02-25 at 10 11 53 AM

Screen Shot 2021-02-25 at 10 12 10 AM

Convert to monorepo

With addition of create-portal-app it makes sense to move to a monorepo setup.

We will use yarn workspaces and, possibly in future, Lerna.

Acceptance

  • What is currently in root is in packages/portal
  • Have root readme (symlinked from packages/portal)
  • Have root package.json setup
  • Have lerna setup

Tasks

  • Create packages
  • Move current code into packages/portal
    • Keep tests in one central location at root (a la next.js)?
  • Root setup
    • Create a root package.json in wordspace mode
    • Symlink the README frompackages/portal into the root
  • Check everything passes
    • Fix test links

FUTURE

  • ? move components out of portal into portal-components package (or even split them up more)

Analysis

Try out apollo graphql server for "Local DB" (local state and remote data cache)

Follow up to research data layer #18

Acceptance

  • We have app set up with local DB with graphql API (e.g. apollo)
  • renderers (react components) access this layer (with some fixture data in it)

Tasks

Extra:

  • Refactor Dataset page to use apollo
  • Refactor Resource page to use apollo

Analysis

Mocking apollo server according to https://www.robinwieruch.de/graphql-server-mock-apollo-client:

  1. /pages/_app.tsx:
import { ApolloProvider } from '@apollo/react-hooks'
import { useApollo } from '../lib/apolloClient'

export default function App({ Component, pageProps }) {
  const apolloClient = useApollo(pageProps.initialApolloState)

  return (
    <ApolloProvider client={apolloClient}>
      <Component {...pageProps} />
    </ApolloProvider>
  )
}
  1. /lib/apolloClient:
import { useMemo } from 'react';
import { ApolloClient } from 'apollo-client';
import { InMemoryCache } from 'apollo-cache-inmemory';
import { SchemaLink } from 'apollo-link-schema';
import { makeExecutableSchema } from 'graphql-tools';

let apolloClient;

const typeDefs = `
  type Query {
    search(query: SearchQuery!): Response!
  }

  type SearchQuery {
    q: String!
    fq: String!
    sort: String!
    rows: Int!
    start: Int!
    facet: String!
    include_drafts: Boolean!
    include_private: Boolean!
    use_default_schema: Boolean!
  }

  type Response {
    success: Boolean!
    result: Result!
  }

  type Result {
    count: Int!
    sort: String!
    facets: String!
    search_facets: String!
    results: [Package!]!
  }

  type Package {
    name: String!
    title: String!
    notes: String!
    resources: [Resource!]!
    organization: Organization!
    metadata_created: String!
    metadata_modified: String!
  }

  type Resource {
    name: String!
    id: String!
    title: String!
    format: String!
    created: String!
    last_modified: String!
    datastore_active: Boolean!
    url: String!
  }

  type Organization {
    name: String!
    title: String!
    description: String!
    created: String!
    image_url: String!
  }
`;

const resolvers = {
  Query: {
    search: (parent, { query }) => ({
      success: true,
      result: {
        count: 2,
        sort: 'score desc, metadata_modified desc',
        facets: {},
        results: [
          {
            name: 'gdp',
            title: 'Country, Regional and World GDP (Gross Domestic Product)',
            notes:
              'Country, regional and world GDP in current US Dollars ($). Regional means collections of countries e.g. Europe & Central Asia. Data is sourced from the World Bank and turned into a standard normalized CSV.',
            resources: [
              {
                name: 'gdp',
                id: 'gdp',
                title: 'GDP data',
                format: 'csv',
                created: '2019-03-07T12:00:36.273495',
                last_modified: '2020-05-07T12:00:36.273495',
                datastore_active: false,
                url: 'http://mock.filestore/gdp.csv',
              },
            ],
            organization: {
              title: 'World Bank',
              name: 'world-bank',
              description:
                'The World Bank is an international financial institution that provides loans and grants to the governments of poorer countries for the purpose of pursuing capital projects.',
              created: '2019-03-07T11:51:13.758844',
              image_url:
                'https://github.com/datahq/frontend/raw/master/public/img/avatars/world-bank.jpg',
            },
            metadata_created: '2019-03-07T11:56:19.696257',
            metadata_modified: '2019-03-07T12:03:58.817280',
          },
          {
            name: 'population',
            title: 'World population data',
            notes:
              'Population figures for countries, regions (e.g. Asia) and the world. Data comes originally from World Bank and has been converted into standard CSV.',
            resources: [
              {
                name: 'population',
                id: 'population',
                title: 'Population data',
                format: 'csv',
                created: '2019-03-07T12:00:36.273495',
                last_modified: '2020-05-07T12:00:36.273495',
                datastore_active: true,
                url: 'http://mock.filestore/population.csv',
              },
            ],
            organization: {
              title: 'World Bank',
              name: 'world-bank',
              description:
                'The World Bank is an international financial institution that provides loans and grants to the governments of poorer countries for the purpose of pursuing capital projects.',
              created: '2019-03-07T11:51:13.758844',
              image_url:
                'https://github.com/datahq/frontend/raw/master/public/img/avatars/world-bank.jpg',
            },
          },
        ],
        search_facets: {},
      },
    }),
  },
};

const executableSchema = makeExecutableSchema({
  typeDefs,
  resolvers,
});

const link = new SchemaLink({ schema: executableSchema });

function createApolloClient() {
  return new ApolloClient({
    ssrMode: typeof window === 'undefined',
    link,
    cache: new InMemoryCache(),
  });
}

export function initializeApollo(initialState = null) {
  const _apolloClient = apolloClient ?? createApolloClient();

  // If your page has Next.js data fetching methods that use Apollo Client, the initial state
  // gets hydrated here
  if (initialState) {
    _apolloClient.cache.restore(initialState);
  }
  // For SSG and SSR always create a new Apollo Client
  if (typeof window === 'undefined') return _apolloClient;
  // Create the Apollo Client once in the client
  if (!apolloClient) apolloClient = _apolloClient;

  return _apolloClient;
}

export function useApollo(initialState) {
  const store = useMemo(() => initializeApollo(initialState), [initialState]);
  return store;
}
  1. /pages/search.tsx:
import { GetServerSideProps } from 'next';
import querystring from 'querystring';
import config from '../config';
import utils from '../utils';
import Head from 'next/head';
import Nav from '../components/home/Nav';
import Input from '../components/search/Input';
import Total from '../components/search/Total';
import Sort from '../components/search/Sort';
import List, { DEFAULT_SEARCH_QUERY } from '../components/search/List';
import { initializeApollo } from '../lib/apolloClient';

function Search() {
  return (
    <>
      <Head>
        <title>Portal | Search</title>
        <link rel="icon" href="/favicon.ico" />
      </Head>
      <Nav />
      <main className="p-6">
        <Input />
        <Total />
        <Sort />
        <List />
      </main>
    </>
  );
}

export const getServerSideProps: GetServerSideProps = async (context) => {
  const apolloClient = initializeApollo();

  await apolloClient.query({
    query: DEFAULT_SEARCH_QUERY,
  });

  return {
    props: {
      initialApolloState: apolloClient.cache.extract(),
    },
    unstable_revalidate: 1,
  };
};

export default Search;
  1. /components/search/List.tsx:
import Item from './Item';
import { useQuery } from '@apollo/react-hooks';
import gql from 'graphql-tag';

export const DEFAULT_SEARCH_QUERY = gql`
  search($query: SearchQuery!) {
    result {
      results
    }
  }
`;

export default function List() {
  const { loading, error, data, fetchMore, networkStatus } = useQuery(
    DEFAULT_SEARCH_QUERY,
    {
      // Setting this value to true will make the component rerender when
      // the "networkStatus" changes, so we are able to know if it is fetching
      // more data
      notifyOnNetworkStatusChange: true,
    }
  );

  if (error) return <ErrorMessage message="Error loading posts." />;
  if (loading) return <div>Loading</div>;

  const { result } = data;

  return (
    <ul>
      {result.map((datapackage, index) => (
        <Item datapackage={datapackage} key={index} />
      ))}
    </ul>
  );
}

Implementing this is returning the following error:

🌵 portal yarn dev
yarn run v1.22.0
$ next dev
ready - started server on http://localhost:3000
> Using external babel configuration
> Location: "/Volumes/128gb-ssd/projects/portal/.babelrc"
error - ./node_modules/simple-git/src/lib/git-executor.js
Module not found: Can't resolve 'child_process' in '/Volumes/128gb-ssd/projects/portal/node_modules/simple-git/src/lib'
wait  - compiling...
error - ./node_modules/simple-git/src/lib/git-executor.js
Module not found: Can't resolve 'child_process' in '/Volumes/128gb-ssd/projects/portal/node_modules/simple-git/src/lib'
^C

[epic] MVP of Full Portal/Catalog app: site components, bootstrap app etc (June 2020)

A decoupled data portal frontend built in javascript. See RFC for background: https://github.com/ckan/ideas/blob/master/rfcs/0005-decoupled-frontend.md

Acceptance criteria

  • Analysis written about frontend See RFC
  • MVP of app using content from both DMS (CKAN e.g. demo.ckan.org) and CMS (e.g. wordpress)
  • create-portal-app command and steps to boot a new portal #33
  • More features
    • Google Analytics
  • Reasonable docs(could reuse http://tech.datopian.com/frontend/ (if any needed))
    • Features
    • Getting started (5-10m tutorial)
    • Booting up
    • Basic config including add DMS (CKAN) and CMS (wordpress)
    • How to customize look and feel
    • How to add a new route
    • How to deploy
    • Feature matchup against CKAN current

Tasks

  • Analysis DONE
  • Stubbing the project
  • Preliminaries: Comparison of next.js vs nuxt.js, research typescript, tests, CI
  • MVP of a portal UI i.e. the key components: home, showcase, org pages #9
  • Data layer See below
  • Integrate CMS (e.g. blog) #32
  • create-portal-app script
  • Refactoring of the MVP to move out components and library code
    • Components
    • Data Access library
  • More features
    • Google analytics
    • i18n
    • Authentication - #34
    • Permissions - #34
  • Documentation
  • Write up and announce

Overview

graph TD

bootscript[Boot script]
docs[Documentation]

boot[Boot up] --> mvpsite[MVP of site]
mvpsite --> bootscript
mvpsite --> docs
mvpsite --> morecomp[More Components]
mvpsite --> auth[Authentication]
bootscript --> announce
docs --> announce[Announce]

classDef done fill:#21bf73,stroke:#333,stroke-width:1px;
classDef nearlydone fill:lightgreen,stroke:#333,stroke-width:1px;
classDef inprogress fill:orange,stroke:#333,stroke-width:1px;
classDef next fill:lightblue,stroke:#333,stroke-width:1px;

class boot done;
class mvpsite nearlydone;
class inprogress inprogress;
class bootscript next;

Detailed

graph TD

analysis[Analysis]
stub[Stub]
prelim[Preliminaries]

subgraph MVP of overall app
  analysis --> stub
  stub --> prelim
  prelim --> frontpage[Front page]

subgraph v0.1 MVP of a portal UI #9
  frontpage --> search
  frontpage --> search[Search page]
  search --> showcase[Dataset showcase]
  showcase --> showcase2[Resource showcase]
  search --> cms["CMS integration (Wordpress)"]
  showcase2 --> org[Org page]
end

datalayer[Data layer => GraphQL]

prelim --> datalayer
datalayer --> livesite{Live Site}
cms --> livesite
org --> livesite

end



subgraph Docs
  prelim --> readme[Stub README with install etc]
end

theme

subgraph v1
end

subgraph Docs
end


classDef done fill:#21bf73,stroke:#333,stroke-width:1px;
classDef nearlydone fill:lightgreen,stroke:#333,stroke-width:1px;
classDef inprogress fill:orange,stroke:#333,stroke-width:1px;
classDef next fill:lightblue,stroke:#333,stroke-width:1px;


class stub done;
class prelim done;
class frontpage done;
class datalayer done;
class search done;
class cms done;
class showcase done;
class showcase2 inprogress;
class org next;
graph TD

subgraph Key
  done[Done]
  nearlydone[Nearly Done]
  inprogress[In Progress]
  next[Next Up]
end

classDef done fill:#21bf73,stroke:#333,stroke-width:1px;
classDef nearlydone fill:lightgreen,stroke:#333,stroke-width:1px;
classDef inprogress fill:orange,stroke:#333,stroke-width:1px;
classDef next fill:lightblue,stroke:#333,stroke-width:1px;

class done done;
class nearlydone nearlydone;
class inprogress inprogress;
class next next;

Preliminaries

  • Choose between javascript and typescript (prob go with typescript) typescript
  • Choose between next and nuxt Chose next.js
  • Configure test suite. Recommended to take a look in the libraries libs: Jest, Mocha, Chai.js, and Sinon.js. Went with Jest
    • We are adding Cypress for functional tests - see PR here - #26
  • Configuration file for Continuous Integration/Continuous Delivery. - #19

Data Layer

  • Research options and choose approach #24 Using Apollo Graphql as internal data layer
  • NextJS working against demo ckan
  • Setup Apollo client with CKAN API and manage state from its cache - #29
  • Connect WordPress API to Apollo client - #32

Documentation

  • Write a README for a new person starting your project
    • Include aspirational getting start (readme drive development) E.g. create-portal-app
  • How to customize the look and feel
  • More complex customization
  • Matchup up against current CKAN functionality
    • Front page
    • Showcase
      • Dataset
      • Resource
    • Catalog Search
    • Showcase page
    • Resource views
    • Collections (tags)
    • Organizations
    • CMS
      • Blog listing
      • Blog post
      • Page

Move portal-experiment into portal and refactor

Move and refactor portal-experiment into portal.js

We already have older packages/portal which is currently more of an example of a full data catalog site running on nextjs. We want to keep this and move it to examples/catalog

Our new portal experiment should then become the main portal package in packages/portal so we can focus on developing there.

Acceptance

  • packages/portal moved to examples/catalog DONE
  • I have portal-experiment integrated into portal.js repo DONE. located at packages/portal. Have also integrated the CLI into that package.
  • updated main README to reflect this DONE. Main README is stubbed atm

Task

  • Agree on repo we are using DONE. We agreed to move portal-experiment into portal.js
  • Clean up the portal repo
    • Move packages/portal to examples/catalog
    • Refactor README to explain this is an example
    • Reboot portal sub package with README

Fix Jest Tests

All the jest tests are broken. We need to fix it.

Current Tests:

  • tests/pages/index.test.tsx
  • tests/components/search/Total.test.tsx
  • tests/components/search/List.test.tsx
  • tests/components/search/Item.test.tsx
  • tests/components/search/Form.test.tsx

Create Tests for Existing Pages

We want to test out the main pages of the app (home, showcase) using the TDD approach.

Acceptance

  • Test exists for the main pages of the app.

Tasks

  • Research how testing works in Next.js
  • Write tests for existing features of the app

Refactor Mocked data to follow the Frictionless specs

Mocked data should follow the Frictionless specs e.g. for dataset page Data Package, for resource use Data Resource spec

Acceptance

  • Mocked data follows the Frictionless Data specs approach

Tasks

  • Refactor current mocked resource page table to follow the data resource spec

CI / CD setup

Acceptance criteria

  • When a new commit is pushed to master branch, it is automatically deployed somewhere so we can use it as a live demo internally and for testing
  • When I review a pull request, I can see live demo of the changes

Tasks

  • Setup Vercel with this repo
  • Add domain address in the README

[research] Data Layer: how next.js connects to backends

How should we get (meta)data / content? What is the "data layer" for this system?

The content fetching divides into two parts:

graph LR

remote[Remote e.g. CKAN API] --fetchers--> localdb[Local DB Cache]
localdb --common api--> render[Renderer]
  • Want a common API / structure for local DB so that renderers have a simple interface and the complexity of fetchers is hidden from them. Options:
    • redux
    • graphql based
  • SUMMARY: atm we are strongly inclining to graphql based approach. (see below for reasoning)
  • fetchers will always be somewhat bespoke

Tasks

  • Research options and summarize approach DONE - see above for summary
  • Try out apollo graphql for local db (put in some fixtures) #24
  • Try out connecting up external API e.g. CKAN to local DB #13

Analysis (in progress)

Next.js data fetching can be done in two ways:

  • getStaticProps - fetches data at build time
  • getServersideProps - fetches data at runtime

We want to use the getServersideProps method for our use case. There's a repo with helpful examples to see how data fetching works in Next. https://github.com/vercel/next.js/blob/canary/examples

What we want ...

fetch could use graphql, raw queries it doesn't matter ...

graph TD

external[External Data Source e.g. CKAN or Wordpress]
react[React Component]
page[Rendered Page]

external --fetch--> react
external --fetch --> redux
external --fetch --> store[Store - non redux]
redux--> react
store --graphql--> react

react --render--> page

gatsby ...

graph TD

external[External] --> source[Source plugin]
source --> store[Internal Store - nodes]
store --graphql--> component[Component]
component --render--> page

Inbox (misc)

Inbox on Graphql approach

Summary

Excellent critique of next.js SSR approach: https://en.paqmind.com/blog/ssr-is-not-the-future and recommendations on how to approach (jan 20 2020)

This section of gatsby doc is useful https://www.gatsbyjs.org/docs/graphql-concepts/#why-is-graphql-so-cool

For a more in-depth look, read why Gatsby uses GraphQL.

  • Eliminate frontend data boilerplate — no need to worry about requesting & waiting for data. Just ask for the data you need with a GraphQL query and it’ll show up when you need it
  • Push frontend complexity into queries — many data transformations can be done at build-time within your GraphQL queries
  • It’s the perfect data querying language for the often complex/nested data dependencies of modern applications
  • Improve performance by removing data bloat — GraphQL enables you to select only the data you need, not whatever an API returns

Connect CMS (WordPress) API to Apollo client

As next step after #29 we want to connect second data source which is WordPress REST API to our Apollo GraphQL client.

Acceptance criteria

  • I can connect a WordPress.com instance by defining CMS env var.
  • All my posts appear at /blog/
  • All my pages appear at /:slug
    • appars at /p/:slug

Tasks

  • Create components for rendering static content: /components/static/Page.tsx and /components/static/Post.tsx
  • Create NextJS page for both page and post types: /pages/[page]/index.tsx and /pages/blog/[post]/index.tsx
  • Setup Apollo client to connect multiple REST APIs: https://www.apollographql.com/docs/link/links/rest/#multiple-endpoints
  • Setup NextJS pages to fetch WordPress API
  • Setup components to use the content from Apollo cache

Update theme of default app

Apply theme from home page mockup in #30 to default app (currently in root folder and will move to packages/portal/ with monorepo #38

Estimation

2d

Acceptance

  • When I create a portal with the default theme it looks like the mockup in #30 (on front page) and uses those fonts etc elsewhere
  • No hard-coded color values (we are using tailwind ...)

Tasks

  • Install current setup
  • Get access to XD
  • Define colors and font size in tailwind config / in css
  • Setup theme and stylesheets based on theme

[epic] v0.2: create-portal-app command and steps

create-portal-app command and steps to boot a new portal.

Job Story

When I'm a portal developer wanting to create a new portal I want a very simple experience (e.g. one command) to boot a new portal that i can then customize.

Approach

We are taking the modern create-react-app / create-next-app approach:

npx create-next-app
# or in yarn
yarn create next-app

BONUS 🍬 : like create-next-app it would be nice to be able to run off a template e.g.

# e.g. next.js
npx create-next-app --example with-styled-components with-styled-components-app
# We would have e.g.
npx create-portal-app --example with-drupal portal-with-drupal-as-cms

Acceptance

  • I can bootstrap a frontend app using a single command
    • create-portal-app package on npm
  • My boot screen looks similar to design in #35
  • (BONUS) can create from examples / templates

Tasks

  • Analysis and Research
    • Look at create-react-app and create-nextjs-app and evaluate which we use
    • Have an outline of how we implement based on this
  • Stub package
    • Create create-portal-app MVP
    • Test Stubbed package (run locally and check if the behavior fits with requirements)
  • Publish create-portal-app and make datopian the owner on npm (reserves it for later) (don't put under @datopin as that would break npx create-portal-app)
    • A couple of projects are already using Portal.js, Portaljs, Portal-js. Need to find a unique name to publish on NPM. We only need create-portal-app so not a big deal I think
  • Basic working script that boots something
  • Show the welcome screen
  • Publish to npm and have it owned by Datopian
    • ❓ script for auto doing this

Analysis

Our approach is different frontend v2 (compositional vs inheritance)

Instead of developing themes, we want to be able to init the frontend app and customize as per our needs. For example, have a look how the following frameworks work.

Inspiration

  • Create React App: https://github.com/facebook/create-react-app
    • init command: npx create-react-app my-app-name
      • you can also npm init react-app my-app-name or yarn create react-app my-app-name
    • run server: npm start
  • GatsbyJS: https://github.com/gatsbyjs/gatsby
    • init command: gatsby new my-blazing-fast-site
      • requires pre-installing cli tool: npm install -g gatsby-cli
    • run server: gatsby develop

In GatsbyJS, you first need to install CLI tool that helps you to init an app, start server etc. We want to simplify the process and use Create React App style => use npx for initiating the app and then we can use npm run ... commands for running server etc.

npx is a very powerful command that's been available in npm starting version 5.2

How does npx works?

npx create-react-app my-cool-new-app installs a temporary create-react-app and calls it, without polluting global installs or requiring more than one step!

Research on create-xxx-app in detail

  • FB create-react-app
    • uses js
    • older
    • LoC:
  • create-nextjs-app
    • uses typescript ( Mostly JS not much Typescript )
    • newer
    • LoC:

FB create-react-app

https://github.com/facebook/create-react-app

https://github.com/facebook/create-react-app/blob/master/tasks/cra.js

Main file for managing all the params for creating react app. tasks/cra.js -> from package.json

  • Specific code

package.json - https://github.com/facebook/create-react-app/blob/master/package.json#L7

"scripts": {
    "build": "cd packages/react-scripts && node bin/react-scripts.js build",
    "create-react-app": "node tasks/cra.js",

cra.js - https://github.com/facebook/create-react-app/blob/master/tasks/cra.js

Child processes here that are used to create project dir.

create-nextjs-app https://github.com/vercel/next.js

Codebase here for next: https://github.com/vercel/next.js/tree/canary/packages/create-next-app

Also let's clean the codebase to use either javascript or typescript. Right now its mixed for open source repo.

package.json

  • builds index.ts to dist and then that is what is packaged as the script ...
  "bin": {
    "create-next-app": "./dist/index.js"
  },
  
  ...
  
  "files": [
    "dist"
  ],
  "scripts": {
    "build": "ncc build ./index.ts -w -o dist/",
  },

index.ts - https://github.com/vercel/next.js/blob/canary/packages/create-next-app/index.ts

Calls this sub lib

  • Use commander to parse arguments and provide context
  • Chalk for terminal string styling
  • Create an example directory (if doesn't exist)
  • Initialize a git directory for generated app
  • Download and unpack the app from github
  • Create app requires following params
    • appPath
    • useNpm
    • useYarn
    • example
    • examplePath
  • Helpers like
    • downloadRepo
    • isRepo
    • URL gives 200 status code
    • extract the tar file
    • downloadExampleRepo

Resource page with Data Explorer or preview

A page for viewing an individual resource exists and it previews the data (if a preview for that data type exists).

In particular, for CSV etc and data in CKAN DataStore it displays the Data Explorer.

NOTE: ⚠️ there is quite a bit of thinking here about how we refactor the data previewing and data explorer. Read the notes in analysis section below. Note, in particular, that the acceptance and tasks are not yet updated to reflect this (we will probably need to refactor this issue into multiple issues).

Acceptance

Tasks

  • Implement /[org]/[dataset]/r/[resource] page that renders necessary components for the resource page:
    • Navbar
    • About resource component, e.g., metadata table with info such as resource name, title, description, format, created/modified date and link to downloading
  • Data Explorer or Data Preview component based on given resource type:
    • If in datastore, we can show the data explorer that has preview + query builder + map/chart builder + pagination
    • If not in datastore, we only render a preview based on format:
      • tabular data => table
      • image/website/pdf => web view
      • geo data => map
      • anything else => nothing

Analysis

Re-using existing Data Explorer library might not be the best way to go here because:

  • Data Explorer we've built using React + Redux wasn't the easiest library to maintain, for example, if you need to fix something in one of the components (eg, Map Builder), you have to go there and do:
    • fix it
    • if unit tests there, run them
    • test with fixtures (cosmos)
    • test with yarn start to load complete app
    • build the dist version by yarn build:package
    • push to github
    • publish to NPM
    • install it in Data Explorer and build the bundle
    • import the bundles (normally huge ~>2-3MB due to dependencies such as Plotly) into the HTML page

As we can see, it is quite a long process and over time we've seen little feature development but only bug fixes. I believe the process was discouraging for developers in general.

Other points:

  • Currently, using Data Explorer as a React component is not clear as it was always been used as bundles that doesn't get called explicitly with arguments, but inits on load and traverses the DOM to find the HTML tag with data-pacakge attribute. It parses it and uses it as props for child components.
  • If we want to use Data Explorer and its components in NextJS, we have to refactor it as it isn't compatible right away, for example, importing CSS, see https://github.com/vercel/next.js/blob/master/errors/css-npm.md

How would I build Data Explorer today with NextJS?

  • Make the Data Explorer a component rather than a library
  • Control components such as Query/Map/Chart builder from NextJS so basically our frontend app becomes the Data Explorer
  • Use something like SWR for client-side data fetching - https://swr.now.sh/

http://tech.datopian.com/data-explorer

graph TD

DataExplorer
DataPackageViews
DataStoreQueryBuilder
ChartBuilder
MapBuilder
Redux

subgraph old
  Redux -.- DataExplorer
  DataExplorer --> DataPackageViews
  DataExplorer --> DataStoreQueryBuilder
  DataExplorer --> ChartBuilder
  DataExplorer --> MapBuilder
end
graph TD

NextJSPage
DataPackageViews
DataStoreQueryBuilder
ChartBuilder
MapBuilder
GraphQL

subgraph new
  NextJSPage --init--> DataPackageViews
  NextJSPage --init--> DataStoreQueryBuilder
  NextJSPage --init--> ChartBuilder
  NextJSPage --init--> MapBuilder
  GraphQL -.- NextJSPage
  GraphQL -.- DataPackageViews
  GraphQL -.- DataStoreQueryBuilder
  GraphQL -.- ChartBuilder
  GraphQL -.- MapBuilder
end
  • User: GET a resource page - /@org/{dataset-name}/r/{resource-name}
  • NextJS (server side) fetches DMS API to get resource metadata which is then cached by GraphQL
  • NextJS responses with HTML + JSON for hydrating the tempalte
  • Browser renders the page with components
  • DataPackageViews component fetches data from the source url (data can be in filestore [or datastore])
    • OR: could load the data server side and pass down ...
  • Renders the data in a table (for tabular data)
    • TODO: other renderings ...``

First steps:

Notes

  • Resource pages do simple (pre)view by default
  • Explorer page for rich exploration (requires)
# url structure
/dataset/myname/ => [implicitly] /dataset/myname/show/default/ # default-branch-latest-revision/
/dataset/myname/resource/xxx => [implicitly] renders  /dataset/myname/show/default/{resource-id}

# explicit revisions
/dataset/myname/show/{revref}/              # dataset showcase
/dataset/myname/show/{revref}/{resource-id} # resource showcase

[Epic] MVP of the Frontend v3 app: home, showcase, org and search pages

We want to build an MVP of a data portal frontend using Next.js. Data can be mocked or come from demo.ckan.org.

MVP URL: https://portal.datopian1.now.sh/

Acceptance

This would be the core of an MVP

  • showcase
    • dataset - can model layout on top part of datahub.io showcase pages
    • resource page - a dedicated resource page
  • org profile page with list of datasets
  • search page for searching datasets (this could be dynamic/pure client side JS or could be server side)
  • home page (semi-optional for now in fact)

Dataset and Resource page

  • A page with a list of datasets - with key information(title, description, etc)
  • All listed dataset leads to a new page with additional details on them.
  • Resource page: details on a resource #12
    • download
    • preview a dataset Issue with existing React app for data explorer. needs some work. See #12 for details
  • Search functionality - search for specific dataset using queries/keywords.

Tasks

Showcase page(s)

  • Stub a page /@myorg/mydataset that is empty
  • Copy over and convert nunjucks template from current frontend-v2 into jsx template (omitted all the preview stuff and resource display)
    • Copy the preview material into a /@myorg/mydataset/r/myresource template
  • Refactor the JSX into templates so we move away from "inheritance" towards "composition" in how we build pages
    • Simplest to start would be to move everything into a component "DatasetShowcaseCore"
  • Stub out the frontpage from frontend v2
  • Replace hard coded page data with hardcoded data in the "controller"
  • Replace hard coded data with real data from a live source ...

Organization page(s)

  • page with a list of all organizations
  • page with a list of related datasets under an org.

How do we have /@myorg/mydataset => lookup data via fetch API from the right place?

We want /@myorg to be dynamic. Creating a dynamic route might work and setting up fetch API to pull data from the dynamic source (more research to be done here)

Admin to setup for work on new Portal.JS (Feb 2021)

We need to setup and "clean the decks" in prep for the re-focused Portal.JS as of Feb 2021.

Acceptance

  • Clear on where we are working on things We are focused in this portal.js repo
  • Old issues tidied and cleared up DONE. Issue list is clean (empty)!

Tasks

Analysis

Existing outstanding issues

image

Connect CKAN API into Apollo GraphQL client

Job story:

When building frontend for my CKAN portal using this framework I want to configure my CKAN backend so that all the metadata shows up in the portal

Acceptance criteria

  • Demo portal site working with CKAN API e.g. demo.ckan.org - http://portal.datopian1.now.sh/
  • Docs in README for how to configure your CKAN backend - 90f3fc0
    • Notes on what is NOT supported atm b19a37e
  • (v brief) document your learnings about apollo and share in issue / with team

Tasks

  • Connect CKAN API into Apollo GraphQL client
    • Initial setup work
    • Search page refactoring
    • Dataset page refactoring
    • Resource page refactoring

Analysis

There are 2 ways of connecting CKAN API with Apollo:

  1. Build Apollo Server with data source as CKAN API.
  2. Use REST link to connect REST API to Apollo client so no server is needed: https://www.apollographql.com/docs/link/links/rest/

Apollo client in Next.js (assuming there is a graphql server):

// `/lib/apolloClient.js`
import { useMemo } from 'react';
import getConfig from 'next/config';
import { ApolloClient } from 'apollo-client';
import { InMemoryCache } from 'apollo-cache-inmemory';

let apolloClient;

function createApolloClient() {
  const { publicRuntimeConfig } = getConfig();
  return new ApolloClient({
    ssrMode: typeof window === 'undefined',
    link: x // This is where we connect the API...,
    cache: new InMemoryCache(),
  });
}

export function initializeApollo(initialState = null) {
  const _apolloClient = apolloClient ?? createApolloClient();

  // If your page has Next.js data fetching methods that use Apollo Client, the initial state
  // gets hydrated here
  if (initialState) {
    _apolloClient.cache.restore(initialState);
  }
  // For SSG and SSR always create a new Apollo Client
  if (typeof window === 'undefined') return _apolloClient;
  // Create the Apollo Client once in the client
  if (!apolloClient) apolloClient = _apolloClient;

  return _apolloClient;
}

export function useApollo(initialState) {
  const store = useMemo(() => initializeApollo(initialState), [initialState]);
  return store;
}

Apollo Server

server:

const { ApolloServer } = require('apollo-server');
const typeDefs = require('./schema');
const { createStore } = require('./utils');

const ckanAPI = require('...');

const store = createStore();

const server = new ApolloServer({
  typeDefs,
  dataSources: () => ({
    ckanAPI: new ckanAPI()
  })
});

server.listen().then(({ url }) => {
  console.log(`🚀 Server ready at ${url}`);
});

ckanAPI:

const { DataSource } = require('apollo-datasource');

class ckanAPI extends DataSource {
  constructor({ store }) {
    super();
    this.store = store;
  }

  /**
   * This is a function that gets called by ApolloServer when being setup.
   * This function gets called with the datasource config including things
   * like caches and context. We'll assign this.context to the request context
   * here, so we can know about the user making requests
   */
  initialize(config) {
    this.context = config.context;
  }

  
  // Search method that searches CKAN portal
  async search({ query }) {
    // ...
    return result;
  }

  async getDataPackage({ name }) {
    // ...
    return datapackage;
  }
}

module.exports = ckanAPI;

schema:

const { gql } = require('apollo-server');

const typeDefs = gql`
  type DataPackage {
    id: ID!
    name: String
    title: String
    resources: Resources
  }
`;

module.exports = typeDefs;

Without Apollo Server

References:

Theming portal howto

We will want instructions (and examples) of theming a portal

When creating a new portal I want to develop a custom theme so that it looks great and aligns with my brand

When deploying a portal I want to be able to install different themes (and switch between them) so that I can quickly try out and theme my site with an existing theme

Research - theming in next.js

Use styled components ...

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.