Code Monkey home page Code Monkey logo

turbot / steampipe-plugin-hackernews Goto Github PK

View Code? Open in Web Editor NEW
17.0 11.0 1.0 301 KB

Use SQL to instantly query stories, users and other items from Hacker News. Open source CLI. No DB required.

Home Page: https://hub.steampipe.io/plugins/turbot/hackernews

License: Apache License 2.0

Makefile 1.12% PLSQL 2.21% Go 96.67%
hackernews hackernews-api hackernews-api-wrapper sql postgresql steampipe steampipe-plugin postgresql-fdw hacktoberfest backup

steampipe-plugin-hackernews's Introduction

image

Hacker News Plugin for Steampipe

Use SQL to query stories, users and other items from Hacker News.

Quick start

Install the plugin with Steampipe:

steampipe plugin install hackernews

Run a query:

select * from hackernews_show_hn

Engines

This plugin is available for the following engines:

Engine Description
Steampipe The Steampipe CLI exposes APIs and services as a high-performance relational database, giving you the ability to write SQL-based queries to explore dynamic data. Mods extend Steampipe's capabilities with dashboards, reports, and controls built with simple HCL. The Steampipe CLI is a turnkey solution that includes its own Postgres database, plugin management, and mod support.
Postgres FDW Steampipe Postgres FDWs are native Postgres Foreign Data Wrappers that translate APIs to foreign tables. Unlike Steampipe CLI, which ships with its own Postgres server instance, the Steampipe Postgres FDWs can be installed in any supported Postgres database version.
SQLite Extension Steampipe SQLite Extensions provide SQLite virtual tables that translate your queries into API calls, transparently fetching information from your API or service as you request it.
Export Steampipe Plugin Exporters provide a flexible mechanism for exporting information from cloud services and APIs. Each exporter is a stand-alone binary that allows you to extract data using Steampipe plugins without a database.
Turbot Pipes Turbot Pipes is the only intelligence, automation & security platform built specifically for DevOps. Pipes provide hosted Steampipe database instances, shared dashboards, snapshots, and more.

Developing

Prerequisites:

Clone:

git clone https://github.com/turbot/steampipe-plugin-hackernews.git
cd steampipe-plugin-hackernews

Build, which automatically installs the new version to your ~/.steampipe/plugins directory:

make

Configure the plugin:

cp config/* ~/.steampipe/config
vi ~/.steampipe/config/hackernews.spc

Try it!

steampipe query
> .inspect hackernews

Further reading:

Open Source & Contributing

This repository is published under the Apache 2.0 (source code) and CC BY-NC-ND (docs) licenses. Please see our code of conduct. We look forward to collaborating with you!

Steampipe is a product produced from this open source software, exclusively by Turbot HQ, Inc. It is distributed under our commercial terms. Others are allowed to make their own distribution of the software, but cannot use any of the Turbot trademarks, cloud services, etc. You can learn more in our Open Source FAQ.

Get Involved

Join #steampipe on Slack →

Want to help but don't know where to start? Pick up one of the help wanted issues:

steampipe-plugin-hackernews's People

Contributors

bigdatasourav avatar cbruno10 avatar dboeke avatar dependabot[bot] avatar e-gineer avatar ers35 avatar lalitlab avatar misraved avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

gabssnake

steampipe-plugin-hackernews's Issues

make id required keycolumn in hackernews_item, fix join to hackernews_item, hackernews_user on id

Joins on id to hackernews_item, hackernews_user do not work due to turbot/steampipe-postgres-fdw#3

> select 
  i.id as article_id, 
  i.title as article_title, 
  u.id as user_id, 
  u.karma,  
  u.about 
from 
  hackernews_best as i, 
  hackernews_user as u
where 
  i.by = u.id
Error: cannot iterate: there was an error executing scanIterator: rpc error: code = Internal desc = 'List' call requires an '=' qual for column: id

"too many open files"

This seems to happen a lot on select * from hackernews_item. Reducing max_items from 5000 to 500 seems to resolve it, but what would prevent it from happening so you could retrieve 1000s of items?

2022-06-20 19:15:34.185 UTC [ERROR] steampipe-plugin-hackernews.plugin: [ERROR] 1655752531209: hackernews_top_story.getT
opStory: query_error="Error while fetching posts story : Get "https://hacker-news.firebaseio.com/v0/item/31809682.json?p
rint=pretty": dial tcp [2600:1901:0:94b6::]:443: socket: too many open files"

Stray characters are printed to the console when using with sqlite 

SQLite version 3.44.2 2023-11-24 11:41:44
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .load ./steampipe_sqlite_hackernews.so
sqlite> select id, title, time from hackernews_show_hn limit 3;
**38612680 cryingpotato**
38612680|Show HN: Advent of Distributed Systems|2023-12-12 14:38:13
**38632970 Rochus**
38632970|Show HN: A pure C89 implementation of Go channels, with blocking selects|2023-12-13 19:31:44
**38629539 ofek**
38629539|Show HN: PyApp – runtime installer for Python applications|2023-12-13 16:17:08
sqlite>

The lines in bold above are unexpected.

They seem to be caused by these unnecessary println statements

steampipe-plugin-hackernews/hackernews/item.go

Lines 67 to 72 in d2a427c

if len(posts) <= 0 {
fmt.Println("none:", id)
return hacknews.Post{Id: id}, nil
}
fmt.Println(posts[0].Id, posts[0].By)
return posts[0], nil

Those printlns should be removed

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.