I mainly work in Go and Python, with a major focus on the fintech/trading niche.
I'm looking to work on some innovative ideas around the intersection of old and new tech as it applies to the end user.
Go API for Finviz
Home Page: https://github.com/d3an/finviz/wiki
License: MIT License
Other CLI libraries, specifically argparse
and click
for Python, support arguments that must be one of a few given options. Figure out if Cobra supports this, and if it does then implement it. If Cobra does not support it, submit a PR to Cobra to fix it.
Hey there, thanks for the great api.
I'd like to get e.g. a particular's ticker Current Ratio
. Based on the screener example on your home page, If I do e.g.
client := quote.New(nil)
stocks, err := client.GetQuotes([]string{"AAPL", "TSLA"})
all parameters/columns yield no information, please see the pic:
Am I doing sth wrong? Any suggestion appreciated 👍
Create a subcommand that accepts a FinViz url as input and returns a DataFrame (or exports to a file) the resulting scrape of that url. If the FinViz app detailed in the url is not supported, print an informative error message.
Examples:
$ finviz urlscraper https://finviz.com/quote.ashx?t=AAPL
OR
$ finviz usr https://finviz.com/screener.ashx?v=111&f=exch_nasd&ft=4
OR (unsupported app)
$ finviz usr https://finviz.com/groups.ashx?g=industry&sg=consumerdefensive&v=210&o=name
[ERROR] Groups app not supported.
Add functions and a CLI subcommand to support the Groups page on FinViz.
Add go-vcr
package. This package is likely based on the pyvcr
package for Python. Its incredibly useful for testing, permitting the avoidance of writing custom patch and mock objects.
Using gopy, figure out if it's possible to export this library as a Python package, especially as to whether or not PyPI will support it.
There are two primary FinViz packages for Python, finviz
and finvizfinance
. Both are decently maintained, but still have a lot of room for improvement. If this package can support all features included in the forementioned packages, improve on outstanding issues, and include even more features (AND documentation), Python developers may begin to use this package instead.
Add support for the charts view (210) along with the different types of graphs and tick frequencies.
The async approach to screener/quote downloads may result in a temporary/permanent service block. Switch to a strict synchronous approach to better avoid Cloudflare roadblocks.
Restructure the existing codebase to support more inherited methods/classes. This extends into supporting other FinViz apps such as News.
Create a partially abstract base class View
and interface ViewInterface
:
type View struct {}
type ViewInterface interface {
GenerateURL() string
Scrape(document *goquery.Document) ([][]string, error)
}
Each app should have a derived object that inherits the predefined interface methods, such as for News:
type NewsView struct {
finviz.View
}
Going deeper, an app should derive its actual views from the app view base class, ex:
type SourceNewsView struct {
NewsView
}
This restructuring should help greatly with view URL generation as derived objects can call parent methods, thus generating the URL in a step by step basis: ('https://finviz.com' OR 'https://elite.finviz.com') + '/news.ashx' + '?v=2'
.
Additionally, this restructuring will help clean up the file structure of the codebase and make it more presentable and less daunting on GitHub.
Every good package has unit tests. Write a few, get the coverage stamp on the README, and maybe find a couple bugs along the way. Having good test coverage is a great way to make sure new code won't break the system on a refactoring or new feature.
The testing
package is probably all that's needed, but it's probably a good idea to patch the requests and use mocks where necessary. Take a look at a few of the other go
testing packages and see if they're necessary.
filter_test.go
screener_test.go
views_test.go
EDIT: Added the coverage badge. Currently using the testing
, govcr
, and testify
packages to write unit tests.
The code for filters.go
is over 3000 lines. There's a lot of repetition, which can probably be refactored.
There has to be a way to refactor the implementation to be simpler. Although the lack of generics prevents a generalization of types, there has to be a better object-oriented (or composition) way to reduce the complexity of the codebase. Figure it out and clean up the code.
Earnings Calendar scrape is producing an empty DataFrame. Fix it.
Line 160 in c781b63
I haven't come across this, yet, so it caught my eye. After an initial search, I see that it's not recommended due to the memory size implications. Can you explain why a pointer to a channel is used? (Still learning Go btw)
Coveralls' badge should be good enough. Add it with goveralls.
"responsively" -> "responsibly"
Fix the module name to be github.com/d3an/finviz
Currently, the maximum number of securities returned in a screen is 20 (free version). Write a function that allows for multiple pages of screen results to be downloaded. Consider running a recursive operation on the existence of the next
button at the bottom of the page.
Allow the default 20 number of rows to be modified by users. Finviz Elite allows this number to be varied up to 100 rows. If it's too much of a hassle, don't bother.
Dataframes are currently returned in the format they were created in. That means that a column that should be in percent is actually represented in string format (i.e. "+35.79%" vs. 0.3579) Create cleaning functions that "type casts" data into the proper format for further analysis.
The FilterValueLookup
covers all 70 filter types and their corresponding values. The dictionary needs to be fixed up so that each value perfectly mirrors that provided on the site. Once this step is completed, shortcuts should be added as well. Consider the Market Cap filter. A possible shortcut for Mega ($200bln and more)
could just be mega
.
As a bonus task, consider adding support for approximate string matching. Take a look at the Levenshtein distance metric and see how others have implemented this technique in their projects.
Ex.
[
{
"A": {
"Avg Volume": "1.47M",
"Change": "0.00%",
"No.": 1,
"Perf Half": "28.54%",
"Perf Month": "6.04%",
"Perf Quart": "13.47%",
"Perf Week": "4.12%",
"Perf YTD": "21.23%",
"Perf Year": "45.95%",
"Price": 103.42,
"Recom": "2.40",
"Rel Volume": 0.97,
"Ticker": "A",
"Volatility M": "1.80%",
"Volatility W": "1.82%",
"Volume": 0
},
"AA": {
"Avg Volume": "7.33M",
"Change": "0.00%",
"No.": 2,
"Perf Half": "23.38%",
"Perf Month": "7.32%",
"Perf Quart": "26.03%",
"Perf Week": "-2.65%",
"Perf YTD": "-31.80%",
"Perf Year": "-19.75%",
"Price": 14.67,
"Recom": "2.70",
"Rel Volume": 0.77,
"Ticker": "AA",
"Volatility M": "4.72%",
"Volatility W": "4.12%",
"Volume": 0
}
}
]
instead of
[
{
"Avg Volume": "1.47M",
"Change": "0.00%",
"No.": 1,
"Perf Half": "28.54%",
"Perf Month": "6.04%",
"Perf Quart": "13.47%",
"Perf Week": "4.12%",
"Perf YTD": "21.23%",
"Perf Year": "45.95%",
"Price": 103.42,
"Recom": "2.40",
"Rel Volume": 0.97,
"Ticker": "A",
"Volatility M": "1.80%",
"Volatility W": "1.82%",
"Volume": 0
},
{
"Avg Volume": "7.33M",
"Change": "0.00%",
"No.": 2,
"Perf Half": "23.38%",
"Perf Month": "7.32%",
"Perf Quart": "26.03%",
"Perf Week": "-2.65%",
"Perf YTD": "-31.80%",
"Perf Year": "-19.75%",
"Price": 14.67,
"Recom": "2.70",
"Rel Volume": 0.77,
"Ticker": "AA",
"Volatility M": "4.72%",
"Volatility W": "4.12%",
"Volume": 0
}
]
FinViz has a News page. Add support for it. Create a new subcommand for the CLI to facilitate this as well.
There are only two views. So, two scraping functions are required (probably).
FinViz offers several views for viewing screen results. Each view is labelled with a unique ID.
The following views exist:
ScreenInputReader
function that would consume a JSON object (marshalled/unmarshalled?) or maybe a byte array and extract a ScreenInput object from the input.screener.go
into a better designed approach. Use interfaces where necessary.This is good functionality to add and may help with writing Python bindings later on.
Check out Bubble Tea framework. Might be nicer than Cobra for creating user-friendly CLIs.
Add functionality to support the output of screener results to a csv or json. Also support the creation of external file results for csv or json.
Refactor CleanScreenerDataFrame
to support both Screener df's and Quote df's.
Create a base command finviz
and add subcommands as functionality increases.
Create a screener
subcommand to run screens against Finviz.
Create a help
sub-subcommand for screener
to help with displaying valid options.
Examples:
$ finviz screener -s TopGainers -o -ChangeFromOpen -f Industry:gold,airlines Price:PriceOver1
# Same as running the RunScreen function and printing the dataframe to screen
$ finviz screener help
# Returns general help for all options (basically lists possible values and syntax)
$ finviz screener help signal filter:industry,sector
# Returns general help for signal and filter, along with specific help for industry, sector filters
Although this API still works, Mario's Python Finviz API had an issue where users received 403 error codes on requests. The issue was fixed by randomizing/setting the user agent to be a more common one.
Add support for user agent generation to avoid a possible block in the future.
Upgrade the project's Go version from 1.14
to 1.15
. Sort out any bugs that may arise.
GetQuoteData
into GetFullQuoteData
and GetBasicQuoteData
. The difference is that GetFullQuoteData
also returns the special sections (Analyst Recommendations, News, Insider Trading, Description, Chart) in its DataFrame.GetInsiderTrading
, GetNews
, GetAnalystRecommendations
, GetDescription
, GetChart
.I am requesting quotes client.GetQuotes([]symbols)
for a bunch of tickers and get panic
if a ticker is not available in FinViz, e.g. ticker SHAC
is not..
error received while scraping quote for 'SHAC': error getting url: 'https://finviz.com/quote.ashx?t=SHAC&ty=c&p=d&b=1', status code: '404'
I could catch the error
, remove the ticker and pull again, but that could be much overhead, specially from FinViz point of view. Could this be fixed internally so the program does not panic but throws warning instead that a ticker is not available?
Add properties to filters, i.e. whether or not they support multiple or custom values (custom values is looking ahead to elite support)
Add functions to support the scrape and export of FinViz's weekly economic calendar. Add a new subcommand to the CLI to support this function.
Fix the url generation bug in which two question marks appear in the final result instead of one. This doesn't affect functionality, but may stand out among FinViz's other requests.
FinViz also has quote pages for its tickers (quote.ashx
). Given a list of tickers, create a function that completes a download and scrape using goroutines for concurrency and efficiency. Add a CLI subcommand for this as well.
$ finviz quote aapl,goog,amzn,tsla
Finviz/Cloudflare may have blocked some of the user agents generated for requests. Update the user agent library for more modern options. Blocked user agents result in HTTP status code 403 and Cloudflare "error code: 1010".
Consider conducting research on which user agents have been compromised (1423 options).
Add breadth-first iteration to the Bulk Tickers views (510 & 520) to preserve order. The tickers in these views are sorted in a horizontal fashion, but are structured (HTML) vertically. i.e. You have an array of td
elements (columns) and within each element is an array of tr
elements (tickers). Implement a method to output a dataframe of tickers (and data) in the correct order specified.
Evaluate whether certain components should be abstracted, i.e., Client
, the backoff procedure, and/or the general request procedure.
The Gota library has a print()
method for dataframes, but the method is private. As such, each call to print a dataframe results in a shortened screen printing. Create a custom PrintDataFrame
function to override these actions and allow the dataframe (regardless of size) to print to screen.
Add descriptions to filters (or in some lookup) so that CLI users can get better understanding on different filter types.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.