Code Monkey home page Code Monkey logo

metagodoc's Introduction

MetaGodoc

This is an attempt to create an alternative to https://godoc.org with more features and stuff.

The primary inspiration is Perl's https://metacpan.org site, which provides a search engine along with a display of all sorts of useful info about the packages it indexes.

Like MetaCPAN, this project uses Elasticsearch as the primary backend for indexing information about packages. If it ever evolves to the point of having logins and such we may also need a SQL database of somet sort (or maybe not).

Random Thoughts on Future Features

Here's a brain dump of things I'd like to achieve with this project ...

All Versions Indexed

For packages which are tagging releases, it'd be really nice to allow users to browse the documentation of each release. We'll always want to index the HEAD of the default branch as well, and that should be the default view for most packages.

How this plays nicely with packages that use gopkg.in is very much TBD, but obviously it should.

Rich Metadata

One of the things I really like about MetaCPAN is the sidebar for a package. See https://metacpan.org/release/Moose for example. The left hand side links to its changelog, website, git repo, reported issues, package review, and more.

While Golang packages don't have all those things (changelogs, sigh), we can certainly link to the repo, offer information about issues, etc.

Better Documentation Organization

The default organization of go Package documentation is not very helpful. Presenting things in an arbitrary order is generally not the best way for users to understand the package.

This is especially problematic for monster packages like https://godoc.org/gopkg.in/olivere/elastic.v6. This is so huge that Chrome has trouble scrolling it smoothly.

And finding anything in it is really difficult. Now, in part, the onus to improve this is on the package author. Splitting this one package up into multiple packages would greatly improve the presentation.

However, it would still be nice to allow package authors more control over the presentation of their documentation.

My vague thought is that package authors could add a metagodoc.json file to their repo that would provide instructions on how to present their documentation. For example, they could specify something like "show the docs for these symbols" first or even provide a more sophisticated grouping of symbols to be split up across multiple pages.

This is all very handwavy at this point, but it's something that is really needed.

Better Search

I think using Elastic has the potential to make for much better search results. We can consider many things:

  • Location of words in the text (package name vs synopsis vs buried somewhere in there).
  • Other content in the repo, such as README files and other docs.
  • Import count - how many other packages use this one? Can we come up with a River of Go like the River of CPAN?
  • Fork vs not fork.
  • How active is the repository?

And yes, I know godoc.org already factors some of those things in.

Modern UI

At the very least, a wider central column.

Random Other Stuff

It'd be nice to be able to see all the repos from a single author, like MetaCPAN's author view.

Insert Your Feature here.

metagodoc's People

Contributors

autarch avatar

Stargazers

 avatar Faris Sufyan avatar astrolemonade avatar Emanuel Bennici avatar  avatar hucsmn avatar  avatar Dmitry Zinin avatar mishudark avatar Krzysztof Kowalczyk avatar Lucas Bremgartner avatar Asdine El Hrychy avatar Kaveh Shahbazian avatar  avatar Gilles Fabio avatar Ellison avatar Cuong Manh Le avatar Damian Gryski avatar Serge Simard avatar gato avatar Burak Sezer avatar Sean Chittenden avatar Jeff Rouse avatar

Watchers

 avatar Farhan Ahmad avatar James Cloos avatar  avatar  avatar

Forkers

telluz moongj

metagodoc's Issues

Pure golang indexer

Why not use bleve ?
Would be good to remove the JVM and elastic.

With bleve you can embed the index into the web page even.

Potential ReDoS Vulnerability or Inefficient Regular Expression in Project: Need for Assessment and Mitigation

Hello,

I am writing to report a potential Regular Expression Denial of Service (ReDoS) vulnerability or Inefficient Regular Expression in the project. This issue arises when specially crafted input strings are used in the context of distributed, high-volume requests, potentially leading to a denial-of-service attack.

Location of Issue:

The vulnerability is related to a regular expression used in the following validation file, which may result in significantly prolonged execution times under certain conditions.

re := regexp.MustCompile(`(?s)<(?:style|script)[^<>]*>.*?</(?:style|script)>|</?[a-z][a-z0-9]*[^<>]*>|<!--.*?-->`)

PoC Files and Comparisons:

// Proof of concept
filename := os.Args[1]
content, err := ioutil.ReadFile(filename)
re := regexp.MustCompile("(?s)<(?:style|script)[^<>]*>.*?</(?:style|script)>|</?[a-z][a-z0-9]*[^<>]*>|<!--.*?-->")
re.ReplaceAllString(string(content), "")

PoC Files Here:
poc.zip

To evaluate the performance of this inefficient regular expression matching with varying input contents, the following commands can be executed within the PoC folder:

time ./poc AttackString10MB.txt
# real    72m38.173s
# user    72m30.083s
# sys     0m5.653s
time ./poc RandomString10MB.txt
# real    0m0.029s
# user    0m0.016s
# sys     0m0.026s
time ./poc AttackString1MB.txt
# real    0m54.028s
# user    0m53.917s
# sys     0m0.088s
time ./poc RandomString1MB.txt
# real    0m0.011s
# user    0m0.007s
# sys     0m0.011s

The significant difference in processing time between random strings and malicious strings highlights the potential effectiveness of this regex for malicious exploitation. And as string length grows, the nonlinear increase in processing time reflects potentially greater risks.

Proposed Solution:

A possible mitigation strategy could include limiting the input length to prevent excessive processing times. If the corresponding function or feature is not in use, it is recommended to clean up risky third-party packages or code content to prevent malicious exploitation through methods such as code injection.

Additional Considerations:

Historically, it was believed that using regex engines with non-backtracking implementations (such as those in Rust or Go) would not lead to ReDoS vulnerabilities. However, recent studies have shown that this is not always the case. I recommend an assessment of how this issue might impact this project.

Thank you for your attention to this matter. Your evaluation and response to this potential security concern would be greatly appreciated.

Best regards,

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.