mev-fyi / data Goto Github PK

The open-source repository mev.fyi aggregates research on Maximal Extractable Value (MEV). Explore curated academic papers, community contributions, and educational content on MEV and related topics.

License: MIT License

Shell 0.59% Python 88.01% Roff 0.03% Jupyter Notebook 11.36%

mev research research-and-development research-tools

data's Introduction

(1) View: data.mev.fyi, (2) Contribute: add.mev.fyi, (3) site: mev.fyi

MEV.FYI is the MEV research content aggregator and search engine

Welcome to mev.fyi - the open-source initiative dedicated to gathering and curating research on Maximal Extractable Value (MEV), incentive alignment, mechanism design, and their implications in the blockchain ecosystem, with a specific focus on Ethereum.

mev.fyi's end-game is to become the go-to resource for MEV research and related topics using a chatbot interface to recommend papers, articles, and videos based on user input (based on nfx.com/chat). This repository can be the basis to create guides and formal definitions of MEV and the relevant Ethereum research as a whole, for education purposes e.g. to formalise and agree on definitions, onboard new (re)searchers [...].

What is MEV?

Maximal Extractable Value (MEV) refers to the total amount that can be extracted from blockchain users by strategically ordering, including, or censoring transactions. It's an emerging area of study and concern in the blockchain space.

Project Goals

Research Aggregation: Gather the latest and most relevant academic papers, articles, and discussions surrounding MEV and related topics.
Community Contributions: Encourage and integrate contributions from the community to ensure the repository remains updated and relevant.
Education: Educate the broader blockchain and Ethereum community about the implications of MEV and how it can shape the future of decentralized platforms.

Where to start the MEV learning journey?

Flashbots' MEV Research repository is a great place to start. It contains a curated list of academic papers, articles, and discussions on MEV and related topics.
Beyond Flashbots, all articles available from sources in resources.md constitute strong foundations to get started on MEV research.
Research papers, can be browsed on the Google Sheet here. This file contains a list of research papers, the papers' title, authors, topics, and link.

YouTube Video Transcripts

We've curated a selection of YouTube channels focused on MEV and related topics. Each channel's videos come with their respective transcripts to enable easy "Ctrl+F" searches. This repository offers an invaluable tool to parse video content with LLMs, create glossaries, definitions, and more.

Disclaimer: each transcript is fetched from Youtube API and automatically generated. The Google algorithm generating said scripts is imperfect and some words can be misinterpreted.

Topics of Interest

Flashbots' topics and definitions list here enumerates topics of interest to the MEV research community. These topics include:

Auction Design
Cryptographic Privacy
Cryptoeconomic Privacy
MEV in L2
MEV Taxonomy
Account Abstraction
Protocol Design
Search Optimization

And, more broadly:

Incentive Alignment in Blockchain Systems
Mechanism Design and its Implications in Ethereum
Transaction Ordering and its Impacts
Auction theory and its Applications in Blockchain
Potential Solutions and Mitigations for MEV

How to Contribute with Research

We welcome contributions from everyone, irrespective of your background or expertise level! You can contribute either using the form here add.mev.fyi or using via GitHub as per the instructions below:

Fork the Repository: Start by forking this repository.
Clone & Create a New Branch: Clone the repository to your local machine and create a new branch.
Make Changes: Add your contributions namely
1. link to a relevant research paper in data/links/arxiv_papers.csv alongside your handle if you want to be noted as paper referrer.
2. Articles, discussion thread, forum post or any educational content can be added to data/links/articles.csv alongside your handle if you want to be noted as article referrer.
Push & Create a Pull Request: Push your changes to your fork and then open a pull request against this repository.

How to Contribute to the tools around Research

There is maintained work item list here that you can pick up and work on. If you want to add a new item, please open an issue and we will discuss it.

Support the Initiative

Love what we're doing? Star the repository and spread the word! Your support helps in keeping this initiative active and growing.

data's People

Contributors

Stargazers

Watchers

Forkers

freddmannen ballestar neeldaftary

data's Issues

[feature] add safeguards when re-fetching PDFs articles to guarantee content's integrity

Right now at every manual fetching of articles via get_articles_content, we overwrite the previous content with the latest fetched one, which propagates past edits.

We would need to add safeguards around making sure that the scrapping process does not lose a significant chunk of the content in the event of malfunction.

[Data ingestion] scrap L2s forum to add governance content

To add governance-related content, we can scrap Discourse forums e.g. Optimism, Arbitrum, Ethereum etc.

[Data ingestion] add further articles

Because some articles are hosted on specific journals or websites which aren't a Discourse, Mirror, Medium backend and so forth, they might be referenced in the CSV aggregate but without their PDF counterpart. They need to be added manually

[Data ingestion] add website referencing

Similarly to indexing by authors, the idea is to have a small description for each website such that the end-user can ask questions like "Return all the VCs investing in MEV" or "What websites write content on shared sequencers"

[Data ingestion] add further research papers

Because some research papers are hosted on specific journals or websites which aren't SSRN, arxiv and so forth, they might be referenced in the CSV aggregate but without their PDF counterpart

[feature] automatically map and scrap the content of any docs

So far we only parsed content on a per URL basis. An update to that would be to parse protocols' documentations by simply parsing the top-most documentations page e.g. the landing page for the docs, get the tree of all URLs in that docs, then scrap and add into database.

[New indexing] Create a mapping of all authors and their socials

The idea is to be able to ask questions like "Return the sources of ", "Who are authors about ", "Who works at RIG" kind of questions.
Essentially it requires creating a new table by parsing the authors from each existing CSV aggregate

[Data ingestion] ZK knowledge

Scrap the relevant sources (TBD) to obtain ZK specialisation

[refactor] Articles processing

The current articles processing has copy pasted methods while we can simplify down with config json and adapting to a few edge cases. Most of the time it is mostly about changing the CSS selector for each title, author, release date (+ correct formatting into yyyy-mm-dd), and content.

We can make something cleaner there where each CSS selector and date format is a parameter.
Even better: automate the process with a scrapping tool

[feature] Create a React front-end for data.mev.fyi Google Sheet

Currently the Google Sheet of data.mev.fyi showing all data aggregates (articles, research papers, and so forth) isn't necessarily the nicest from a user experience perspective namely there is no user authentication, it would not remember filtering, and so forth. A nicer react frontend e.g. a second page on mev.fyi would drastically improve the experience of browsing the content database

[Data ingestion] process twitter threads

Scrap Twitter threads in a readable format for the backend e.g. save as PDF
Would be great to explicitly indicate whether that tweet or thread is an answer to another tweet or thread, in which case it would be relevant to add as first post in the chronological order

[feature] automatically scrap the referenced websites to fetch the latest published articles

Right now we manually add articles from websites. We can add a functionality to scrap the websites and get the latest articles without manually adding them. We can implement the same curation system as for YouTube video processing

[feature] implement autodoc to automatically generate docs from github code repositories

We can implement https://github.com/context-labs/autodoc to automatically generate documentations from github code repositories. That would be one step closer to make it easier to onboard developers to any protocol