Crawl and convert any website into LLM-ready markdown. Build by Mendable.ai
This repository is currently in its early stages of development. We are in the process of merging custom modules into this mono repository. The primary objective is to enhance the accuracy of LLM responses by utilizing clean data. It is not ready for full self-host yet - we're working on it
Firecrawl is an API service that takes a URL, crawls it, and converts it into clean markdown. We crawl all accessible subpages and give you clean markdown for each. No sitemap required.
Pst. hey, you, join our stargazers :)
![](https://private-user-images.githubusercontent.com/44934913/325014160-53c4483a-0f0e-40c6-bd84-153a07f94d29.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQyNDMyNDQsIm5iZiI6MTcyNDI0Mjk0NCwicGF0aCI6Ii80NDkzNDkxMy8zMjUwMTQxNjAtNTNjNDQ4M2EtMGYwZS00MGM2LWJkODQtMTUzYTA3Zjk0ZDI5LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjElMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODIxVDEyMjIyNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTY0YjgzODQyM2ZjMzJlNDJiMGE2Mjg1YmUwOWJmMjRjOThiZTZlMjVmZDEyMGEzMWI3NzBhMmFiODM4NmQ0ODMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.IvsVyy1HJUmzAkXe2Hs0Z9pB7mbCjqLs6oGaBZO6dtI)
We provide an easy to use API with our hosted version. You can find the playground and documentation here. You can also self host the backend if you'd like.
- API
- Python SDK
- Node SDK
- Langchain Integration ๐ฆ๐
- Llama Index Integration ๐ฆ
- Langchain JS Integration ๐ฆ๐
- Want an SDK or Integration? Let us know by opening an issue.
To run locally, refer to guide here.
To use the API, you need to sign up on Firecrawl and get an API key.
Used to crawl a URL and all accessible subpages. This submits a crawl job and returns a job ID to check the status of the crawl.
curl -X POST https://api.firecrawl.dev/v0/crawl \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer YOUR_API_KEY' \
-d '{
"url": "https://mendable.ai"
}'
Returns a jobId
{ "jobId": "1234-5678-9101" }
Used to check the status of a crawl job and get its result.
curl -X GET https://api.firecrawl.dev/v0/crawl/status/1234-5678-9101 \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer YOUR_API_KEY'
{
"status": "completed",
"current": 22,
"total": 22,
"data": [
{
"content": "Raw Content ",
"markdown": "# Markdown Content",
"provider": "web-scraper",
"metadata": {
"title": "Mendable | AI for CX and Sales",
"description": "AI for CX and Sales",
"language": null,
"sourceURL": "https://www.mendable.ai/"
}
}
]
}
Used to scrape a URL and get its content.
curl -X POST https://api.firecrawl.dev/v0/scrape \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer YOUR_API_KEY' \
-d '{
"url": "https://mendable.ai"
}'
Response:
{
"success": true,
"data": {
"content": "Raw Content ",
"markdown": "# Markdown Content",
"provider": "web-scraper",
"metadata": {
"title": "Mendable | AI for CX and Sales",
"description": "AI for CX and Sales",
"language": null,
"sourceURL": "https://www.mendable.ai/"
}
}
}
Used to search the web, get the most relevant results, scrap each page and return the markdown.
curl -X POST https://api.firecrawl.dev/v0/search \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer YOUR_API_KEY' \
-d '{
"query": "firecrawl",
"pageOptions": {
"fetchPageContent": true // false for a fast serp api
}
}'
{
"success": true,
"data": [
{
"url": "https://mendable.ai",
"markdown": "# Markdown Content",
"provider": "web-scraper",
"metadata": {
"title": "Mendable | AI for CX and Sales",
"description": "AI for CX and Sales",
"language": null,
"sourceURL": "https://www.mendable.ai/"
}
}
]
}
Coming soon to the Langchain and LLama Index integrations.
pip install firecrawl-py
from firecrawl import FirecrawlApp
app = FirecrawlApp(api_key="YOUR_API_KEY")
crawl_result = app.crawl_url('mendable.ai', {'crawlerOptions': {'excludes': ['blog/*']}})
# Get the markdown
for result in crawl_result:
print(result['markdown'])
To scrape a single URL, use the scrape_url
method. It takes the URL as a parameter and returns the scraped data as a dictionary.
url = 'https://example.com'
scraped_data = app.scrape_url(url)
Performs a web search, retrieve the top results, extract data from each page, and returns their markdown.
query = 'what is mendable?'
search_result = app.search(query)
We love contributions! Please read our contributing guide before submitting a pull request.
It is the sole responsibility of the end users to respect websites' policies when scraping, searching and crawling with Firecrawl. Users are advised to adhere to the applicable privacy policies and terms of use of the websites prior to initiating any scraping activities. By default, Firecrawl respects the directives specified in the websites' robots.txt files when crawling. By utilizing Firecrawl, you expressly agree to comply with these conditions.