Releases · mendableai/firecrawl

Firecrawl V1 is here! With that we introduce a more reliable and developer friendly API.

August 29th, 2024

Here is what’s new:

Output Formats for /scrape. Choose what formats you want your output in.
New /map endpoint for getting most of the URLs of a webpage.
Developer friendly API for /crawl/{id} status.
2x Rate Limits for all plans.
Go SDK and Rust SDK
Teams support
API Key Management in the dashboard.
onlyMainContent is now default to true.
/crawl webhooks and websocket support.

Learn more about it here

Start using v1 right away at https://firecrawl.dev

What's Changed (including v0 + v1)

Delete .DS_Store by @szepeviktor in #8
[Bugfix] added normalized apikey to craw/status route by @rafaelsideguide in #12
[Feat] improving reative paths by @rafaelsideguide in #4
Fix typos by @szepeviktor in #9
[Feat] Added html to markdown table parser by @rafaelsideguide in #11
Option to extract only the main content, excluding headers, navs, footers etc. by @nickscamara in #14
[Feat] Adding pdf parser by @rafaelsideguide in #17
adding ci-cd workflow by @rafaelsideguide in #20
adding workflow by @rafaelsideguide in #21
adding env secrets by @rafaelsideguide in #22
[Feat] Added TSDocs and types for js-sdk by @rafaelsideguide in #28
Added option to replace all relative paths with absolute paths by @rafaelsideguide in #25
[Bugfix] Fixed scrape preview test by @rafaelsideguide in #30
Caleb: fixing some documentation and rebuilding the server by @calebpeffer in #32
Rate limit fixes for crawl status by @nickscamara in #36
Better logging by @nickscamara in #35
[Feat] Added type declarations by @rafaelsideguide in #31
Refactor api routes by @nickscamara in #37
Logging by @nickscamara in #38
Cjp/making db auth optional <> Running project locally by @calebpeffer in #40
chore: add context.close by @mattzcarey in #46
Fixes table parsing for websites such as news.ycombinator.com (HN) by @nickscamara in #52
[Feat] Server health check + slack message by @rafaelsideguide in #53
[Feat] Added blocklist for social media urls by @rafaelsideguide in #55
[Feat:mvp] Search Endpoint => serp api + firecrawl => 🔥 🔍 by @nickscamara in #56
[Feat] Added anthropic vision api by @rafaelsideguide in #5
[Bugfix] Trim and Lowercase all urls by @rafaelsideguide in #13
Implements the ability for the crawler to output all the links it found, without scraping by @nickscamara in #34
Serper params by @nickscamara in #62
Support for tbs, filter, lang, country and location with Serper search. by @rogerserper in #61
[Feat] Added allowed urls by @rafaelsideguide in #64
/search support in node sdk by @nickscamara in #72
Free credits increase by @nickscamara in #75
[Bugfix] JS-SDK: Remove dotenv and add tests by @mdp in #68
[Feat] Coupon system by @rafaelsideguide in #66
Specific website params support by @nickscamara in #83
Greenpay fixes by @nickscamara in #84
[Feat] Implemented retry attempts to handle 502 errors by @rafaelsideguide in #67
feat: LLM Extraction (mvp) by @nickscamara in #90
Update README.md by @bllchmbrs in #110
Add Posthog Logging by @ericciarla in #109
Refactor of main web scraper + Partial data streaming by @nickscamara in #120
[Feat] Added includeHTML option by @rafaelsideguide in #126
Cancel Job Route by @nickscamara in #129
[Feat] Added max depth option by @rafaelsideguide in #130
Add keyAuth endpoint by @ericciarla in #131
[Test] Added integration tests suite by @rafaelsideguide in #118
Adds Zod Integration for LLM Extraction in the Firecrawl JS SDK by @nickscamara in #135
[Docs] Updated examples by @rafaelsideguide in #137
Switching to AGPL - We Need Your Consent! by @calebpeffer in #134
Nsc/refactor scraping order by @nickscamara in #139
Update models.ts by @ericciarla in #144
Timeout on /scrape by @nickscamara in #145
[Doc] Added default value for crawlOptions.limit by @rafaelsideguide in #142
feat: 4x-5x faster crawler (fast mode) by @nickscamara in #149
Add Docker Compose for easy self hosting by @chand1012 in #119
refactor: fix typo in WebScraper/index.ts by @eltociear in #27
[Tests] Added crawl test suite -> crawl improvements by @rafaelsideguide in #153
feat: Docx Support by @nickscamara in #158
Fixes pdfs not found if .pdf is not present by @nickscamara in #29
Update README.md: Typo fix by @elimisteve in #160
[Feat] Added rate limits by @rafaelsideguide in #151
Allow override of API URL by @mattjoyce in #166
feat: HyperDX Integration by @nickscamara in #167
beta: Fire-Engine fallback by @nickscamara in #174
Add additional file extensions to crawler.ts by @tractorjuice in #77
[Bug] Fixing /crawl limit by @rafaelsideguide in #143
Update issue templates by @rafaelsideguide in #180
[Feat] Added proxy and media blocking support for Playwright by @JakobStadlhuber in #181
update: wait until body attached in playwright-service by @qyou in #170
feat: Allow privacy/legal/ other pages in social media websites by @nickscamara in #168
[Bug] Added data check for python SDK by @rafaelsideguide in #176
Fix FIRECRAWL_API_URL bug, also various PyLint fixes by @mattjoyce in #178
[Feat] Added idempotency key to crawl route by @rafaelsideguide in #132
Feat: Provide more details for 429 error msg by @simonha9 in #190
Limit on /search is not deterministic by @Keredu in #186
Various PyPi Metadata by @mattjoyce in #191
[Test] Added sdk e2e tests by @rafaelsideguide in #183
Allow users to manually set the waitFor param on /scrape by @nickscamara in #200
[Feat] Added custom scraping conditions for readme docs by @rafaelsideguide in #204
Feat/screenshot support by @ericciarla in #207
feat: New pricing/limits changes by @nickscamara in #216
[sdk] Fixes waiting status not being present on check status by @nickscamara in #218
Fixed fire-engine content bug by @rafaelsideguide in #228
Use @ instead of # for default BULL_AUTH_KEY. Hash mark is reserved...