Question about roboto and memory usage

[joshterrill]

Hello,

I didn't find anything in the docs, so I figured I would ask the question here. When the docs refer to the process of "Downloading and processing" the page, where does that information go? And does it get cleared after it's been "processed"? How is roboto when it comes to memory consumption? Do these tasks require a lot of memory? And if you "download and process" a ton of pages (like just letting it run forever and scrape every link it can find) will memory eventually become an issue?