Comments on: Scrape Performance Tweaks

By: ooklah

ooklah — Wed, 03 Nov 2004 18:10:55 +0000

Your downloading all of the mt forums…. what possible use does this have and why?

By: codepoetica

codepoetica — Thu, 21 Oct 2004 15:49:51 +0000

Looking for breaks in consecutive numbering I deemed complex a routine, so I opted for something a little simpl’r, if a bit more processor-hungry.

Before downloading possible-thread-X, I look through the list of posts I’ve already downloaded. If there is a post with ID number X there already, then it can’t be an undownloaded, actually existing thread. Thus, I skip it, and incremenent X until this test fails.

This doesn’t indicate an actual thread number, as there could be threads that were there, but have been deleted, but it reduces the number of unnessessary downloads by not pulling down pages that are already known to contain no threads.

By: Jay Random

Jay Random — Thu, 21 Oct 2004 07:10:02 +0000

Thus, we can start downloading threads numbering at the first number greater than the last thread downloaded, which is not found in the local list.

Are the postnumbers within threads really all consecutive? That would be impressive, considering that they weren’t under UBB. Do you scan for breaks in consecutive numbering in the local list before setting the the highest postnumber therein as the next potential thread number to retrieve?

Or am I mis-parsing how you described your shortcut?

By: Plains in the Dreaming » Scrape Progress: 100%!

Plains in the Dreaming » Scrape Progress: 100%! — Tue, 30 Nov 1999 00:00:00 +0000

[...] nder: coding cwdb — codepoetica @ 10:28h We have attained completion. Our goal of downloading the first 1.66 Million (possibly existing) forum [...]