Slow Import Speed

[pat3392]

Heya,

I'm trying to import a large number of hands, a little over 900k hand history files, but as the import progresses the expected completion time somehow increases. I've tried importing it a few times on different computers, with the same results. With this current import, the expected completed time was seventy something hours, it has now reached 80 and the expected time has increased to 540 hours.

[WhiteRider]

Importing very large amounts of files can slow a bit over time. If you stop it now it won't do any harm. Make a note of the last file it processed and then start the import again and just select the remaining files or folders.
If you have the housekeeping options enabled in the Manual Import tab (which is definitely recommended) then disable them before you stop the import. If you're going to be importing more hands again straight away there is no point doing housekeeping now as it will need to be done again after the remaining hands are imported. Be sure to turn the options on again before you start the remaining import, though.

[pat3392]

So you're telling me that there's nothing I can do, and that I simply have to wait longer than 20 days to import it all, and at this rate, longer than two months? The import speed itself decreases as time passes, this can't be averted?

[WhiteRider]

If you stop importing and import the remainder of the files it won't take that long.

[pat3392]

Why so? Does it matter that my hold'em cache is at 0%? This is a rather tedious process, made particularly cumbersome by my hand histories not being organised well at all, can PT3 auto-import/stop/reimport? Once this is finally done(expecting very roughly 100 million hands) is it going to take a year to import this into PT4? If PT isnt't designed for so many hands, what are my options? Will I have to use seperate databases for different hands and if so, would that slow my computer down at all? Can I have auto-import import hands into two different databases? For example, my recent hand database and ongame/888 database. Would I be able to purchase one of those database machines to process this information for me quickly?

[kraada]

Honestly if you are just getting started importing a hand volume of this quantity you might want to just start off with PT4 in our next beta release. To my knowledge the largest database we've recorded anybody as having was 40M hands. The system should be able to handle databases of that size (assuming you have the hard drive space - a 100M hand database of all full ring hands would be approximately 1TB in size), but importing will definitely take time.

I'd think the best way to do it would be to run batches of imports, with maybe 5000 or 10000 files per batch. That way each batch finishes reasonably quickly but when you want to stop it's not that long until your next batch finishes.

PT4 will (when you take housekeeping time into account) end up being faster at the total import time than PT3 is and this will save you the need to do it twice. The only reason I say to wait for the next beta is that there is one small import error corrected for the next beta that requires a reimport to fix (specifically it's with the count of the number of players on streets where a player was blinded all in - if that case doesn't bother you, you can use our current beta).

[pat3392]

When will the next beta be approixmately released?

[WhiteRider]

I don't have a confirmed date for it yet, but I expect it will be some time this week (subject to successful internal testing).

[pat3392]

I'm not sure how accurate the 100 million hand thing is; I have a little under 100GB in hand histories, any idea how much hard-drive space that'll take up?

[WhiteRider]

I don't have an estimate for that - it's possible Kraada will, but I don't remember having a figure for that as it is very dependent on the site and type of games. I would expect that to take a very large amount of space, though - text hand histories don't take up much space so that sounds like a lot of hands.