Imported 1.5 Million hands.. REALLY slow..

[kraada]

The postgres user is a sandboxed user account - it's created in order to restrict the postgres processes from having access elsewhere. It's a feature, not a bug

As I did say originally in this thread, the first cluster really does take a while. I'm quite surprised at the 14 hour mark, though, as when I personally ran speed tests (using Beta 28.4), importing 1,322,364 hands in 8946 files, PT3 took 4 hours and 21 minutes to complete the import, for an import time of 84.4 hands per second. The full housekeeping took an additional 1 hour and 31 minutes, so the total import and housekeeping time for PT3 was 5 hours and 52 minutes, or 62.6 hands per second, which is a whole lot less than you.

Were you using a whole lot of very small files? I suppose if you had a ton of very small files and a lot of regulars at the stake in question, the cluster might have a whole lot more work to do to get things orderly, but I'm not sure.

Either way now that it's done, the next cluster really ought to take less time. Updating the cache will take about as long - do you have a lot of custom stats? That would increase cache update time.

[genemish]

Update...

i gave Postgres acct admin rights, rebooted (for it to kick in), created a 2nd DB (TEST) and imported a fresh set of hands (only about 25k).. i immediately noticed the postgres.exe process jumping in CPU usage from 10 to 30 thoughout the import AND housekeeping.. THIS WAS NEVER THE CASE BEFORE.. it would stay at 0-1%..

i think this was it.. dunno why it needed admin rights

i will try another big batch of hands tonight and will report.

Gene

[kraada]

Out of curiosity, do you have UAC enabled? I'm not sure that ever got asked; if so the right answer way back when may have been to disable UAC (tutorial) though giving the postgres user admin rights will largely do the same thing, though you'll still have the annoying popup box floating around at times . . .

[genemish]

UAC is disabled..

[kraada]

Seemed like it was worth asking. Try that big batch and let me know how things go now.

[genemish]

11:52pm -- just started import of 1.6MM hands (into the same main DB)..

both PokerTracker and postgres.exe processes are chewing at the CPU.. update in the morning.

wish me good luck!

Gene.

[WhiteRider]

Good luck, and do let us know..

[genemish]

Update
----

Importing 1.67MM hands took 3h:43m.. which is good.. both pokertracker and postgres processes were jumping during this...

housekeeping has been running for 2h:20m and is still on clustering..

CPU -- postgres.exe process uses only 1-2% at most.. is this really normal?

I will let this finish -- could be soon but i have a feeling its going to be 12+ hours... and then i will test a theory of creating a new DB and re-importing a big bulk, as it seemed to be working yesterday after i gave postgres admin rights.. perhaps the new DB will inherit proper rights?? I dunno, shot in the dark

also, just to get this out of the way, the "bulk" files are all txt, roughly 650 hands per file (2,500+ files in this batch)..

PC - HP xw6400, 2 x Xeon 5160 3.0GHz, 4GB RAM, Win 7..

[kraada]

The vast majority of the time consumed in clustering is in reading and writing files from your hard drive, so it does make sense that postgres would not be using a lot of CPU time during clustering. The cache update is much more CPU intensive so you should see more postgres CPU usage during cache updating.

Your computer is definitely more than sufficient in terms of power. Is this reimporting into the same database you were using earlier?

The postgres user having or not having admin rights when the database is created should not make a difference to the actual structure of the database at all, as far as I'm aware, but I'll definitely look into this a bit more and see if I can find anything.

[genemish]

So the main DB is the one that housekeeping is taking a long time with..

when i granted admin rights to postgres yesterday, i created a new test DB and imported 25k hands... the postgres.exe process was going nuts during the import AND the clustering part..

i will let this housekeeping finish and then create a new test DB and try importing again..