Manual Import from Directory very slow

[PlayingAround]

Hi,
I just bought PT4 and am now trying to import some hands manually. I deinstalled PT3 and Postgres 8.4 and then reinstalled Postgres 9.1.08. I then installed PT4 4.02.1. BEfore importimg hands I manually changed the postgres setting to (I use a Core i5 with 12 gb ram):

shared_buffers = 2000MB # min 128kB
#work_mem = 100MB # min 64kB
#maintenance_work_mem = 640MB # min 1MB

I started the import in PT4 into a newly created db and after 20000 hands the importspeed has droped to 28 h/s and the remaining time has increased to 2:53:40.

Is this speed - if you can call it that - normal ?

Thx for any help.

Ciao
Lars

[kraada]

You need to take the # away from the beginning of the work_mem and maintenance_work_mem lines for those to be used - anything after a # is not acted upon.

Are these hands original played hands, exported hands from another database or datamined hands? Exported hands will import the fastest of the three - and considerably so - so if those were the kinds of hands you were looking at I would be worried based on the speeds you quote. On the other hand, sites that send you datamined hand histories often give them out of order in a single file and that causes considerable slowdowns during import and then your speeds would not surprise me.

[PlayingAround]

The hands are the original Pokerstars hands, as saved by the pokerstars client. They were imported and moved to the processed folder by the PT4 beta. I'll remove the # and start over again. Since I choose a directory to import from I'll have a bunch of duplicate hands but oh well, they are skipped pretty fast. Import speed is down to 17 h/s btw. I'll report back on the new times.

The hands are stored on a samsung spinpoint hd, the db on a crucial m4 ssd btw.

[kraada]

What are you using for antivirus software? If that is configured to read every file on opening (or writing, or both) that will slow down imports considerably. You'll want to exclude the PostgreSQL directory from those scans and you'll see a big improvement immediately after doing so.

[PlayingAround]

The import finished: 200000 hands, 160000 duplicates, 1000 errors, 616 tourney summaries in 3:03:38 = 19 h/s.

I use Avira AntiVir personal editon and it seems it is configured for on access scan. The option must have been set during one of the automatic updates because I certainly did not set it.

Just to be sure - because I want to import all of my hands after I'm sure everything is working right - I will create a new DB and import the directory again. I'll report back...

[PlayingAround]

60000 hands so far, speed 31 h/s.

Any other ideas?

[PlayingAround]

Tried again. Used the "tune db" option in PT4, rebooted the PC and started the import from the same folder.

Import started with 38 h/s and is now (186k imported) at 19 h/s.

I use Win7 64 bit, could that have anything to do with the problem?

On a side note: it's a bad design choice imo to aut. import from subfolders without visually showing that this is so. A choice to do so (checkbox "include subfolders") would be much more user friendly.

[WhiteRider]

Did you install the 64-bit version of PostgreSQL? I assume you did, and that will give somewhat better performance.
That import speed is slower than I'd expect for your machine's specs, but it does depend on the site and type of hands, and the order the hands are in. Are these original hand history files or exports?

[PlayingAround]

Yes, I installed the 64bit Version of PostgreSQL. The Hands are 70% Zoom and the Rest are MTT's.

The hands are the original Pokerstars hands, as saved by the pokerstars client. They were imported and moved to the processed folder by the PT4 beta.

[ElHive]

My guess is MTTs are slowing the import down especially if you play MTTs with huge fields like the Sunday Storm. The tournament summaries have every single player in them so with an average field of 30K players tourneys like the Storm tend to slow the import down dramatically IMO (or let's say they reduce the hands/h-ratio dramatically since during the import of a huge summary there are no hands imported).