Importing Duplicates Very Slow

[cl3537]

Dear Josh,

I am importing Fulltilt hand histories about 100,000 at a time to test the duplicate skipping. It is working but much slower than importing them originally.
Shouldn't importing duplicates be faster not slower.

Manual Import 443 Files 130,000 Hands (Originally about 12 Minutes To Import)
Second Time on the same batch took 1:32 to skip the the files as duplicates.

Also the number of hands are the same but PT2 has 4965 minutes but PT3 has 4898 minutes

Regards,
Jordan

[Josh]

When PT3 encounters a duplicate, it checks what is currently in the database. If the new hand has more information, it overwrites the old information. This checking is what causes duplicates to take longer.

What report in PT3 are you looking at to get the minutes?

[mofo]

Dunno if this is applicable, but when the original raw HH is saved in the database, just add some quickly calculable hash for the hand. Then while importing, just calc the hash and compare to the already existing hand's hash. Because I suspect most duplicate cases are when the hands are identical, this would speed things up quite a bit?

[Josh]

It don't think that would speed it up much actually, since the delay is caused by reading from the database for every hand. Your method would require that too.