"duplicates" when datamining FTP

[VoodooSG]

Hi there,

I just started, 1 hour ago, datamining on FTP using "spadeeye". Here are my import statistics :
hands : 1223
errors: 0
duplicates : 5939

Is this normal ???

I'm using the auto-import hand, all in a NEW database so why do I have that many duplicates ?

Thanks !

[WhiteRider]

That must include previous hands, yes? Do you have hands from multiple sources? Are they maybe played and observed at the same time or something?
If it's a new clean DB (are you sure?) then you shouldn't have duplicates unless you have the same hands from different sources.

[VoodooSG]

My c:/program files /.../hand history was empty before the datamining session, and no hands in my database (just created).

But like after 5 minutes I had a very high number of duplicates hand. meaby PT3 tried to import them again and again but it was already imported ?

My hands all comes from one sources (but a couple of files) : all from :My c:/program files /ftp.../hand history.

Yes im sure it is a new database.

ALSO :

I made a test 3 days ago. I imported a couple of hands in PT2 : 0 errors and around 30 000hands imported (coming from around 300 differents files). I try after with PT3 : I got a ratio of 5 good hand : 1 duplicates . Also I got more imported hands then when I imported the hands with PT2 ! But if you do the number of hand minus the duplicates, its close to 30 000 hands.... all done again in a fresh new database !

[franktastic]

I have also been getting duplicates on fulltilt, all my data is from the same source althought I have also been getting exception error shutdowns.

[VoodooSG]

Hi again, I just came back to home. I did, while auto-importing, datamining from 15:17PM to 2:30AM . Note that I was datamining 17 tables at a times, so I end up with 719 files into my "C:\Program Files\Full Tilt Poker\HandHistory" . PT3 was still importing hands 10 minutes after the closing of all the tables... is it normal ? And it was still running.

So after around 11 hours of datamining 17 tables, heres the numbers ! :
15 921 hands
0 errors
...64 635 duplicates !!!!

and after 10 minutes of closing all the tables :
16 151 hands
0 errors
65 558 duplicates

So after I stopped the auto-import , 106 of the 719 files were still in the "C:\Program Files\Full Tilt Poker\HandHistory" directories, the rest was moved fine to the "C:\Program Files\PokerTracker 3\Processed" directories. I just decided to manually import those 106 files, having that results :
4962 hands
0 errors
18173 duplicates

I just dont know what to do now...

thanks for the support

[VoodooSG]

Ok so I made 2 more test : testing auto importing when I got only 6 files in my "C:\Program Files\Full Tilt Poker\HandHistory" directories.

I imported around 500 hands without any duplicates !

Meaby PT3 got confused when they are too many files (719) ?

anyways !

waiting for solution

[WhiteRider]

PT3's auto import is designed to be low-intensity so that it does not impact on CPU usage very much. It may be that if your computer is slow/old or very busy while datamining a lot of tables the auto import is not keeping up properly (I know that at least one other person has reported this issue with a LOT of tables open). Auto import will keep reading hands from each file in turn (not all of one file at once) and you need to wait for it to catch up before stopping it, because only files that have been completely processed will be moved. If you stop and start the import before this point the files will not be moved and will all be read again.
Could that expain it? Did you stop/start the import?

[VoodooSG]

yes my computer is going slow when datamining 17 tables + auto-importing . I got WindowsXP/AMD Athlon 2800+/1gig of Ram/+good video card . old comp but not that much !

I did a test 1 week ago. I tried to import A LOT of hands (around 35 000) from around 400 files (using the auto-import again). I had a ratio of 5 good hands / 1 duplicates (5:1) all that in a new fresh database.

BUT :

This morning I'm having good result I dont know why !

Just tried to import 200 files with the auto-import :
So far I'm at 22 029 hands AND 0 duplicates !!!

I did some change (allowing more "freedom" to postgres) in my firewall but lol can it be the reason ? !!!

Anyway thanks for the fast anwser WhiteRider!

[WhiteRider]

If you are importing a lot of existing files/hands like that you should really use the manual import as it is MUCH faster.

[sirtoffel]

I have the same problem and no solution (although the slowing down might be the problem)

on another note, my PT3 isnt moving the processed files from PT3 although ive set it correctly to do so. Wtf?