Duplicate Hands in import

[booger]

O.K. so these are the results from importing over 4000 hands. Can anyone tell me why only 900 were imported, why the original fiels are still in the hand history directory and have not been moved to the "processed" directory and how do I get rid of the duplicate hands? I think I have about 5Kk duplicate hands in my database right now!


2008/07/15 19:26:40: Import complete. 900 hands in 17 files were imported. (0 errors, 0 duplicates)
2008/07/15 19:26:41:
2008/07/15 19:26:41: Clustering...
2008/07/15 19:27:02: Clustering complete.

At this point I fugured maybe PT imports em 900 at a time so I ran the manual import again:

2008/07/15 19:27:32: Import beginning
2008/07/15 19:27:32: Import file: D:\FullTilt\HandHistory\LHS1552.tmp
2008/07/15 19:27:32: + Complete (1 hands, 0 errors, 37 duplicates)
2008/07/15 19:27:32: Import file: D:\FullTilt\HandHistory\LHS3537.tmp
2008/07/15 19:27:34: + Complete (3 hands, 0 errors, 107 duplicates)
2008/07/15 19:27:34: Import file: D:\FullTilt\HandHistory\LHS3B13.tmp
2008/07/15 19:27:34: + Complete (1 hands, 0 errors, 7 duplicates)
2008/07/15 19:27:34: Import file: D:\FullTilt\HandHistory\LHS4128.tmp
2008/07/15 19:27:35: + Complete (2 hands, 0 errors, 47 duplicates)
2008/07/15 19:27:35: Import file: D:\FullTilt\HandHistory\LHS5B7B.tmp
2008/07/15 19:27:37: + Complete (3 hands, 0 errors, 125 duplicates)
2008/07/15 19:27:37: Import file: D:\FullTilt\HandHistory\LHS66B.tmp
2008/07/15 19:27:37: + Complete (1 hands, 0 errors, 15 duplicates)
2008/07/15 19:27:37: Import file: D:\FullTilt\HandHistory\LHS7B1A.tmp
2008/07/15 19:27:38: + Complete (1 hands, 0 errors, 43 duplicates)
2008/07/15 19:27:38: Import file: D:\FullTilt\HandHistory\LHS829F.tmp
2008/07/15 19:27:38: + Complete (1 hands, 0 errors, 1 duplicates)
2008/07/15 19:27:38: Import file: D:\FullTilt\HandHistory\LHS842A.tmp
2008/07/15 19:27:40: + Complete (4 hands, 0 errors, 72 duplicates)
2008/07/15 19:27:40: Import file: D:\FullTilt\HandHistory\LHS86D7.tmp
2008/07/15 19:27:40: + Complete (2 hands, 0 errors, 48 duplicates)
2008/07/15 19:27:40: Import file: D:\FullTilt\HandHistory\LHS94DF.tmp
2008/07/15 19:27:41: + Complete (2 hands, 0 errors, 37 duplicates)
2008/07/15 19:27:41: Import file: D:\FullTilt\HandHistory\LHSAA98.tmp
2008/07/15 19:27:43: + Complete (0 hands, 0 errors, 103 duplicates)
2008/07/15 19:27:43: Import file: D:\FullTilt\HandHistory\LHSAC35.tmp
2008/07/15 19:27:44: + Complete (2 hands, 0 errors, 51 duplicates)
2008/07/15 19:27:44: Import file: D:\FullTilt\HandHistory\LHSBD60.tmp
2008/07/15 19:27:45: + Complete (4 hands, 0 errors, 69 duplicates)
2008/07/15 19:27:45: Import file: D:\FullTilt\HandHistory\LHSCCA2.tmp
2008/07/15 19:27:46: + Complete (2 hands, 0 errors, 41 duplicates)
2008/07/15 19:27:46: Import file: D:\FullTilt\HandHistory\LHSD510.tmp
2008/07/15 19:27:46: + Complete (1 hands, 0 errors, 7 duplicates)
2008/07/15 19:27:46: Import file: D:\FullTilt\HandHistory\LHSF74.tmp
2008/07/15 19:27:48: + Complete (1 hands, 0 errors, 90 duplicates)
2008/07/15 19:27:48: Import complete. 31 hands in 17 files were imported. (0 errors, 900 duplicates)
2008/07/15 19:27:48:
2008/07/15 19:27:48: Clustering...
2008/07/15 19:28:08: Clustering complete.

[WhiteRider]

Manual Import does not move processed files, you need to do this yourself after it completes.
I don't know why those 31 files were missed the first time - was there anything in the import log before that?

[booger]

I dont really remember if there was anything. I will purge the database and try again then I will post the results. If it again only imports a part of the hands I guess it's a bug?

[WhiteRider]

Yes, although invalid hands will not be imported. But certainly if you import the same set of files twice and you get any hands imported in the second run that weren't imported in the first run then that is a problem.

[booger]

O,K so I purged the database and mined FT for another 4,000 hands. I have added a link to some screen shots... they are free so dont expect the greatest quality

http://profile.imageshack.us/user/booger_PT3/

After I had about 4K hands I ran the manual import from directory. It imported 553 hands and then got stuck... I have the screen shots for about 30 minute gap on the link, I also have a screenshot of the Hand History directory so you can look at the size of each file and maybe you can make a rough estimate about how many hands it holds.

2008/07/16 12:26:51: Import beginning
2008/07/16 12:26:51: Import file: D:\FullTilt\HandHistory\LHS291F.tmp
2008/07/16 12:26:53: + Complete (116 hands, 0 errors, 0 duplicates)
2008/07/16 12:26:53: Import file: D:\FullTilt\HandHistory\LHS3473.tmp
2008/07/16 12:26:53: + Complete (25 hands, 0 errors, 0 duplicates)
2008/07/16 12:26:53: Import file: D:\FullTilt\HandHistory\LHS3DF1.tmp
2008/07/16 12:26:54: + Complete (39 hands, 0 errors, 0 duplicates)
2008/07/16 12:26:54: Import file: D:\FullTilt\HandHistory\LHS5ACD.tmp
2008/07/16 12:26:54: + Complete (9 hands, 0 errors, 0 duplicates)
2008/07/16 12:26:54: Import file: D:\FullTilt\HandHistory\LHS60A2.tmp
2008/07/16 12:26:55: + Complete (32 hands, 0 errors, 0 duplicates)
2008/07/16 12:26:55: Import file: D:\FullTilt\HandHistory\LHS692F.tmp
2008/07/16 12:26:56: + Complete (122 hands, 0 errors, 0 duplicates)
2008/07/16 12:26:56: Import file: D:\FullTilt\HandHistory\LHS756D.tmp
2008/07/16 12:26:56: + Complete (17 hands, 0 errors, 0 duplicates)
2008/07/16 12:26:56: Import file: D:\FullTilt\HandHistory\LHS8321.tmp
2008/07/16 12:26:57: + Complete (26 hands, 0 errors, 0 duplicates)
2008/07/16 12:26:57: Import file: D:\FullTilt\HandHistory\LHS8544.tmp
2008/07/16 12:26:58: + Complete (19 hands, 0 errors, 0 duplicates)
2008/07/16 12:26:58: Import file: D:\FullTilt\HandHistory\LHS8913.tmp
2008/07/16 12:26:58: + Complete (38 hands, 0 errors, 0 duplicates)
2008/07/16 12:26:58: Import file: D:\FullTilt\HandHistory\LHS9C9B.tmp
2008/07/16 13:08:43: Import interrupted. 553 hands in 11 files were imported. (0 errors, 0 duplicates)
2008/07/16 13:08:44:

After waiting 40 minutes I ended the import and did another manual import.

2008/07/16 13:08:43: Import interrupted. 553 hands in 11 files were imported. (0 errors, 0 duplicates)
2008/07/16 13:08:44:
2008/07/16 13:09:38: Import beginning
2008/07/16 13:09:38: Import file: D:\FullTilt\HandHistory\LHS25FF.tmp
2008/07/16 13:09:38: + Complete (27 hands, 0 errors, 0 duplicates)
2008/07/16 13:09:38: Import file: D:\FullTilt\HandHistory\LHS2757.tmp
2008/07/16 13:09:38: + Complete (19 hands, 0 errors, 0 duplicates)
2008/07/16 13:09:38: Import file: D:\FullTilt\HandHistory\LHS3473.tmp
2008/07/16 13:09:40: + Complete (88 hands, 0 errors, 25 duplicates)
2008/07/16 13:09:40: Import file: D:\FullTilt\HandHistory\LHS541D.tmp
2008/07/16 13:09:40: + Complete (30 hands, 0 errors, 0 duplicates)
2008/07/16 13:09:40: Import file: D:\FullTilt\HandHistory\LHS568F.tmp
2008/07/16 13:09:41: + Complete (32 hands, 0 errors, 0 duplicates)
2008/07/16 13:09:41: Import file: D:\FullTilt\HandHistory\LHS5ACD.tmp
2008/07/16 13:09:43: + Complete (86 hands, 0 errors, 9 duplicates)
2008/07/16 13:09:43: Import file: D:\FullTilt\HandHistory\LHS5F4A.tmp
2008/07/16 13:09:43: + Complete (11 hands, 0 errors, 0 duplicates)
2008/07/16 13:09:43: Import file: D:\FullTilt\HandHistory\LHS60A2.tmp
2008/07/16 13:09:44: + Complete (109 hands, 0 errors, 32 duplicates)
2008/07/16 13:09:44: Import file: D:\FullTilt\HandHistory\LHS653.tmp
2008/07/16 13:09:45: + Complete (9 hands, 0 errors, 0 duplicates)
2008/07/16 13:09:45: Import file: D:\FullTilt\HandHistory\LHS69C0.tmp
2008/07/16 13:09:46: + Complete (46 hands, 0 errors, 0 duplicates)
2008/07/16 13:09:46: Import file: D:\FullTilt\HandHistory\LHS718B.tmp
2008/07/16 13:09:46: + Complete (45 hands, 0 errors, 0 duplicates)
2008/07/16 13:09:46: Import file: D:\FullTilt\HandHistory\LHS756D.tmp
2008/07/16 13:09:48: + Complete (91 hands, 0 errors, 17 duplicates)
2008/07/16 13:09:48: Import file: D:\FullTilt\HandHistory\LHS998E.tmp
2008/07/16 13:09:48: + Complete (7 hands, 0 errors, 0 duplicates)
2008/07/16 13:09:48: Import file: D:\FullTilt\HandHistory\LHS9C9B.tmp
2008/07/16 13:09:50: + Complete (39 hands, 0 errors, 110 duplicates)
2008/07/16 13:09:50: Import file: D:\FullTilt\HandHistory\LHSAD58.tmp
2008/07/16 13:09:52: + Complete (93 hands, 0 errors, 0 duplicates)
2008/07/16 13:09:52: Import file: D:\FullTilt\HandHistory\LHSCA14.tmp
2008/07/16 13:09:52: + Complete (48 hands, 0 errors, 0 duplicates)
2008/07/16 13:09:52: Import file: D:\FullTilt\HandHistory\LHSD7D6.tmp
2008/07/16 13:09:52: + Complete (1 hands, 0 errors, 0 duplicates)
2008/07/16 13:09:52: Import file: D:\FullTilt\HandHistory\LHSFABD.tmp
2008/07/16 13:09:53: + Complete (13 hands, 0 errors, 0 duplicates)
2008/07/16 13:09:53: Import file: D:\FullTilt\HandHistory\LHSFB52.tmp
2008/07/16 13:09:53: + Complete (39 hands, 0 errors, 0 duplicates)
2008/07/16 13:09:53: Import complete. 833 hands in 19 files were imported. (0 errors, 193 duplicates)
2008/07/16 13:09:54:
2008/07/16 13:09:54: Clustering...
2008/07/16 13:10:05: Clustering complete.

This time 833 hands are imported, of which 19 appear to be new hands and 193 are duplicates - is this a bug since PT3 is not supposed to import any duplicates?

Now I purge the database and after copy pasting the TMP files at another location, do an auto import.

2008/07/16 13:09:53: Import complete. 833 hands in 19 files were imported. (0 errors, 193 duplicates)
2008/07/16 13:09:54:
2008/07/16 13:09:54: Clustering...
2008/07/16 13:10:05: Clustering complete.
2008/07/16 13:45:49: Full Tilt: Starting auto-import.
2008/07/16 13:45:49: Auto-Import beginning
2008/07/16 13:45:49: Full Tilt: Found 19 files to monitor.
2008/07/16 13:45:49: Full Tilt: Loading hand history file: D:\FullTilt\HandHistory\LHS2757.tmp
2008/07/16 13:45:49: Full Tilt: Loading hand history file: D:\FullTilt\HandHistory\LHS28B6.tmp
2008/07/16 13:45:49: Full Tilt: Loading hand history file: D:\FullTilt\HandHistory\LHS42AC.tmp
2008/07/16 13:45:49: Full Tilt: Loading hand history file: D:\FullTilt\HandHistory\LHS541D.tmp
2008/07/16 13:45:49: Full Tilt: Loading hand history file: D:\FullTilt\HandHistory\LHS59C9.tmp
2008/07/16 13:45:49: Full Tilt: Loading hand history file: D:\FullTilt\HandHistory\LHS5F4A.tmp
2008/07/16 13:45:49: Full Tilt: Loading hand history file: D:\FullTilt\HandHistory\LHS718B.tmp
2008/07/16 13:45:49: Full Tilt: Loading hand history file: D:\FullTilt\HandHistory\LHS7542.tmp
2008/07/16 13:45:49: Full Tilt: Loading hand history file: D:\FullTilt\HandHistory\LHS78DF.tmp
2008/07/16 13:45:49: Full Tilt: Loading hand history file: D:\FullTilt\HandHistory\LHS8314.tmp
2008/07/16 13:45:49: Full Tilt: Loading hand history file: D:\FullTilt\HandHistory\LHS8B05.tmp
2008/07/16 13:45:49: Full Tilt: Loading hand history file: D:\FullTilt\HandHistory\LHS8C7D.tmp
2008/07/16 13:45:49: Full Tilt: Loading hand history file: D:\FullTilt\HandHistory\LHS998E.tmp
2008/07/16 13:45:49: Full Tilt: Loading hand history file: D:\FullTilt\HandHistory\LHS9C9B.tmp
2008/07/16 13:45:49: Full Tilt: Loading hand history file: D:\FullTilt\HandHistory\LHSA9F9.tmp
2008/07/16 13:45:49: Full Tilt: Loading hand history file: D:\FullTilt\HandHistory\LHSCA14.tmp
2008/07/16 13:45:49: Full Tilt: Loading hand history file: D:\FullTilt\HandHistory\LHSD7D6.tmp
2008/07/16 13:45:49: Full Tilt: Loading hand history file: D:\FullTilt\HandHistory\LHSEEAB.tmp
2008/07/16 13:45:49: Full Tilt: Loading hand history file: D:\FullTilt\HandHistory\LHSFABD.tmp
2008/07/16 13:49:23: Full Tilt: Loading hand history file: D:\FullTilt\HandHistory\LHS7FE3.tmp
2008/07/16 13:54:12: Full Tilt: Loading hand history file: D:\FullTilt\HandHistory\LHSE932.tmp
2008/07/16 13:56:27: Full Tilt: Loading hand history file: D:\FullTilt\HandHistory\LHSF8EB.tmp

It has been like this for about 10 minutes now.

What does this mean? Do I need to stop mining before any hands will be actually imported into PT3?

[WhiteRider]

It looks like you left the mining going while you did the imports, is that right?
There are 19 files in the second import, but only 18 in your image of the folder, and also at least one of the files in the first import isn't included in the second import, so I guess that means you closed the table?

This time 833 hands are imported, of which 19 appear to be new hands and 193 are duplicates - is this a bug since PT3 is not supposed to import any duplicates?

The 19 you see is the number of files imported, not the number of hands - 833 hands were imported. The 193 duplicates are the hands which were imported in the first manual import which you had to abandon.

Auto import is not designed to do a bulk import like that - it is designed to run while the hands are being generated so it reads them one at a time.
If you want to test the manual import, generate a set of hand history files then stop the import - you will need to move the files before closing the tables so they don't get deleted. Then do a manual import on that static set of files.

[booger]

It looks like you left the mining going while you did the imports, is that right?

Yes i did.

There are 19 files in the second import, but only 18 in your image of the folder

I think that's because I took the image before I started the import. Another file must have been created in the meantime.

at least one of the files in the first import isn't included in the second import, so I guess that means you closed the table?

no I did not close any tables or any trny lobbies. I let them finish and then let FT close the lobbies.

The 19 you see is the number of files imported, not the number of hands - 833 hands were imported. The 193 duplicates are the hands which were imported in the first manual import which you had to abandon.

Duh' I am such a duffer so there are 193 duplicates and not 19. I thought PT3 was not supposed to import duplicates, did it happen because I ran different types of imports on the same files?

If you want to test the manual import....

I will do that and post the results.

P.S. Thanks WhiteRider you are doing a great job... I dont know what everyone would do without your advise

[WhiteRider]

so there are 193 duplicates and not 19. I thought PT3 was not supposed to import duplicates, did it happen because I ran different types of imports on the same files?

It doesn't import the duplicates - it recognises them and ignores the.
It found duplicates because you ran a manual import on a set of files, and then ran it again on the same files, so it re-read the same files which include (obviously) the same hands that have already been imported.

[booger]

ahhh that explains it but then why did it only recognize 193 duplicates and not all 553 hands from the previous import?

[WhiteRider]

ahhh that explains it but then why did it only recognize 193 duplicates and not all 553 hands from the previous import?

Because the files are different in the second import.
Are the tables you are watching SNGs, so that the tables come and go?
Once the table is closed the file will be removed by FT, so next time you import it won't be there. I think this is what is causing the confusion.