Weeding Notes

[whiskyjohn]

Rant on.

Ok so I have been customizing my notes and I have come to the following conclusions:

1) Processing Hands is *SLOW*
2) Purging Notes is *SLOW*

If I let my computer spend days and days processing hands only to discover I want to change or even
delete one of my notes, my only option is to wipe EVERYTHING and start again. Seriously, that's the
worst solution possible!

There's no way to delete notes for an individual player. No way to delete a specific note for an
individual player. No way to re-read hands that were already processed.

This would be like Microsoft telling me to reinstall Windows every time they came out with an update
or a bug fix. Only slower because I can reinstall Windows in a fraction of the time it takes to re-read
all the hands with new note definitions!

Com'on guys!

Rant off.

[WhiteRider]

Notes aren't taken for players or for rules - they are taken by processing a hand. i.e. A hand is processed and any notes which match for that hand are applied to the appropriate players.
We appreciate that clearing and recreating notes is inconvenient this way, and it's possible the development team will be able to rework it at some point in the future but it isn't a simple change so I don't expect it to change any time soon.

If you want to clear all notes to start again then 'purge all automatic notes' should happen almost instantly since it's a single database operation, but clearing by date will take a bit longer.

Notes are recreated in date order with the newest hands processed first so that you always have the most relevant notes first.

[whiskyjohn]

Notes aren't taken for players or for rules - they are taken by processing a hand. i.e. A hand is processed and any notes which match for that hand are applied to the appropriate players.
We appreciate that clearing and recreating notes is inconvenient this way, and it's possible the development team will be able to rework it at some point in the future but it isn't a simple change so I don't expect it to change any time soon.

Yeah I get it, but that's exactly the problem with the design! We are taking notes on *players* not on *hands*. Who cares that PlayerX in hand #123456789
raised UTG? I may never see that note on PlayerX because maybe I played him a couple times and never see him again. And the fact that he raised one
hand UTG is of no value to me. I care that VillainX raised UTG. I also care about *when* VillainX raised UTG, but I'll leave that discussion for another time.

But let's say I've processed a bunch of hands and I've noticed that VillainX makes some curious plays. I want to write a custom rule to catch some of these
plays that i didn't notice before. What must I do? Well, first I must test the rule to make sure it is written correctly. But now I want to put this rule to work
and get the card ranges that match the rule. I must delete the whole damn database for ALL players (including that PlayerX I don't care about) and reprocess
ALL rules just to check out if my custom rules gives a result which is meaningful. And what if it doesn't and I want to delete those results? Oh shit, I need
to delete EVERYTHING and reprocess again!

I am assuming that must people use NoteTracker to get cardranges on player habits. As such, doesn't it make sense to test a new rule on a specific player,
go see what cardranges that rule returns, and then decide if that rule should be extended to all players, for all dates? Ideally, if I don't like the results I could
delete them from a specific player without having to reprocess all that players rules. At worst, I would reprocess all that player's hands for all rules. But
reprocessing the entire database for all players and all rules would be the last last resort!

If you want to clear all notes to start again then 'purge all automatic notes' should happen almost instantly since it's a single database operation, but clearing by date will take a bit longer.

Instantly? I have a very fast computer and an SSD and still nothing happens instantly with Pokertracker!

John

[kraada]

How long does the purge take for you? It really should take a fraction of a second. Reprocessing is the long part and I agree we have ways to improve here.

[whiskyjohn]

How long does the purge take for you? It really should take a fraction of a second. Reprocessing is the long part and I agree we have ways to improve here.

Well the last few times I purged all notes it took at least a couple minutes. Now the computer has taken a couple days
to process over 1mil hands so I'd rather not purge all that again just to test the time.

Perhaps I need to recluster the database again, but that really shouldn't affect hand notes right?

John

[kraada]

It definitely shouldn't take anywhere near that long. You were just using the Purge All Notes button, correct?

[whiskyjohn]

It definitely shouldn't take anywhere near that long. You were just using the Purge All Notes button, correct?

Yes I am sure. I have about 40 different custom notes.

Also, for other reasons, I am running PostgreSQL 9.1.8 which I know is not officially supported by PokerTracker.
Is there any reason to think this version would cause imports and purges to run so slowly? Data for custom HUD
stats seems to import in a reasonable amount of time.

John

[kraada]

I ran some tests as a result of this post and am seeing some delays; you might find the date version to be faster. We made some changes in the most recent couple of point versions of PT4 that should help things work better on 9.1+, so if you're not running our latest release, please do upgrade. The problems in the past could definitely cause hangs in import.

[whiskyjohn]

I ran some tests as a result of this post and am seeing some delays; you might find the date version to be faster. We made some changes in the most recent couple of point versions of PT4 that should help things work better on 9.1+, so if you're not running our latest release, please do upgrade. The problems in the past could definitely cause hangs in import.

Ok at least it's not just me. I'm running PTv4.10.8 at the moment.

I wasn't really ranting about the purge speed though. That's the *least* of my concerns. I could even live
with slow imports, knowing that one day faster processors and faster drives would help. The main concern
is with the design itself. I can think of several features that would be nice to add, but before any of that
is done, there should really be a way to reprocess notes without rebuilding the whole damn database!
I think of notes in terms of specific players, not in terms of hands played since N days. And I think
most others think that way too.

While I'm on the subject, maybe you can answer how the cardrange values are internally counted.
For example, if I have the following note for Early Position Raise First In:

EP RFI{55+,A7s+,KTs+,QTs+,JTs, T9s, 98s, 87s, 76s, 65s, 54s, ATo+,KJo+, 72o} (300)

I see I have a sample size of 300. Now I added 72o to the range myself but assume it was really
there for this player. Does PT know how many 72o hands are in this range, or does it just add 72o
to the range if it wasn't there before, and increment the sample size? I picked that hand because
if the answer is 1, maybe the player was just being cute or on tilt or whatever and that hand
should be dropped.

I also added T9s-54s. Let's say there are multiple instances of these hands in the
range but they were all played way back in 2012 or something like that. These should
also be dropped.

So if the individual hand frequencies are not lost, it would be very nice to have a
counter for each possible hand. The default would obviously be to include all hands
into the range. But 72o and other goofy hands could be set to a nonzero number
to get rid of some noise through the signal. If the player really does play 72o with
some regularity, then it will show up in the range.

Timestamping each hand in the range would be far more detailed, however, but
it would allow us filter hand ranges that haven't been played in X months/years/etc.

And finally, the ability to get a Hand Range Vizualizer for any cardrange note taken
would be even more useful. Here we could see the exact breakdown, the number of
each hands played, and the report of individual hands sorted by date(and other ways)
so we could go back and replay each hand in the range that matched the note in the
cardrange. Now we could clearly see that 72o was only played once, or small suited
connectors haven't been played since 2012, etc, and we could tune the range displayed
in the notes more finely.

Just a few more noteworthy thoughts...

John

[kraada]

You can get rid of older data by using the purge by date options (which should run faster, according to my tests) and you can also set NT to only take notes on hands that occurred after a certain point. So on the first of the month you could purge data older than, say, 3 months and update the note taking time and only see the most recent 3 months of data.

If any hand has been proven to be in his range, it is listed, and the count is the number of shown down hands we've seen.