is this for real? (import speed)

[kraada]

Sorry, I wrote that quickly and forgot about your hard drive array - that statement is true for most people

I've definitely seen a good number of people lately reporting extremely slow CLUSTER times. But I haven't yet been able to pin down a cause . . .

[Debith]

Sorry, I wrote that quickly and forgot about your hard drive array - that statement is true for most people

I've definitely seen a good number of people lately reporting extremely slow CLUSTER times. But I haven't yet been able to pin down a cause . . .

So, were I correct that server does the job?

Is there any information I could provide to help to solve this?

[kraada]

Yes, the server does the job. Unfortunately there isn't much more that PT3 does other than send the CLUSTER command to the database. You could consider asking the PostgreSQL people why CLUSTER is running so slowly; I'm afraid at the moment I don't have any more ideas.

[Debith]

The clustering job finished yesterday after 105h of working.

During the process, I noticed one thing though. Physical memory was full, which probably explains why server did things so slowly. Need to do some shopping... before anything else.

Thanks for help!

[WhiteRider]

How much RAM do you have?

[Debith]

How much RAM do you have?

Just 2GB

[WhiteRider]

I wouldn't have thought that would cause a bottleneck. Are you running lots of other applications?

[Debith]

My first intention was just to make it file server, which is why I didn't but too much memory. There are also SSH, Samba and FTP servers. Now that no clients are connected to postgres, I got 1GB free memory.

I'm not sure, is the memory the bottleneck, but it's a hunch as the memory was so low during the operation.

[kraada]

I don't think memory in general should be the bottleneck but if you increased the values in the postgresql.conf for memory usage too far it could cause slowdowns.

[Germaniac2]

The real bottleneck is the access time from the hard drive; people with RAID arrays have been known to get speeds in excess of 300h/sec.

You lost me.

Server machine consists array 5 hard-drives, which is equivalent to RAID (but much better). Is it the server who does the clustering work, or the client? In my case, computer 2 has issued the clustering job to server. Of course it makes sense, that server would do all the work, but postgres server is using mere 1% of cpu power on server computer (which is dual-core). If this is the case, that server should be my worker, how can I tell it to start using those resources it has?

Thanks!

An array of 5 standard HHDs is usually not equivalent to a well setup SATA II RAID array in speed, and under normal circumstances certainly not much better even with top shelf hardware. It mostly depends on your HDDs controller and possibly some other parts of your hardware. While the number of 300 Hands/sec kraada quotes seems a bit high, I run at an average of 240 Hands/sec. on file imports. I use just two 300 GB SATA II WD VelociRaptor HDDs configured as a striped RAID [0] controlled by an nVIDIA 780a chipset on my FoxConn Destroyer motherboard with an AMD Phenom II 940 Deneb (Quad Core) CPU running at 3.60 GHz, RAM is 8 GB. While you could improve upon that speed by using three or four HDDs in RAID0, the risk you're taking that a failure might occur also increases. [And I can tell you from personal experience over the past five years that Windows OSs hate RAIDs with a passion.] The best solution if you want more speed and have the budget for it is an SSD such as OCZ's Z-Drive, which is actually an array of SSDs with its own RAID controller on a PCIE card. The drawbacks: pricing starts at $1,375 for the 250 GB mainstream unit and goes up to $5,126 for the 1 TB enterprise unit; plus you'll be giving up at least one PCIE slot in your machine. The plus: sequential read speeds up to 770 (870) MB/sec., sequential write speeds up to 640 (780) MB/sec. Considering these numbers, I would estimate the performance of this drive for PokerTracker users to easily reach 400-450+ Hands/sec. providing you have a decent motherboard and CPU.

You can find the OCZ Z-Drive at: http://www.newegg.com/Product/Product.aspx?Item=N82E16820227497

Hope this helps

PS: If anyone here has tried the Z-Drive, I'd love to get your input & results.