is this for real? (import speed)

PostgreSQL is the database server used to store information. Do you have a question or are you having problem with PostgreSQL? If so, post them here.

Moderator: Moderators

Re: is this for real? (import speed)

Postby Debith » Tue Dec 01, 2009 2:19 am

Germaniac2 wrote:An array of 5 standard HHDs is usually not equivalent to a well setup SATA II RAID array in speed, and under normal circumstances certainly not much better even with top shelf hardware. It mostly depends on your HDDs controller and possibly some other parts of your hardware. While the number of 300 Hands/sec kraada quotes seems a bit high, I run at an average of 240 Hands/sec. on file imports. I use just two 300 GB SATA II WD VelociRaptor HDDs configured as a striped RAID [0] controlled by an nVIDIA 780a chipset on my FoxConn Destroyer motherboard with an AMD Phenom II 940 Deneb (Quad Core) CPU running at 3.60 GHz, RAM is 8 GB. While you could improve upon that speed by using three or four HDDs in RAID0, the risk you're taking that a failure might occur also increases. [And I can tell you from personal experience over the past five years that Windows OSs hate RAIDs with a passion.] The best solution if you want more speed and have the budget for it is an SSD such as OCZ's Z-Drive, which is actually an array of SSDs with its own RAID controller on a PCIE card. The drawbacks: pricing starts at $1,375 for the 250 GB mainstream unit and goes up to $5,126 for the 1 TB enterprise unit; plus you'll be giving up at least one PCIE slot in your machine. The plus: sequential read speeds up to 770 (870) MB/sec., sequential write speeds up to 640 (780) MB/sec. Considering these numbers, I would estimate the performance of this drive for PokerTracker users to easily reach 400-450+ Hands/sec. providing you have a decent motherboard and CPU.

I never compared about speed of any RAID setup vs ZFS filesystem. ZFS is just simply much better system than RAID.

Still, throughput for the server is so good that it can easily put 100Mb LAN connection to its knees. Copying data to server eats around 50% of network bandwidth. And there's my problem. PT3 barely uses 2% of network bandwidth, when it is importing hands to server. And server barely uses 2% of CPU power to cluster the database. Clustering took 103 hours and was interrupted many times before that. My settings may be screwed, but not sure where to start look for (Need to post to postgres mailing lists). In other words, I'd like to see the resources to be used, before start improving the resources.

btw, I'm using single SSD in all my home computers and they alone can provide 200+ hands/s for import speed (just taking numbers from head, could do real benchmarking for bigger chunk of hands since 2000 hands were imported so quickly that it run 240h/s, which is fairly small number of hands).
Debith
 
Posts: 24
Joined: Tue Mar 04, 2008 2:31 pm

Re: is this for real? (import speed)

Postby Germaniac2 » Tue Dec 01, 2009 5:53 am

Hi Debith,

I was not aware of the ZFS filesystem setup and can't say that I know anything about it. However, when scanning through the Wikipedia article
on ZFS, the section on limitations reads: "ZFS has no defragmentation utility. Usage of COW with often changed files lead to high fragmentation."
If that is correct and your array is highly fragmented, that could explain both the low CPU usage and slow speed. I know that PT3 performs noticably
faster for me after running a defragmentation cycle through System Mechanic.

The full Wikipedia article on ZFS is here: http://en.wikipedia.org/wiki/ZFS
Germaniac2
 
Posts: 19
Joined: Fri Oct 16, 2009 2:30 pm

Re: is this for real? (import speed)

Postby Debith » Tue Dec 01, 2009 10:16 am

Germaniac2 wrote:Hi Debith,

I was not aware of the ZFS filesystem setup and can't say that I know anything about it. However, when scanning through the Wikipedia article
on ZFS, the section on limitations reads: "ZFS has no defragmentation utility. Usage of COW with often changed files lead to high fragmentation."
If that is correct and your array is highly fragmented, that could explain both the low CPU usage and slow speed. I know that PT3 performs noticably
faster for me after running a defragmentation cycle through System Mechanic.

The full Wikipedia article on ZFS is here: http://en.wikipedia.org/wiki/ZFS


That was quite interesting news. I don't believe fragmentation is that much of issue yet, because the setup is quite new and drives are basically empty still. Fragmentation can be issue for SSD's too.
Debith
 
Posts: 24
Joined: Tue Mar 04, 2008 2:31 pm

Previous

Return to PostgreSQL [Read Only]

Who is online

Users browsing this forum: No registered users and 75 guests

cron