Page 1 of 1

Bulk importing. Is this normal?

PostPosted: Tue Oct 07, 2008 2:20 am
by revigres
http://img505.imageshack.us/my.php?imag ... 276lt0.jpg

After importing 10 mil hands which lasted for over 3 days, the housekeeping procedure closes to 2 days. Is this normal? Is the cache update the last step? Any tips to speed up the process?
Thanks

Re: Bulk importing. Is this normal?

PostPosted: Tue Oct 07, 2008 5:06 am
by WhiteRider
I haven't had any data on such a large database, but yes, the cache update will take a long time. When it finishes, would you mind telling us how long it took?

Re: Bulk importing. Is this normal?

PostPosted: Wed Oct 08, 2008 12:44 pm
by revigres
It's at 80 hours and still running. What's weird is that CPU usage doesn't go higher than 30% and averages 10%. Why is it taking so long and doesn't use any CPU? I think I'll give it another day and then shut it down.

Re: Bulk importing. Is this normal?

PostPosted: Wed Oct 08, 2008 1:35 pm
by WhiteRider
Check out this post by Kraada - it should allow you to check whether it's still doing anything.
It's really hard to say how long it will take - as I said, I haven't had experience with such large databases.

Re: Bulk importing. Is this normal?

PostPosted: Wed Oct 08, 2008 10:36 pm
by revigres
after 87 hours, I got the message Housekeeping canceled: There were errors during the housekeeping process:
Unable to execute query: CREATE TABLE "new_holdem_hand_player_detail" AS SELECT * FROM "holdem_hand_player_detail" ORDER BY id_player DESC, id_hand DESC;; Reason: Fatal Error (ERROR: could not extend relation 1663/17632/40967: No space left on device HINT: Check free disk space. )

While i still have 21.6 GB left. Any clues?

Re: Bulk importing. Is this normal?

PostPosted: Thu Oct 09, 2008 4:19 am
by WhiteRider
Postgres uses a lot of space to do these operations - pretty much the size of the database as I understand it. When the operation ends the space will be freed up, so presumably that's what's happened here - the space will be released even if the operation fails.

highfalutin