RE: Speeding up PostgreSQL data loading
You are viewing a single comment's thread:
We generally don't need much shared_buffers to begin with because ZFS is already doing the caching. I probably will have to include my postgresql.conf in the repository.
I remember experimenting maintenance_work_mem and I do already hit the max disk utilization pretty much on gen4 NVMe raid 0.
I will certainly take a look at your video and try your suggestions. Thank you.
0
0
0.000
I saw how you're using ZFS for compression and therefore prioritizing ARC, that all seemed quite sensible. That Open Street Map loader application will gobble up to 90GB of RAM for its cache, and on 64GB servers I was surprised how well PostgreSQL continued to work even with
shared_buffersand the working memory settings all dropped to 1GB. I hope you're able to find a useful trick or too in the talk, things like turning WAL off can be helpful when you're cycling on local development and just want the load time down. I was doing NVMe RAID0 for a while, I have a 4TB Firecuda 530 I'm planning to use for testing your code out. Keep up the good work and we're really rooting for you here. Migrating apps from SQL Server to OSS has been a good sized chunk of my life.