I'm still new to Linux and have been working with this highly inefficient environment we have setup. Currently, the way it is setup is 4x web servers with 15GB of RAM behind a load balancer, 1 web server (administration + some sites) and a 30GB DB server. It is a Drupal environment hosting 30k unique domains. The problem with this platform is none of the sites have a large amount of reoccurring traffic. I should note that this is on Rackspace. Some of the servers are on 1st generation and I know we would see a great performance boost in migrating to new architecture, but we don't always have enough bandwidth for that.
Varnish sits in the very front and we get about a 15% cache hit ratio at the very best. I recently had a developer add a bot blacklist to varnish because I thought this was pegging the server, but it hasn't helped recently. If varnish is a miss, it will then retrieve the content from Apache which is pegging most servers at around 100% CPU. I've calculated that each process needs up to about 140-150mb per process and this is what we configured the prefork_module:
<IfModule mpm_prefork_module> StartServers 36 MinSpareServers 8 MaxSpareServers 16 MaxClients 110 MaxRequestsPerChild 2000 </IfModule> The next part, we get to is the MySQL DB. I ran the mysqltuner.pl and had the recommendation that I could increase the table and query cache limit.
-------- General Statistics -------------------------------------------------- [--] Skipped version check for MySQLTuner script [OK] Currently running supported MySQL version 5.1.72-ius-log [OK] Operating on 64-bit architecture -------- Storage Engine Statistics ------------------------------------------- [--] Status: -Archive -BDB -Federated +InnoDB -ISAM -NDBCluster [--] Data in MyISAM tables: 7G (Tables: 88) [--] Data in InnoDB tables: 18G (Tables: 556) [!!] Total fragmented tables: 303 -------- Security Recommendations ------------------------------------------- [OK] All database users have passwords assigned -------- Performance Metrics ------------------------------------------------- [--] Up for: 6d 0h 10m 10s (1B q [1K qps], 7M conn, TX: 4053B, RX: 197B) [--] Reads / Writes: 88% / 12% [--] Total buffers: 13.6G global + 25.2M per thread (1000 max threads) [!!] Maximum possible memory usage: 38.3G (127% of installed RAM) [OK] Slow queries: 0% (1M/1B) [OK] Highest usage of available connections: 69% (698/1000) [OK] Key buffer size / total MyISAM indexes: 1.5G/2.2G [OK] Key buffer hit rate: 100.0% (23M cached / 1K reads) [OK] Query cache efficiency: 57.8% (531M cached / 920M selects) [!!] Query cache prunes per day: 53918921 [OK] Sorts requiring temporary tables: 0% (2K temp sorts / 72M sorts) [!!] Joins performed without indexes: 30472 [OK] Temporary tables created on disk: 0% (371 on disk / 6M total) [OK] Thread cache hit rate: 94% (401K created / 7M connections) [!!] Table cache hit rate: 3% (2K open / 54K opened) [OK] Open file limit used: 0% (3/20K) [OK] Table locks acquired immediately: 100% (519M immediate / 519M locks) [!!] InnoDB data size / buffer pool: 18.6G/12.0G -------- Recommendations ----------------------------------------------------- General recommendations: Run OPTIMIZE TABLE to defragment tables for better performance Reduce your overall MySQL memory footprint for system stability Adjust your join queries to always utilize indexes Increase table_cache gradually to avoid file descriptor limits Variables to adjust: *** MySQL's maximum memory usage is dangerously high *** *** Add RAM before increasing MySQL buffer variables *** query_cache_size (> 12M) join_buffer_size (> 5.0M, or always use indexes with joins) table_cache (> 2048) Performance on the DB "looks fine". CPU hovers around 30%, Memory is at 30% and I'm generally seeing the disk at a consistent 500 IOPs.
Any troubleshooting recommendations to truly see where the bottleneck is? Again, most of the traffic is just bots crawling and I would think those connections should drop quickly.
[link][16 comments]