Optimizing Mysql Database

[yuvrajkc]

We are currently running 4-vicibox servers in cluster:
- ViciBox Redux v.5.0.2-130807
VERSION: 2.8-410a
BUILD: 130824-2319
© 2013 ViciDial Group

Server1: Database only
Server2: Asterisk + Web + Agent Login
Server3: Asterisk only
Server4: Asterisk + Web + Agent login

During peak time the mysql CPU Utilization reaches . 300%
http://prntscr.com/3fjjb7

Because of the high CPU Usage we are having issues like, same call being dialed multiple times, agents not being able to click on Leave 3-way button.

Current Database config:

Code: Select all

cat /etc/my.cnf
# Basic ViciBox my.cnf for a quad-core 8-GB RAM or so
# Use the dedicate my-big.cnf for 16+GB RAM and 8+ Cores
#

# The following options will be passed to all MySQL clients
[client]
#password   = your_password
port      = 3306
socket      = /var/run/mysql/mysql.sock

# Here follows entries for some specific programs

# The MySQL server
[mysqld]
port      = 3306
socket      = /var/run/mysql/mysql.sock
datadir   = /var/lib/mysql
skip-external-locking
key_buffer_size = 4096M
max_allowed_packet = 50M
table_open_cache = 18192
sort_buffer_size = 4M
net_buffer_length = 8K
read_buffer_size = 4M
read_rnd_buffer_size = 16M
myisam_sort_buffer_size = 128M
thread_cache_size = 50
query_cache_size = 0
thread_concurrency=32
skip-name-resolve
connect_timeout=60
long_query_time=3
#log_slow_queries
log-slow-queries=/var/log/mysql/log-slow-queries.log
max_connections=2000
open_files_limit=24576
max_heap_table_size=64M
expire_logs_days=1
default-storage-engine=MyISAM
table_definition_cache=8192
#table_cache=8192
myisam_recover
myisam_repair_threads=1

# This will disable networking
#skip-networking

# Replication Master Server (default)
# binary logging is required for replication
log-bin=mysql-bin

# binary logging format - mixed recommended
binlog_format=mixed

# required unique id between 1 and 2^32 - 1
# defaults to 1 if master-host is not set
# but will not function as a master if omitted
server-id   = 1

# Replication Slave (comment out master section to use this)
#
# To configure this host as a replication slave, you can choose between
# two methods :
#
# 1) Use the CHANGE MASTER TO command (fully described in our manual) -
#    the syntax is:
#
#    CHANGE MASTER TO MASTER_HOST=<host>, MASTER_PORT=<port>,
#    MASTER_USER=<user>, MASTER_PASSWORD=<password> ;
#
#    where you replace <host>, <user>, <password> by quoted strings and
#    <port> by the master's port number (3306 by default).
#
#    Example:
#
#    CHANGE MASTER TO MASTER_HOST='125.564.12.1', MASTER_PORT=3306,
#    MASTER_USER='joe', MASTER_PASSWORD='secret';
#
# OR
#
# 2) Set the variables below. However, in case you choose this method, then
#    start replication for the first time (even unsuccessfully, for example
#    if you mistyped the password in master-password and the slave fails to
#    connect), the slave will create a master.info file, and any later
#    change in this file to the variables' values below will be ignored and
#    overridden by the content of the master.info file, unless you shutdown
#    the slave server, delete master.info and restart the slaver server.
#    For that reason, you may want to leave the lines below untouched
#    (commented) and instead use CHANGE MASTER TO (see above)
#
# required unique id between 2 and 2^32 - 1
# (and different from the master)
# defaults to 2 if master-host is set
# but will not function as a slave if omitted
#server-id       = 2
#
# The replication master for this slave - required
#master-host     =   <hostname>
#
# The username the slave will use for authentication when connecting
# to the master - required
#master-user     =   <username>
#
# The password the slave will authenticate with when connecting to
# the master - required
#master-password =   <password>
#
# The port the master is listening on.
# optional - defaults to 3306
#master-port     =  <port>
#
# binary logging - not required for slaves, but recommended
#log-bin=mysql-bin

# Uncomment the following if you are using InnoDB tables
#innodb_data_home_dir = /var/lib/mysql
#innodb_data_file_path = ibdata1:10M:autoextend
#innodb_log_group_home_dir = /var/lib/mysql
# You can set .._buffer_pool_size up to 50 - 80 %
# of RAM but beware of setting memory usage too high
#innodb_buffer_pool_size = 16M
#innodb_additional_mem_pool_size = 2M
# Set .._log_file_size to 25 % of buffer pool size
#innodb_log_file_size = 5M
#innodb_log_buffer_size = 8M
#innodb_flush_log_at_trx_commit = 1
#innodb_lock_wait_timeout = 50

# The safe_mysqld script
[safe_mysqld]
log-error   = /var/log/mysql/mysqld.log
socket      = /var/run/mysql/mysql.sock

[mysqldump]
socket      = /var/run/mysql/mysql.sock
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates

[myisamchk]
key_buffer_size = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
interactive-timeout

[mysqld_multi]
mysqld     = /usr/bin/mysqld_safe
mysqladmin = /usr/bin/mysqladmin
log        = /var/log/mysqld_multi.log
# user       = multi_admin
# password   = secret

# If you want to use mysqld_multi uncomment 1 or more mysqld sections
# below or add your own ones.

# WARNING
# --------
# If you uncomment mysqld1 than make absolutely sure, that database mysql,
# configured above, is not started.  This may result in corrupted data!
# [mysqld1]
# port       = 3306
# datadir    = /var/lib/mysql
# pid-file   = /var/lib/mysql/mysqld.pid
# socket     = /var/lib/mysql/mysql.sock
# user       = mysql

# [mysqld2]
# port       = 3307
# datadir    = /var/lib/mysql-databases/mysqld2
# pid-file   = /var/lib/mysql-databases/mysqld2/mysql.pid
# socket     = /var/lib/mysql-databases/mysqld2/mysql.sock
# user       = mysql

# [mysqld3]
# port       = 3308
# datadir    = /var/lib/mysql-databases/mysqld3
# pid-file   = /var/lib/mysql-databases/mysqld3/mysql.pid
# socket     = /var/lib/mysql-databases/mysqld3/mysql.sock
# user       = mysql

# [mysqld6]
# port       = 3309
# datadir    = /var/lib/mysql-databases/mysqld6
# pid-file   = /var/lib/mysql-databases/mysqld6/mysql.pid
# socket     = /var/lib/mysql-databases/mysqld6/mysql.sock
# user       = mysql


Database server Hardware:

Code: Select all

cat /proc/cpuinfo
processor   : 0
vendor_id   : GenuineIntel
cpu family   : 6
model      : 26
model name   : Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping   : 5
microcode   : 0x19
cpu MHz      : 2666.612
cache size   : 8192 KB
physical id   : 0
siblings   : 8
core id      : 0
cpu cores   : 4
apicid      : 0
initial apicid   : 0
fpu      : yes
fpu_exception   : yes
cpuid level   : 11
wp      : yes
flags      : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips   : 5333.22
clflush size   : 64
cache_alignment   : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 1
vendor_id   : GenuineIntel
cpu family   : 6
model      : 26
model name   : Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping   : 5
microcode   : 0x19
cpu MHz      : 2666.612
cache size   : 8192 KB
physical id   : 1
siblings   : 8
core id      : 0
cpu cores   : 4
apicid      : 16
initial apicid   : 16
fpu      : yes
fpu_exception   : yes
cpuid level   : 11
wp      : yes
flags      : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips   : 5333.34
clflush size   : 64
cache_alignment   : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 2
vendor_id   : GenuineIntel
cpu family   : 6
model      : 26
model name   : Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping   : 5
microcode   : 0x19
cpu MHz      : 2666.612
cache size   : 8192 KB
physical id   : 0
siblings   : 8
core id      : 2
cpu cores   : 4
apicid      : 4
initial apicid   : 4
fpu      : yes
fpu_exception   : yes
cpuid level   : 11
wp      : yes
flags      : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips   : 5333.22
clflush size   : 64
cache_alignment   : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 3
vendor_id   : GenuineIntel
cpu family   : 6
model      : 26
model name   : Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping   : 5
microcode   : 0x19
cpu MHz      : 2666.612
cache size   : 8192 KB
physical id   : 1
siblings   : 8
core id      : 2
cpu cores   : 4
apicid      : 20
initial apicid   : 20
fpu      : yes
fpu_exception   : yes
cpuid level   : 11
wp      : yes
flags      : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips   : 5333.34
clflush size   : 64
cache_alignment   : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 4
vendor_id   : GenuineIntel
cpu family   : 6
model      : 26
model name   : Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping   : 5
microcode   : 0x19
cpu MHz      : 2666.612
cache size   : 8192 KB
physical id   : 0
siblings   : 8
core id      : 1
cpu cores   : 4
apicid      : 2
initial apicid   : 2
fpu      : yes
fpu_exception   : yes
cpuid level   : 11
wp      : yes
flags      : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips   : 5333.22
clflush size   : 64
cache_alignment   : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 5
vendor_id   : GenuineIntel
cpu family   : 6
model      : 26
model name   : Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping   : 5
microcode   : 0x19
cpu MHz      : 2666.612
cache size   : 8192 KB
physical id   : 1
siblings   : 8
core id      : 1
cpu cores   : 4
apicid      : 18
initial apicid   : 18
fpu      : yes
fpu_exception   : yes
cpuid level   : 11
wp      : yes
flags      : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips   : 5333.34
clflush size   : 64
cache_alignment   : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 6
vendor_id   : GenuineIntel
cpu family   : 6
model      : 26
model name   : Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping   : 5
microcode   : 0x19
cpu MHz      : 2666.612
cache size   : 8192 KB
physical id   : 0
siblings   : 8
core id      : 3
cpu cores   : 4
apicid      : 6
initial apicid   : 6
fpu      : yes
fpu_exception   : yes
cpuid level   : 11
wp      : yes
flags      : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips   : 5333.22
clflush size   : 64
cache_alignment   : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 7
vendor_id   : GenuineIntel
cpu family   : 6
model      : 26
model name   : Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping   : 5
microcode   : 0x19
cpu MHz      : 2666.612
cache size   : 8192 KB
physical id   : 1
siblings   : 8
core id      : 3
cpu cores   : 4
apicid      : 22
initial apicid   : 22
fpu      : yes
fpu_exception   : yes
cpuid level   : 11
wp      : yes
flags      : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips   : 5333.34
clflush size   : 64
cache_alignment   : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 8
vendor_id   : GenuineIntel
cpu family   : 6
model      : 26
model name   : Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping   : 5
microcode   : 0x19
cpu MHz      : 2666.612
cache size   : 8192 KB
physical id   : 0
siblings   : 8
core id      : 0
cpu cores   : 4
apicid      : 1
initial apicid   : 1
fpu      : yes
fpu_exception   : yes
cpuid level   : 11
wp      : yes
flags      : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips   : 5333.22
clflush size   : 64
cache_alignment   : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 9
vendor_id   : GenuineIntel
cpu family   : 6
model      : 26
model name   : Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping   : 5
microcode   : 0x19
cpu MHz      : 2666.612
cache size   : 8192 KB
physical id   : 1
siblings   : 8
core id      : 0
cpu cores   : 4
apicid      : 17
initial apicid   : 17
fpu      : yes
fpu_exception   : yes
cpuid level   : 11
wp      : yes
flags      : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips   : 5333.34
clflush size   : 64
cache_alignment   : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 10
vendor_id   : GenuineIntel
cpu family   : 6
model      : 26
model name   : Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping   : 5
microcode   : 0x19
cpu MHz      : 2666.612
cache size   : 8192 KB
physical id   : 0
siblings   : 8
core id      : 2
cpu cores   : 4
apicid      : 5
initial apicid   : 5
fpu      : yes
fpu_exception   : yes
cpuid level   : 11
wp      : yes
flags      : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips   : 5333.22
clflush size   : 64
cache_alignment   : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 11
vendor_id   : GenuineIntel
cpu family   : 6
model      : 26
model name   : Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping   : 5
microcode   : 0x19
cpu MHz      : 2666.612
cache size   : 8192 KB
physical id   : 1
siblings   : 8
core id      : 2
cpu cores   : 4
apicid      : 21
initial apicid   : 21
fpu      : yes
fpu_exception   : yes
cpuid level   : 11
wp      : yes
flags      : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips   : 5333.34
clflush size   : 64
cache_alignment   : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 12
vendor_id   : GenuineIntel
cpu family   : 6
model      : 26
model name   : Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping   : 5
microcode   : 0x19
cpu MHz      : 2666.612
cache size   : 8192 KB
physical id   : 0
siblings   : 8
core id      : 1
cpu cores   : 4
apicid      : 3
initial apicid   : 3
fpu      : yes
fpu_exception   : yes
cpuid level   : 11
wp      : yes
flags      : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips   : 5333.22
clflush size   : 64
cache_alignment   : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 13
vendor_id   : GenuineIntel
cpu family   : 6
model      : 26
model name   : Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping   : 5
microcode   : 0x19
cpu MHz      : 2666.612
cache size   : 8192 KB
physical id   : 1
siblings   : 8
core id      : 1
cpu cores   : 4
apicid      : 19
initial apicid   : 19
fpu      : yes
fpu_exception   : yes
cpuid level   : 11
wp      : yes
flags      : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips   : 5333.34
clflush size   : 64
cache_alignment   : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 14
vendor_id   : GenuineIntel
cpu family   : 6
model      : 26
model name   : Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping   : 5
microcode   : 0x19
cpu MHz      : 2666.612
cache size   : 8192 KB
physical id   : 0
siblings   : 8
core id      : 3
cpu cores   : 4
apicid      : 7
initial apicid   : 7
fpu      : yes
fpu_exception   : yes
cpuid level   : 11
wp      : yes
flags      : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips   : 5333.22
clflush size   : 64
cache_alignment   : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 15
vendor_id   : GenuineIntel
cpu family   : 6
model      : 26
model name   : Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping   : 5
microcode   : 0x19
cpu MHz      : 2666.612
cache size   : 8192 KB
physical id   : 1
siblings   : 8
core id      : 3
cpu cores   : 4
apicid      : 23
initial apicid   : 23
fpu      : yes
fpu_exception   : yes
cpuid level   : 11
wp      : yes
flags      : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips   : 5333.34
clflush size   : 64
cache_alignment   : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:


Code: Select all

cat /proc/meminfo
MemTotal:       37137984 kB
MemFree:         1316464 kB
Buffers:          100208 kB
Cached:         31558640 kB
SwapCached:            0 kB
Active:         25516460 kB
Inactive:        9003468 kB
Active(anon):    2876816 kB
Inactive(anon):     8676 kB
Active(file):   22639644 kB
Inactive(file):  8994792 kB
Unevictable:        3120 kB
Mlocked:               0 kB
SwapTotal:       4200444 kB
SwapFree:        4200444 kB
Dirty:              3368 kB
Writeback:             0 kB
AnonPages:       2864360 kB
Mapped:            32148 kB
Shmem:             21128 kB
Slab:             995948 kB
SReclaimable:     943784 kB
SUnreclaim:        52164 kB
KernelStack:        2568 kB
PageTables:        13396 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    22769436 kB
Committed_AS:    4996872 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      180472 kB
VmallocChunk:   34359556775 kB
HardwareCorrupted:     0 kB
AnonHugePages:   2789376 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       69808 kB
DirectMap2M:    37668864 kB


Can someone please assist in optimizing the database server to be able to take more load?

Thanks.

[williamconley]

Before we go into "let's improve the system!" it would be good to know if you are simply hammering that server. It's not "unlimited", lol.

How many calls are "live/dialing/etc" in the system when this overload occurs?

Are you seeing any errors in any logs?

Is it a virtual server?

15kSAS drives or SSD drives with 6G/sec throughput?

Did you install the same Vicibox version on this server as the other servers in the cluster, or did you install some other OS/MySQL?

When you installed the DB server, did you answer the questions regarding "roles" and tell it to have a high-capacity Database? (I forget the actual question, but there is a superDB question for enterprise level DB settings)

Often helpful if you capture the Q/A section during the installation and store it in a text file (for these moments). Just before you hit "Y" these answers are correct (to initiate the actual install).

Next: There HAVE actually been a few threads on this topic that actually included the my.cnf settings ... you should be able to find them.

[yuvrajkc]

Hi William,

Thanks for quick reply.

How many calls are "live/dialing/etc" in the system when this overload occurs?
*** http://prntscr.com/3gsse8

Are you seeing any errors in any logs?
*** Can you please point which log should I be reviewing..

Is it a virtual server?
*** No, all servers are dedicated.

15kSAS drives or SSD drives with 6G/sec throughput?
*** I know the drive is not SSD.
not sure on its speed.

Did you install the same Vicibox version on this server as the other servers in the cluster, or did you install some other OS/MySQL?
*** Yes, all servers are running the exact same vicibox version(installed from same CD).

When you installed the DB server, did you answer the questions regarding "roles" and tell it to have a high-capacity Database? (I forget the actual question, but there is a superDB question for enterprise level DB settings)
*** I don't quite remember. Can I look into astguiclient.conf to check what was my answer ?

[williamconley]

1) Whoa: 658 active calls? Trunk Short/Fill 1347/440? 43 agents dial level 2.8 ... ? I'm seeing some "conflicting" numbers. You've obviously got a campaign set MUCH freakin higher than 2.8 to get the system to request that many channels.

2) The 300% is actually "3.0" meaning 3 cores busy out of ... an undisclosed number. Which is not a problem at all if you have 8 cores. If you have 4 cores, it is cause for concern (and you should get a machine with 8 cores!) but not actually "overloaded" (technically over the border, but not enough to freak out ... just enough to plan to improve the server ... soon).

3) Logs are in /var/log plus /var/log/asterisk plus /var/log/astguiclient (have a look! )

4) I bet they aren't virtual, no freakin way they would hold 10% of this load, LOL

5) You'll want to check your drives and find out if they are SATA or SAS and 3G/sec vs 6G/sec and the rotation speed also. Should be 15K RPM SAS drives with 6G (if you're gonna push it like this!)

6) High capacity DB will result in a slightly different "my.cnf", but I don't think it makes a difference in the astguilcient.conf file. I will note that you'll want to NOT run any screens (keepalives) on the DB server. Move them all to another server. Also do not use the DB server for web access or register any phones to it or have any trunks or asterisk functionality. It would be a reasonable idea to actually shut down apache and asterisk (but not really necessary as long as you don't use them during work hours).

Summary: Your system is not behaving poorly. It's actually doing quite well for the hammering its taking.

[yuvrajkc]

1. I guess while selecting the Campaign ID in "Choose report display option", perhaps I did all campaigns, therefore the dial level is displayed wrong.
http://prntscr.com/3gyzgt This is how the campaign is configured to dial.

2. OK, The DB server load average is < 5 when the mysql process is running at > 300%.

3. I know the location of log, I was looking for the appropriate log filenames I should be looking for, and what should I be looking for in those log files. As you saw in the real time report, the dialer is dialing too many calls and its difficult to go through the huge log file without the clue what I'm looking for.

4.

5. 10k SAS, 6 of them on a raid 5 (HP's advance raid configuration)

6. DB server is running only database and 2 screens:

Code: Select all

screen -x
There are several suitable screens on:
   4458.ASTVDadapt   (Detached)
   2134.ASTVDautoFILL   (Detached)

Our agents are using a ubuntu system with firefox. Could that cause any issues ?

[williamconley]

change the adapt intensity to LESS intense (kinda like wasting gas while on the road ... don't make change every 15 nano seconds ... it's like "hit the gas, no hit the brake, no the gas, no the brake ..."). Or perhaps just to even. Observe that it should smooth out the dialer.

10 calls waiting for agents? 30:1 dialing? Avaialble only NO? Why are you even using adapt when you really just want "dial like crazy"? You're wasting valuable CPU power making calculations (and a lot of them, and very often) that you have no intention of using.

You should have 15K SAS and RAID 10 NOT RAID 5.

You should move those two scripts to another server.

2. OK, The DB server load average is < 5 when the mysql process is running at > 300%.

Um ... load average 5 = 500%. 300% is load average 3.0. So why are you treating these like they are different numbers? Where are you getting them? 100% is defined as "one cpu". Load average is the number of "busy cpus .. on avarage". 1.0 = 100% ... etc.

If you have 4 cores, your system is in overload. If you have 8 cores, you're at capacity. But some of the above changes (and perhaps some others) could reduce waste.

It all hinges, of course, on whether you are experiencing problems. If not, don't go nuts trying to fix the NONbroken system. LOL