Slow MySQL 5.5.22 database engine

Slooooooow UPDATE query with MySQL 5.5

I am using a local MySQL server a lot to handle, prepare and restructure big research tables. Ubuntu Precise uses MySQL server 5.5 while the previous distros used 5.1. I thought that might be good until I tried to import a table with some dozen variables and some thousand rows with and UPDATE statement which took some seconds (10 min!!).

nick rulez on forum.mysql.com quantified this fact and revealed that the default database engine changed from “MyISAM” to “InnoDB” and that indeed InnoDB is considerably slower in this regard.

So I want MyISAM back.

To list the available and default engines:

show engines
which produces

mysql> show engines;
+--------------------+---------+----------------------------------------------------------------+--------------+------+------------+
| Engine | Support | Comment | Transactions | XA | Savepoints |
+--------------------+---------+----------------------------------------------------------------+--------------+------+------------+
| MyISAM | YES | MyISAM storage engine | NO | NO | NO |
| CSV | YES | CSV storage engine | NO | NO | NO |
| MRG_MYISAM | YES | Collection of identical MyISAM tables | NO | NO | NO |
| BLACKHOLE | YES | /dev/null storage engine (anything you write to it disappears) | NO | NO | NO |
| MEMORY | YES | Hash based, stored in memory, useful for temporary tables | NO | NO | NO |
| PERFORMANCE_SCHEMA | YES | Performance Schema | NO | NO | NO |
| ARCHIVE | YES | Archive storage engine | NO | NO | NO |
| InnoDB | DEFAULT | Supports transactions, row-level locking, and foreign keys | YES | YES | YES |
| FEDERATED | NO | Federated MySQL storage engine | NULL | NULL | NULL |
+--------------------+---------+----------------------------------------------------------------+--------------+------+------------+
9 rows in set (0.00 sec)


Setting the database engine

When you create a new table, you can specify which storage engine to use by adding an ENGINE table option to the CREATE TABLE statement:

CREATE TABLE t (i INT) ENGINE = INNODB;

If you omit the ENGINE option, the default storage engine is used. Normally, this is MyISAM, but you can change it by using the –default-storage-engine server startup option, or by setting the default-storage-engine option in the my.cnf configuration file.

You can set the default storage engine to be used during the current session by setting the storage_engine variable:

SET storage_engine=MYISAM;

[…]

To convert a table from one storage engine to another, use an ALTER TABLE statement that indicates the new engine:

ALTER TABLE t ENGINE = MYISAM;

I want MyISAM all the time so I decided for the my.cnf option. But where is my.cnf? According to debianadmin:

sudo nano /etc/mysql/my.cnf

Now append
default-storage-engine = MyISAM
Safe and exit with Ctrl-o, Ctrl-x and restart the server.

sudo restart mysql

and MyISAM it is.

Advertisement

Move a table between MySQL databases

Having organized a lot of datasets in severeal MySQL databases on the same (local)-server the I needed to move a table to another database. The posted solution is copy-paste from Eric Bergen on the MySQL forum:

Alter table can be used to move tables from one db to another.

alter table old_db.foo rename new_db.foo

Sane PATH variable in Emacs on Mac OS X

On Mac OS X the system PATH variable is not recognized by emacs. This means that one can not simply type

mysql

in the emacs shell to get into the database. The emacs shell complains about “binary not found”.

Indeed

echo $PATH

reveals that emacs just looks into /bin, /usr/bin, /sbin and /usr/sbin.

To set the $PATH variable inside emacs one can append the following lines to the .emacs file (found on github, hattip Alex Payne):

; sane path
(setq path "/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/mysql/bin")
(setenv "PATH" path)

Next time Emacs starts one can go to the shell and

mysql

presents the database prompt.

Compressed backup of MySQL database

Wrote several posts on this topic, but none was 100% right. The following is a blockquote from webcheatsheet.com and looks much better researched then my previous tries:

Back up your MySQL Database with Compress

If your mysql database is very big, you might want to compress the output of mysqldump. Just use the mysql backup command below and pipe the output to gzip, then you will get the output as gzip file.

$ mysqldump -u [uname] -p[pass] [dbname] | gzip -9 > [backupfile.sql.gz]
If you want to extract the .gz file, use the command below:

$ gunzip [backupfile.sql.gz]

Restoring your MySQL Database

Above we backup the Tutorials database into tut_backup.sql file. To re-create the Tutorials database you should follow two steps:

Create an appropriately named database on the target machine
Load the file using the mysql command:
$ mysql -u [uname] -p[pass] [db_to_restore] < [backupfile.sql]
Have a look how you can restore your tut_backup.sql file to the Tutorials database.

$ mysql -u root -p Tutorials < tut_backup.sql
To restore compressed backup files you can do the following:

gunzip < [backupfile.sql.gz] | mysql -u [uname] -p[pass] [dbname]

MySQL tables must have indexes

A newbee to MySQL myself I had a rather odd experience: A simple inner join of three research tables took severeal hours. I posted the problem on the forum.mysql.com and the following is the result of the post.

To give the conclusion first:

1. Always create indexes for each key-variable in each and every MySQL table, no matter how small or big.
   ALTER TABLE MyTable ADD INDEX(MyID)
   where MyTable is a Table in the database and MyID is one of the key variables.

2. Never use brackets in consecutive joint statements! The right way is:
   select * from Table_A A inner join Table_B B on A.id = B.id inner join Table_C on A.id = C.id;
   Brackets force the creation of temporary tables and increase execution time

The first one is the most important. It reduced the time for a join from more then 10 hours to a couple of minutes.

MySQL backup

Just for the record: How to combine mysqldump and zip to archive all MySQL databases on the host. I am using a simple MySQL database server on localhost, to organise research tables before analysis.

mysqldump --all-databases | zip -9 allDB_backup_110415.sql.zip -

mysqldump --all-databases writes the content of all databases into the pipe and
zip -9 Filename - compresses the standard input (note the dash ‘-‘ at the end!) to ‘Filename’ (-9 gives maximum compression).

The reverse following the man page of ‘mysqldump’:

You can load the dump file back into the server like this:

shell> mysql db_name < backup-file.sql

Or like this:

shell> mysql -e "source /path-to-backup/backup-file.sql" db_name

Emacs as MySQL frontend

After working quite a time with some minor MySQL databases in the (Emacs-) Terminal I just looked up the preinstalled SQL related emacs functions. Just entered M-x sql TAB and indeed the autocompletion showed a function sql-mysql, as expected…

I gave it a try with
M-x sql-mysql
and after prompting for database, servername, username and password Emacs connected to the database and presented the MySQL shell. So I bound sql-mysql to some keyboard shortcut, BUT entering the whole connection parameters each and every time was not acceptable.

Atomized.org has a really nice post on Enhancing Emacs’ SQL Mode (you can have a look, but you cannot read the post before pasting the content to a text editor). There I found some excellent functions which would provide a solution:

(setq sql-connection-alist
'((pool-a
(sql-product 'mysql)
(sql-server "1.2.3.4")
(sql-user "me")
(sql-password "mypassword")
(sql-database "thedb")
(sql-port 3306))
(pool-b
(sql-product 'mysql)
(sql-server "1.2.3.4")
(sql-user "me")
(sql-password "mypassword")
(sql-database "thedb")
(sql-port 3307))))

(defun sql-connect-preset (name)
"Connect to a predefined SQL connection listed in `sql-connection-alist'"
(eval `(let ,(cdr (assoc name sql-connection-alist))
(flet ((sql-get-login (&rest what)))
(sql-product-interactive sql-product)))))

(defun sql-pool-a ()
(interactive)
(sql-connect-preset 'pool-a))

Now, you can just run sql-pool-a and get connected right away. Because the buffers have good names, you can easily fire up many connections.

I included it in my .emacs file and appended
(DefGlobKey "s-a" 'sql-pool-a)
and with a keystroke the database promt appears.

Thx, atomized.org, but what’s that webpage giving you an epileptic fit looking at it?!. Excellent page 🙂

MySQL command line with OS X 10.6.

After installing MySQL 5.1.54 on Mac OS X 10.6.6 (Snow Leopard) I tried to access the server in the terminal and got:

-bash: mysql: command not found

Bash could not find the path, so I had to add it. I am not that familiar with the internals of the unix terminal anyway, so I found a highly informative thread on superuser.com.

Especially the post of Jim Logan solved the issue:

Take a look at the file /etc/paths, which is used by /usr/libexec/path_helper, which is used by /etc/profile. For MacPorts, use “sudo” to append “/opt/local/bin” to the end of the /etc/paths file and reopen the terminal window.

So I opened the terminal and used nano:
sudo nano /etc/paths

and appended the line
/usr/local/mysql/bin

to the file, saved, exited, restarted the terminal …. and voila, solved.

Fixing MySQL autostart in Mac OS X

I switched my work laptop – MacBook Air 3.2 – not without headaches… No Ubuntu at the time of writing. This will change.

But for now I needed a MySQL server running, Emacs not colliding with the Mac desktop and R all talking nicely to each other.

After downloading the MySQL-5.5.8-osx10.6-x86_64.dmg and installing MySQL-5.5.8 on MacOSX-10.6.6 (Snow Leopard) everything went fine first.

1. Install the database,
2. install the Startup script,
3. install the extension for the System Preference dialogue.

I did everything worked – until restart. OSX complained about a failing startup script (MySQL) and that was that. The menu in the system preferences did not help and trying to start the MySQL server from the shell did also not work (without throwing any error code).

The solution of debenriver (scroll down a bit) worked for me. In a nutshell:

1. Fix the user rights of the startup script (that was what OSX complained about):
   sudo chown -R root:wheel /Library/StartupItems/MySQLCOM
   That should fix the problem, but it is not enough.
2. Restart the computer
3. Open a terminal and navigate to MySQL
   cd /usr/local/mysql
4. Change to superuser mode for a while
   sudo echo
5. Set a root password for your database
   sudo bin/mysqld_safe &
   hit enter after the & and continue
   bin/mysqladmin -u root password 'xxxxxxxx'
   to enter a root password for the MySQL database (replace xxxxxx with your chosen password. This should probably be different from your computer password. Don’t forget the single quotes).

UPDATE: No, 5.5.8 does not work. After a reboot (or something) MySQL refused again to startup. I followed the recommendations on the same thread I quoted, removed MySQL 5.5 and installed MySQL 5.1.54 which was also the current version in the Ubuntu repositories on my old workhorse. Works just fine.

MySQL Error 1366: Incorrect Integer Value for Column

Trying to load a colon separated file into a MySQL database table I ran into an error 1366.

1366 Error Incorrect Integer Value for Column 'MyIntegerColumn' at Row 1

The format of MyIntegerColumn was INT as you could imagine.

The point is that missing values in the .csv file where interpreted as an empty string (”) and MySQL 5.1 complains about putting a string record into an integer variable. I did not find any usefull solution via google: how to load missing values as NULL into an integer column.

The MySQL-Reference solved the problem:

Handling of NULL values varies according to the FIELDS and LINES options in use:

For the default FIELDS and LINES values, NULL is written as a field value of \N for output, and a field value of \N is read as NULL for input (assuming that the ESCAPED BY character is “\”).

Since the ESCAPED BY character is “\” by default, it is sufficient to open the text/csv file and substitute two consecutive fieldseparators (‘,,’ e.g.) with ‘,\N,’. Save the file and repeat your loading syntax. In my case it was:

LOAD DATA LOCAL INFILE '/path/to/YourLoadFile.csv' INTO TABLE YourTargetTable FIELDS TERMINATED BY ',' IGNORE 1 LINES;

Voila. Worked for me.