My Sessions at UC2009

I’m speaking at the User Conference this year, with a half-day tutorial and three further sessions. The running theme is performance, both in terms of the performance of your queries, and in terms of scaling up. Scale Up, Scale Out, and High Availability: Solutions and CombinationsThis is the big tutorial. It’s difficult to resolve what I’ll be talking about into a few sentences, but think about all of the different technologies available here – replication, partitions, sharding, DRBD, memcached – I’ll be talking about all of them, and more importantly combinations of the different solutions and where the potential performance gains and pitfalls are. I’ll also be using the opportunity to demonstrate some of the more obscure combinations that you can use to provide the environment you need. How I used Query Analysis to Speed Up my ApplicationsFor query analysis, I’ll start with some of the basic methods available to us for performance monitoring, including EXPLAIN and DTrace, before I look at the query analysis provided by MySQL Enterprise Monitor. As an advisor to the group I’ve been looking at it for a while and used it on my own sites to identify a range of different query problems. Improving performance by running MySQL multiple timesIt isn’t talked about much, but there are times when running a single instance of MySQL doesn’t get you either the performance or environment that you need to support your applications. In this presentation I’m going to look at some of the benefits, from simply running multiple instances, to using solutions like VMware, Xen, LDOMs, BSD Jails, and Solaris Containers. Using MySQL with the Dojo ToolkitThe final presentation is something a little more fun. The Dojo Toolkit is a JavaScript kit for developing AJAX applications. There are some really fun things you can do with Dojo, but getting the best combination and cool and efficient with MySQL is an art. We’ll look at two quick examples; the first is a browsable interface to large quantities of data. The other is dynamic graphing using MySQL as the backend. If, within the bounds of any of these presentations there is something you would like covered, please let me know.

DTrace in MySQL: Documentation and a MySQL University Session

DTrace has been something that I’ve been trying to get into the MySQL server for more than a year now.After a combination of my own patches and working with Mikael Ronstrom and Alexey Kopytov we finally have a suite of probes in MySQL 6.0.8. Better still, after a short hiatus while I was busy working on a million-and-one other things, the documentation for those probes is now available: Tracing mysqld with DTrace. The documentation is comparatively light and deep all at the same time. It’s lightweight from the perspective that I’ve added very little detail on the mechanics of DTrace itself, since there is no need to replicate the excellent guides that Sun already provide on the topic. At the same time, I’ve tried to provide at least one (and sometimes two) D script examples for each of the groups of probes in the 6.0.8 release. So what next for MySQL DTrace probes? Well, the next version of the probes has already been worked on. I’ve been testing them for a month or so, and due to a problem with the probes on SPARC I couldn’t approve the patch, but I managed to resolve that last week. The new patch extends the probes to enable a more detailed look at certain operations, and it enables us to expand the probes to be placed anywhere within the server, including individual engine-level row operations. If you want a demonstration of DTrace in MySQL, some of the things you can monitor without using the user probes we’ve added, and those new probes I just mentioned, then you will want to attend the MySQL University session this Thursday (12th Feb), at 14:00UTC where I’ll be doing all of the above. As a side note, because I know there are people interested, last week I also finished the patch for these probes to go into the MySQL 5.1.30 version that we will be putting into OpenSolaris. Sunanda is working on getting that release out there as I type this.

LOSUG Presentation Slides Now Available

My presentation at LOSUG on tuesday went down like a house on fire – I think it would be safe to say that the phrase for the evening was ‘It’s a cache!’. For that to make sense, you need to look at the slides, which are now available here. Attendance was great, but it seems the last minute change of day meant that some people missed the session. We had 151 people register, and about 80 turned up on the night.

LOSUG January: MySQL/DTrace and Memcached

Next Tuesday (27th Jan), I’ll be speaking at the London OpenSolaris User Group again. For those that follow the LOSUG meetings, we normally the third thursday of the month, but due to the overwhelming popularity of the event this month (more 100 registrations so far) we have had to push the event back to the last Tuesday of the month. This month, I’ll be talking about the DTrace probes that we have added into MySQL and demonstrating their use and functionality. Along the way I’ll also cover some of the internals of MySQL and how it works (and how they relate to the DTrace probes we’ve added), how to use the probes to analyze and diagnose your queries, and how I’ve already used the DTrace probes to provide information up to the Query Analysis functionality within Enterprise Monitor. After we’ve looked at performance monitoring and optimization with DTrace, I’ll then demonstrate how to get a little more performance out of your application using MySQL by taking advantage of the memory cache offered by Memcached. As before, I’ll provide a link to the finished presentation once I’ve demonstrated everything to the LOSUG folks on tuesday. If you happen to be in London, and don’t already have plans of some sort, please feel free to come along – food and drink will be provided after the session and I’ll be around for as long as I can for any questions.

Multiple VCS Updates and Cleanups

I spend a lot of time updating a variety of different repositories of different varieties and denominations, and I hate having to do that all by hand – I’d rather just go up into a top-level directory and say update-all and let a script work out what to do, no matter what different repos are there. I do it with a function defined within my bash profile/rc scripts, and it covers git, bzr, svn, bk, and cvs. The trick is to identify what type of directory we are updating. I do this, lazily, for each type individually, rather than for each directory, but I’ve found this method to be more reliable.

update-all () { for file in `ls -d */.svn 2>/dev/null`; do realdir=`echo $file|cut -d/ -f1`; echo Updating in $realdir; ( cd $realdir; svn update ); done; for file in `ls -d */.bzr 2>/dev/null`; do realdir=`echo $file|cut -d/ -f1`; echo Updating in $realdir; ( cd $realdir; bzr pull ); done; for file in `ls -d */.git 2>/dev/null`; do realdir=`echo $file|cut -d/ -f1`; echo Updating in $realdir; ( cd $realdir; git pull ); done; for file in `ls -d */CVS 2>/dev/null`; do realdir=`echo $file|cut -d/ -f1`; echo Updating in $realdir; ( cd $realdir; cvs up ); done; for file in `ls -d */BitKeeper 2>/dev/null`; do realdir=`echo $file|cut -d/ -f1`; echo Updating in $realdir; ( cd $realdir; bk pull ); done; unset realdir}

That’s it – a quick way to update any directory of repos.

MySQL University: MySQL and OpenSolaris

On Thursday, November 13, 2008 (14:00 UTC / 14:00 BST / 15:00 CET), I’ll be presenting a MySQL University session on MySQL and OpenSolaris. The presentation will be similar to the presentation I did at the London OpenSolaris Users Group in July, you can see that presentation by visiting the LOSUG: July 2008 page. The presentation on thursday will be slightly different – I’ll be providing a bit more hands-on information about how to install MySQL, how to configure and change the configuration and some more detail on solutions like the Webstack and Coolstack distributions. I’ll also cover our plans for the inclusion of MySQL 5.1 in OpenSolaris, which will happen next year, and provide some examples on the new DTrace probes that we have been adding to MySQL generally. Of course, if there’s anything specific you want me to talk about, comment here and I’ll see if I can squeeze it into the presentation before thursday.

Compiling MySQL Workbench on Gentoo

The Workbench team have just announced the release of Workbench for Linux, including binary packages and source packages with instructions on how to build. I’m a Gentoo Linux user, so I prefer building from source, and you’ll need to emerge the following packages (and note the USE) requirement as part of the source build process:

# USE="svg" emerge libzip libxml2 libsigc++     libglade libgtksourceviewmm media-libs/glut mysql lua     ossp-uuid libpcre libgnome gtk+ pango cairo

Depending on your config and platform, you may need to bypass some package masking by adding the packages to your /etc/portage/package.keywords file. Then download and install the ctemplate library from google code page. The current Gentoo version is 0.90, and you really should install the 0.91 version. With the required packages and libraries in place, download the Workbench sources and then build:

# cd mysql-workbench-5.1.4alpha# ./ make# make install

That should build and install MySQL Workbench for you. Just to confirm, here’s a screenshot of the built Workbench running on Gentoo Linux and displaying to my Mac OS X-based desktop.


Summarizing virtualization technologies from Sun

Inspired by a recent discussion on Xen Discuss about what different virtualization solutions were available from Sun I thought I’d take the information provided by Volker A. Brandt and Bernd Schemmer and put it into a convenient table.

Name Sparc x86 Method OS Kernel Guest OS
xVM X software(1) many various
VirtualBox X software many various
Containers/Zones X X software one Solaris/branded
LDOMs X hardware many Solaris(2)
Domains (Mx000 series) X hardware many Solaris
Domains (E10K, SF##K series, v1280, v4800) X hardware many Solaris, Linux

Footnotes(1) with CPU assistance for “full” virtualisation(2) experimental Linux/BSD (?) supportActually, Sun have a pretty good summary, but some of the technologies are hidden behind the hardware on which they run, for example LDOM is a firmware-level solution built into many of the SPARC hardware solutions.