Summarizing virtualization technologies from Sun

Inspired by a recent discussion on Xen Discuss about what different virtualization solutions were available from Sun I thought I’d take the information provided by Volker A. Brandt and Bernd Schemmer and put it into a convenient table.

Name Sparc x86 Method OS Kernel Guest OS
xVM X software(1) many various
VirtualBox X software many various
Containers/Zones X X software one Solaris/branded
LDOMs X hardware many Solaris(2)
Domains (Mx000 series) X hardware many Solaris
Domains (E10K, SF##K series, v1280, v4800) X hardware many Solaris, Linux

Footnotes(1) with CPU assistance for “full” virtualisation(2) experimental Linux/BSD (?) supportActually, Sun have a pretty good summary, but some of the technologies are hidden behind the hardware on which they run, for example LDOM is a firmware-level solution built into many of the SPARC hardware solutions.

Podcast Producer: Writing Actions

The second article in my Podcast Producer series is now available on Apple Developer Connection. Podcast Producer: Writing Actions discusses the actions that make up the processing of podcasts sent to Podcast Producer into the final formats and associated blog posts, emails and other material that publishes the podcast out to the world. From the introduction:

In order to write effective actions for Podcast Producer, you must understand the different properties that are executed from the command line. Individually, actions are discrete operations, but when chained together they provide a powerful processing environment so that the output from one action is compatible with the input of the next action within the process.In this article, designed for Podcast Producer administrators developing their own workflows, you will learn about how to write and execute actions. Armed with this knowledge, you can easily develop your own actions to build different workflows. You will also learn about how to ensure that your actions are flexible and executable within the Xgrid environment and how to manage execution and resources during processing.

Read: Podcast Producer: Writing Actions

MySQL Documentation Myths

There are a few myths surrounding the MySQL documentation and how it works, and I thought I’d try and dispel some of those myths if I can. If you have any more questions or misunderstandings you want clarified, let me know. Myth:

MySQL Documentation is written by the developers.

Reality

MySQL Documentation is written by a dedicated team of writers with help and input from the developers. There are four main writers, Paul DuBois, Tony Bedford, Jon Stephens, and MC Brown (me!), plus our Team Lead, Stefan Hinz. All the documentation staff are employed full time for the sole purpose of writing documentation. Sure, some of us get involved in other things too, but that’s basically the nature of the job. Some of us simply cannot help ourselves.

Myth

Docs team members are just writers and have no technical expertise.

Reality

It’s tempting to come back with a rude response to this one, but it is a comment I heard from someone at a conference. The reality is that all of us have some technical background, unsurprisingly often with MySQL. Some of us have expertise elsewhere too. Speaking only for myself, go look at MCslp.com for more info. If you want details, feel free to ask, but know that this myth is definitely busted.

Myth

The documentation is updated very rarely.

Reality

Our main tree, mysqldoc, is publicly available, and if you want to go view the commits to that tree, please feel free. It doesn’t take much to see that we commit to that tree all day, and every time we change something, the documentation gets rebuilt. How frequently? Well, on a typical day we will generate 10-15 new versions of each reference manual. It’s actually difficult to rebuild more frequently than that due to the sheer size of the documentation. If you want to check the build date of the documentation, check the intro/preface of each document. The build and build date information is included there.

Myth

The MySQL documentation is small and unused.

Reality

You’d be amazed how many people need to be told RTFM, but a surprising number of people who criticize the MySQL documentation have actually never read it, or, they looked at it years ago and haven’t bothered to look recently because they couldn’t find what they were looking for before.The reality is that our documentation is over 2000 pages per reference manual, which means over 10,000 pages now just for MySQL. There are hundreds more pages on the GUI tools, Workbench, Cluster/NDBAPI, and the Enterprise Monitor.As to the popularity, the hits to the online pages of our manual exceed the hits to every other section of the MySQL website by a significant factor. For the downloadable formats we get an average of 200,000 downloads in all the various formats each month, with occasional spikes up to 800,000. For the online manuals, the documentation pages make up about 45% of all the traffic on mysql.com. Or to say it another way, we account for almost half of all the web traffic that MySQL receives, including downloads. In short, we have no shortage of interested readers.

Myth

Docs team don’t read comments

Reality

Actually, we all get an email each time you post a comment and all of us will read it, determine whether it is suitable, useful, or (occasionally) spam, and either ignore it, delete it, or comment on it accordingly. Often that will happen within minutes of your leaving the comment. If there’s a non-standard reply to your comment, you’ll get that too. Now, we are aware that the comments system has it’s faults. For one, we have one comment system for all the different versions of the manual, which means comments can be confusing and even misleading. We’re fixing that. We’re also trying to address the problem that some comments are really tips, while others are just plain comments and observations. Any other comments or criticisms, let us know. We may not be unaware of the problem, but if we know your pain we can do something about it.

Myth

Docs team don’t accept bugs or corrections

Reality

You can report a bug or correction to us using the standard bugs.mysql.com, or drop us an email to docs@mysql.com.

Myth

Docs are ‘closed source’

Reality

The docs are not closed source – you can download the DocBook XML and the files and tools required to build them (well, beyond the XML parsers, Perl, and other bits and pieces). You can get hold of the repos (via SVN), on the Tech Resources page. That said, we don’t allow anybody to commit changes, but see the response above for information on how to provide changes and fixes. Again this is something we are working to improve on.

Myth

MySQL Documentation is not distributable

Reality

This mostly comes out of the fuss around Debian dropping the man pages from their MySQL distributions You can see the description why here: MySQL Documentation and Debian/Ubuntu. The short answer is that it is a mis-understanding in our license for the documentation, which is not released under the same license as MySQL. You can provide documentation if you provide MySQL, but not on it’s own. The reason for this is that our documentation is updated so regularly that we want to ensure that we only get genuine, up to date, versions of our documentation out there. Trust me, do a search for MySQL and some term and you will find versions of the manual that are months, or even years out of date, which is no help to anybody. It’s about trying to make our documentation readable and usable and not misleading.


OK that’s enough myths busted today, but if you hear any more, or just have additional questions, feel free to ask.

Teaching OpenSolaris

While looking for some information on OpenSolaris in preparation for my talk this week I came across some excellent material providing backgrounds on OpenSolaris for both instructors and students.The material is part of the
Curriculum Development Resources at OpenSolaris.org
and is available as PDFs for download. The documents are short and easy to read, but packed with lots of useful information and a good read for anybody interested in understanding more about the technology and functionality in OpenSolaris.

More Datacenters in a Container

Not exactly portable (unless you have a container lorry or ship handy), but I’d love to have one of Sun’s Sun’s Modular Datacenters available to use.The Modular Datacenter (otherwise known as Project Blackbox) squeezes 7 or 8 racks, depending on the the configuration fits 240 or 280 Us of rackspace into a standard shipping container. The idea is that you can deploy one of these containers very quickly with your desired set up, either to handle all of your datacenter requirements (which it could), or it’s used as a quick solution for a particular project.Interestingly, I can also see it as a potential solution for cloud computing. Not only would a one of these containers make an excellent datacenter in it’s own right, but the ability to quickly expand the capacity when you need kinda takes the idea of cloud computing and easily expanded capacity to another level.Sun are not the only people doing these datacenters in a container though:

All of them promote the density and power of the systems and, more importantly in todays climate, their energy efficiency.In an ideal world we’d line these all up next to each other and compare them, but I hardly think that would be practical. It would, however, be interesting to see what sort of power, performance and flexibility these systems offered.

DimDim and MySQL University

Stop the press! My boss, Stefan Hinz, has just started blogging, with his first post here: Using NetBeans with MySQL.So who is he? Well, Stefan is the guy that keeps the rest of us in the docs team in check and makes sure we do what we’re asked, when we’re are asked and that all of the machinery, legalities and management tasks happen in the background. Without him we really couldn’t function as effectively as we do. It’s wonderful to see some other Docs team members getting in on the act (to be fair to the rest of the team, Jon is also a blogger). We are all writers, you would think the blogging would come as a natural extension. Behind the tease is the simple fact that the improved system for MySQL University I was talking about is getting a trial run this week. We’ve been trying out Dimdim for web conferencing with David van Couvering and I have to say I’m pretty impressed. We chatted, we tried out screen-sharing, presentations and the whiteboard functionality and it all worked really nicely. We’re going to be using it for the MySQL University session this week, Using MySQL with NetBeans. Space will be limited, but feel free to join us if you can.

Using the MySQL Doc source tree

I’ve mentioned a number of times that the documentation repositories that we use to build the docs are freely available, and so they are, but how do you go about using them? More and more people are getting interested in being able to work with the MySQL docs, judging by the queries we get, and internally we sometimes get specialized requests. There are some limitations – although you can download and access the docs and generate your own versions in various formats, you are not allowed to distribute or supply that iinformation, it can only be employed for personal use. The reasons and disclaimer for that are available on the main page for each of the docs, such as the one on the 5.1 Manual. Those issues aside, if you want to use and generate your own docs from the Subversion source tree then you’ll need the following:

  • Subversion to download the sources
  • XML processors to convert the DocBook XML into various target formats; we include DocBook XML/XSLT files you’ll need.
  • Perl for some of the checking scripts and the ID mapping parts of the build process
  • Apache’s FOP if you want to generate PDFs, if not, you can ignore.

To get you started, you must download the DocBook XML source from the public subversion repository. We recently split a single Subversion tree with the English language version into two different repositories, one containing the pure content, and the other the tools that required to build the docs. The reason for that is consistency across all of our repositories, internally and externally, for the reference manual in all its different versions. Therefore, to get started, you need both repositories. You need check them out into the same directory:

$ svn checkout http://svn.mysql.com/svnpublic/mysqldoc$ svn checkout http://svn.mysql.com/svnpublic/mysqldoc-toolset

Assuming you have the downloaded the XML toolkit already, make sure you have the necessary Perl modules installed. You’ll need Expat library, and the following Perl modules:

  • Digest::MD5
  • XML::Parser::PerlSAX
  • IO::File
  • IO::String

If you have CPAN installed, you can install them automatically using perl -MCPAN -e 'install modulename', or use your respective package management system to install the modules for you. You’ll get an error message if there is something missing. OK, with everything in place you are ready to try building the documentation. You can change into most directories and convert the XML files there into a final document. For example, to build the Workbench documentation, change into the Workbench directory. We use make to build the various files and dependencies. To build the full Workbench documentation, specify the main file, workbench, as the target, and the file format you want to produce as the extension. For example, to build a single HTML file, the extension is html. I’ve included the full output here so that you can see the exact output you will get:

make workbench.htmlset -e;
../../mysqldoc-toolset/tools/dynxml-parser.pl
--infile=news-workbench-core.xml --outfile=dynxml-local-news-workbench.xml-tmp-$$ --srcdir=../dynamic-docs --srclangdir=../dynamic-docs;

mv dynxml-local-news-workbench.xml-tmp-$$ dynxml-local-news-workbench.xmlmake -C ../refman-5.1 metadata/introduction.idmapmake[1]: Entering directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/refman-5.1'../../mysqldoc-toolset/tools/idmap.pl refman/5.1/en introduction.xmlmake[1]: Leaving directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/refman-5.1'make -C ../refman-5.1 metadata/partitioning.idmapmake[1]: Entering directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/refman-5.1'../../mysqldoc-toolset/tools/idmap.pl refman/5.1/en partitioning.xmlmake[1]: Leaving directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/refman-5.1'make -C ../refman-5.1 metadata/se-merge.idmapmake[1]: Entering directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/refman-5.1'../../mysqldoc-toolset/tools/idmap.pl refman/5.1/en se-merge.xmlmake[1]: Leaving directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/refman-5.1'make -C ../refman-5.1 metadata/se-myisam-core.idmapmake[1]: Entering directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/refman-5.1'../../mysqldoc-toolset/tools/idmap.pl refman/5.1/en se-myisam-core.xmlmake[1]: Leaving directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/refman-5.1'make -C ../refman-5.1 metadata/sql-syntax-data-definition.idmapmake[1]: Entering directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/refman-5.1'../../mysqldoc-toolset/tools/idmap.pl refman/5.1/en sql-syntax-data-definition.xmlmake[1]: Leaving directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/refman-5.1'make -C ../workbench metadata/documenting-database.idmapmake[1]: Entering directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'../../mysqldoc-toolset/tools/idmap.pl workbench//en documenting-database.xmlmake[1]: Leaving directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'make -C ../workbench metadata/foreign-key-relationships.idmapmake[1]: Entering directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'../../mysqldoc-toolset/tools/idmap.pl workbench//en foreign-key-relationships.xmlmake[1]: Leaving directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'make -C ../workbench metadata/forward-engineering.idmapmake[1]: Entering directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'../../mysqldoc-toolset/tools/idmap.pl workbench//en forward-engineering.xmlmake[1]: Leaving directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'make -C ../workbench metadata/grt-shell.idmapmake[1]: Entering directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'../../mysqldoc-toolset/tools/idmap.pl workbench//en grt-shell.xmlmake[1]: Leaving directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'make -C ../workbench metadata/images.idmapmake[1]: Entering directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'../../mysqldoc-toolset/tools/idmap.pl workbench//en images.xmlmake[1]: Leaving directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'make -C ../workbench metadata/installing.idmapmake[1]: Entering directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'../../mysqldoc-toolset/tools/idmap.pl workbench//en installing.xmlmake[1]: Leaving directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'make -C ../workbench metadata/layers.idmapmake[1]: Entering directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'../../mysqldoc-toolset/tools/idmap.pl workbench//en layers.xmlmake[1]: Leaving directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'make -C ../workbench metadata/notes.idmapmake[1]: Entering directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'../../mysqldoc-toolset/tools/idmap.pl workbench//en notes.xmlmake[1]: Leaving directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'make -C ../workbench metadata/plugins.idmapmake[1]: Entering directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'../../mysqldoc-toolset/tools/idmap.pl workbench//en plugins.xmlmake[1]: Leaving directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'make -C ../workbench metadata/printing.idmapmake[1]: Entering directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'../../mysqldoc-toolset/tools/idmap.pl workbench//en printing.xmlmake[1]: Leaving directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'make -C ../workbench metadata/reference.idmapmake[1]: Entering directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'../../mysqldoc-toolset/tools/idmap.pl workbench//en reference.xmlmake[1]: Leaving directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'make -C ../workbench metadata/reverse-engineering.idmapmake[1]: Entering directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'../../mysqldoc-toolset/tools/idmap.pl workbench//en reverse-engineering.xmlmake[1]: Leaving directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'make -C ../workbench metadata/server-connection-wizard.idmapmake[1]: Entering directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'../../mysqldoc-toolset/tools/idmap.pl workbench//en server-connection-wizard.xmlmake[1]: Leaving directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'make -C ../workbench metadata/stored-procedures.idmapmake[1]: Entering directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'../../mysqldoc-toolset/tools/idmap.pl workbench//en stored-procedures.xmlmake[1]: Leaving directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'make -C ../workbench metadata/tables.idmapmake[1]: Entering directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'../../mysqldoc-toolset/tools/idmap.pl workbench//en tables.xmlmake[1]: Leaving directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'make -C ../workbench metadata/text-objects.idmapmake[1]: Entering directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'../../mysqldoc-toolset/tools/idmap.pl workbench//en text-objects.xmlmake[1]: Leaving directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'make -C ../workbench metadata/tutorial.idmapmake[1]: Entering directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'../../mysqldoc-toolset/tools/idmap.pl workbench//en tutorial.xmlmake[1]: Leaving directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'make -C ../workbench metadata/validation-plugins.idmapmake[1]: Entering directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'../../mysqldoc-toolset/tools/idmap.pl workbench//en validation-plugins.xmlmake[1]: Leaving directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'make -C ../workbench metadata/views.idmapmake[1]: Entering directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'../../mysqldoc-toolset/tools/idmap.pl workbench//en views.xmlmake[1]: Leaving directory `/nfs/mysql-live/mysqldocs/working/Docs/mysqldoc/workbench'XML_CATALOG_FILES="../../mysqldoc-toolset//catalog.xml" xsltproc --xinclude --novalid

--stringparam repository.revision "`../../mysqldoc-toolset/tools/get-svn-revision`"

--param map.remark.to.para 0

--stringparam qandaset.style ""

../../mysqldoc-toolset/xsl.d/dbk-prep.xsl workbench.xml > workbench-prepped.xml.tmp2../../mysqldoc-toolset/tools/bug-prep.pl workbench-prepped.xml.tmp../../mysqldoc-toolset/tools/idremap.pl --srcpath="../workbench ../gui-common ../refman-5.1 ../refman-common ../refman-5.0" --prefix="workbench-" workbench-prepped.xml.tmp > workbench-prepped.xml.tmp2mv workbench-prepped.xml.tmp2 workbench-prepped.xmlrm -f workbench-prepped.xml.tmpXML_CATALOG_FILES="../../mysqldoc-toolset//catalog.xml" xsltproc --xinclude --novalid

--stringparam l10n.gentext.default.language en

--output workbench.html-tmp

../../mysqldoc-toolset/xsl.d/mysql-html.xsl

workbench-prepped.xml../../mysqldoc-toolset/tools/add-index-navlinks.pl workbench.html-tmpmv workbench.html-tmp workbench.html

There’s lots in the output above, and I’ll describe the content as best I can without going in to too much detail in this piece. First off, the make triggers some dependencies, which are the creation of a number of ‘IDMap’ files. These files contain information about the content of the files and are used to help produce valid links in to other parts of the documentation. I’ll talk about ID mapping more in a later post. The next stage is to build the ‘prepped’ version of the documentation, which combines all of the individual files into one large file and does some pre-processing to ensure that we get the output that we want. The next is the remapping. This uses the IDMap information built in the first stage and ensures that any links in the documentation that go to a document we know about, like the reference manual, point to the correct online location. It is the ID mapping (and remapping) that allows us to effectively link between documents (such as the Workbench and Refman) without us having to worry about creating a complex URL link. Instead, we just include a link to the correct ID within the other document and let the ID mapping system do the rest. The final stage takes our prepped, remapped, DocBook XML source and converts it into the final XML using the standard DocBook XSL templates. One of the benefits of us using make is that because we build different stages in the build process, when we build another target, we dont have to repeat the full process. For example, to build a PDF version of the same document, the prepping, remapping and other stages are fundamentally the same, which is why we keep the file, workbench-prepped.xml. Building the PDF only requires us to build the FO (Formatting Objects) output, and then use fop to turn this into PDF:

$ make workbench.pdfXML_CATALOG_FILES="../../mysqldoc-toolset//catalog.xml" xsltproc --xinclude --novalid

--output - ../../mysqldoc-toolset/xsl.d/strip-remarks.xsl workbench-prepped.xml
| XML_CATALOG_FILES="../../mysqldoc-toolset//catalog.xml" xsltproc --xinclude --novalid

--stringparam l10n.gentext.default.language en

--output workbench.fo-tmp ../../mysqldoc-toolset/xsl.d/mysql-fo.xsl -Making portrait pages on USletter paper (8.5inx11in)mv workbench.fo-tmp workbench.foset -e;
if [ -f ../../mysqldoc-toolset/xsl.d/userconfig.xml ]; then

../../mysqldoc-toolset/tools/fixup-multibyte.pl workbench.fo workbench.fo.multibyte;

mv workbench.fo.multibyte workbench.fo;

fop -q -c ../../mysqldoc-toolset/xsl.d/userconfig.xml workbench.fo workbench.pdf-tmp > workbench.pdf-err;
else

fop -q workbench.fo workbench.pdf-tmp > workbench.pdf-err;
fimv workbench.pdf-tmp workbench.pdfsed -e '/hyphenation/d'

You can see in this output that the prepping and remapping processes don’t even take place – the process immediately converts the prepped file into FO and then calls fop. That completes our whirlwind tour of the basics of building MySQL documentation, I’ll look at some more detailed aspects of the process in future blog posts. Until then, you might want to read our metadocs on the internals in MySQL Guide to MySQL Documentation.