Cautionary Information
2.1 Scope of This Chapter
In this chapter we will cover some of the major issues encountered when using Qt and/or databases. These will be the kind of things which “shouldn't” happen. When they do happen you chase your tail for hours and sometimes days tracking the problem down.
2.2 Development Restrictions
If you come from a development environment on a robust platform which was centered around a code management library, developing with the Qt tool set is going to seem like an act of extreme sacrilege. The Unix, and by extension Linux, platforms do not have any concepts of logical names or distributed architectures. Here, you have at best, a PC mindset. That mindset, being small, simple, and not looking further ahead than the weekend, developed a tool set which matched its thought patterns. There is only one guiding principle.
Everything in a single directory tree.
You will find this philosophy wrapped up in every Open Source project you download or use. Back in the days when we only had indexed files we could muddle through. When you start actually running software as an installed image and need multiple versions, this philosophy starts to break down. It goes completely out the window when you need to run D,T,S, and P versions of databases. ( Development, integration Testing, production Support, and Production, all on the same machine or cluster. Generally running Production on your development machine(s) is part of a disaster recovery plan.) I covered the bulk of the DTSP issues in ISBN13: 9780
977086603, please refer to it for further information.
2.3 Building a Qt Program
Once you have all of your source files in a single directory, you need to create a Qt project file. If Qt has been correctly installed on your machine, you can do this by simply typing:
1)
2) qmake -project
3)
A file using the current directory name with a .pro extension will be generated. The name of the executable file which comes out of the entire build process will also be the same as the directory. (I think you are now starting to see why I harped on the “Everything in a single directory tree” philosophy.) It was never a good idea on midrange platforms to have a directory provide the name for an executable contained within the directory, and it was an even worse idea on the mainframe platform, but hey, the Open Source community is all gaga over using a convention which has failed miserably on every other platform.
When dealing with a relational database via the Qt libraries, you need to manually edit the resulting PRO file to add the following highlighted line prior to generating the Makefile.
4) ################################################################
5) # Automatically generated by qmake (2.01a) Sun Jun 6 19:10:04 2010
6)###############################################################
7)
8) TEMPLATE = app
9) TARGET =
10) DEPENDPATH += .
11) INCLUDEPATH += .
12) QT += sql
13)
14) # Input
I don't know why the qmake tool isn't smart enough to do a quick scan inside of the files and determine an application uses the SQL classes. Once you've added the line and saved the file you can run qmake one more time without any parameters to generate a Makefile. Your final act of building an executable is to type make and watch it run. Assuming you don't have any typos or other compilation errors, you will get an executable with the same name as your directory.
Because this section of the book is about development restrictions, take a look at the other lines above our highlighted line. The “.” stands for the current directory. Everything is assumed to be in your current directory. If you want to change the resulting executable name you must provide a value to the TARGET line. When you are using third party libraries and need to force recompilation due to a header file changing, you need to add the directory tree holding the header files to DEPENDPATH. To ensure the compiler looks in that third party directory for header files, you need to also add it to the INCLUDEPATH. Use a space or a “;” to separate entries in the path list. There is a long list of variables which can be defined in this file. Use your favorite Web search engine (hopefully not Google) and search for “Qt includepath” without the quotes. You should see at least one link to a Nokia/Trolltech documentation page. Hopefully the documentation page has a version number close to the version you are using of Qt so everything works the way it is described.
2.4 The Database Username Restriction
One of the most frustrating aspects you will encounter is obtaining the username and password for database access. It sounds simple. Using a procedural tool, you would pop up a window to prompt for these, then delete that window and display your main form or forms. Even with our Java applet in “The Minimum You Need to Know About Service Oriented Architecture” we handled this cleanly. Not so when it comes to designing your Qtbased applications. Handling this one task dramatically impacts your application design. Most examples you will find for Qt concentrate only on the GUI aspect since it was originally a GUI library. Each will have a main.cpp file which looks much like the following.
1) #include <QApplication>
2)
3) #include "mydialog.h"
4)
5) int main(int argc, char *argv[])
6) {
7) QApplication app(argc, argv);
8) MyDialog *dialog = new MyDialog;
9) dialog->show();
10) return app.exec();
11) }
When the initial GUI object is a dialog you typically allocate it by calling new and let the Qt environment delete the attached object when the application exits. What you have to remember is that Qt programs aren't just GUI OOP programs, they are event driven. Everything occurs in the event loop handled by the QApplication object method exec(). I've been through this many times with other products. I'm very glad I am not covering device IO in this book like I did in my first Zinc book. When you end up handling IO from 32+ port serial cards and controlling relays via digital IO and Opto22 components you end up having to code your own event loop to handle such things.
Adding a database to this mix opens nearly the same can of worms, not because you have to hack your own event loop, but because in the Qt world the database object is global data. It really needs to be owned by the main.cpp file so it never goes out of scope and isn't garbage collected on you. Keep in mind what I told you about C++ in “The Minimum You Need to Know to Be an OpenVMS Application Developer”: Once an object goes out of scope it is eligible for garbage collection. Some compilers garbage collect immediately and others postpone the collection until idle time. Java developers will have an easier time understanding this given that Java almost always waits until idle time to perform garbage collection.
Your second major problem with adding a database to the mix is the method different databases use to validate a user. Some relational engines are configured so that any user on the local machine has automatic validation, other relational databases provide their own user authentication. Some databases use both methods. The first for local users and the second for users accessing via TCP/IP services.
Many of you are probably having trouble understanding what I'm complaining about, so let me give you a small example. Here is the main.cpp from the xpnsqt project.
1) /*************************************************************
2) * Originally created by Roland Hughes at Logikal Solutions.
3) * Project is being released under GPL2 license.
4) *
5) * At some point in the future either Logikal Solutions or Roland Hughes
6) * may elect to publish a book with this original source code in it. If
7) * the book is written it will be part of "The Minimum You Need to Know"
8) * book series.
9) ******************************************************************/
10) #include <QtGui>
11) #include <QtSql>
12) #include <QtDebug>
13)
14) #include "XpnsLogonDialog.h"
15)
16) int main(int argc, char *argv[])
17) {
18) int i_x;
19) QString qtDbName;
20) QTime theTime = QTime::currentTime();
21)
22)
23) QApplication app(argc, argv);
24)
25) {
26) QString driverName = "QPSQL";
27)
28) //
29) // Set up information for driver and see if driver is available
30) //
31) qtDbName = "xpns" + theTime.toString("hhmmsszzz");
32) QSqlDatabase db = QSqlDatabase::addDatabase(driverName, qtDbName);
33) if (!QSqlDatabase::isDriverAvailable( driverName)) '7b
34) QMessageBox::critical( 0, "Missing PostgreSQL Driver",
35) "The Qt driver for PostgreSQL "
36) + driverName
37) + " is not installed. You must install it before running this program",
38) QMessageBox::Ok);
39) db = QSqlDatabase(); // reset to avoid warning
40) return 0;
41) }
42)
43) XpnsLogonDialog *xpnsLogonDialog = new XpnsLogonDialog( 0, qtDbName);
44) xpnsLogonDialog->show();
45) i_x = app.exec();
46) qDebug() << "About to delete dialog";
47) delete xpnsLogonDialog;
48) qDebug() << "Dialog deleted, now clearing database";
49)
50) db = QSqlDatabase::database(); // clear connection to avoid error
51) }// end scope of logon dialog
52)
53) QSqlDatabase::removeDatabase(qtDbName);
54)
55) return i_x;'7d
Pay special attention to the highlighted '7b'7d surrounding the inner code in this module (listing lines 25 through 51.) This is required to ensure the db object goes out of scope and is garbage collected. Just prior to exiting that block I have to assign the db object to the default database. You can only remove a database from the global storage area once all references to it have been removed.
The global QSqlDatabase object creates a “default” database object whenever you include SQL support in your program. Many developers will only use the default database. The difference between a default database and other databases is the default database doesn't have a name. Notice the highlighted line calling the method addDataBase(). The first parameter is a string containing the driver name. The second parameter is a name you give this database object. Any other module in your program can use any database object you add to the global database object. In affect it is a list of databases your program has access to and is stored globally. You cannot remove a database from this list as long as there is still an object referencing it. Oddly enough, the reference count isn't reduced when the db object goes out of scope and is garbage collected. I assume they will fix this bug eventually. For now, you need to remove that reference by reassigning the object to the default database.
If you fail to remove named database objects, or fail to remove all references to them, you will receive warning messages in the default output window when your application closes. Many developers simply ignore these messages and set there applications up to be launched from screen icons rather than run inside of a terminal window. If there is no default output device (like a terminal window) you simply don't see these messages.
Notice how I had to launch a login dialog as the initial screen/form for this application. When we get to the chapter covering this application you will see that the login dialog had to launch the actual main screen/form once it completed the login process. This isn't the pinnacle of OOP benefits.
Why did I have to launch the login dialog as the initial form? Application design. In order to have combo boxes populated from various tables in your database, you have to actually have a connection to the database and those tables. In order to get that connection, you have to ensure you have a username and password. You cannot assume the database will allow free access to anyone who wants it. In fact, the current wrapper provided by Qt doesn't allow you to connect without a username and password, even when PostgreSQL allows this as we have seen. Now my login dialog is tightly coupled with the application.
If you choose to not have any combo boxes or database populated pick lists on your main form, you can write a generic login dialog which can be popped up after your main form has been created and displayed. This generic dialog will be loosely coupled with your application and infinitely reusable. The problem is that a developer rarely gets to make the choices about what is and isn't on a form. Users want it all and they don't care how you have to warp the architecture to produce it.
2.5 Garbage Collection Restrictions
Java rots your brain. I really admire developers who can easily slide between Java in the morning and C++ in the afternoon without ever creating a memory leak on any system and without using tools to grind their application checking for memory leaks. My personal guess is that there are about five such people in the software development universe. I have no idea who they are, but I firmly believe there cannot be more than five of them. I especially doubt that number is increasing.
When you are programming in Java, you have to allocate everything except the scalar types with new. The JVM keeps track of your references and does garbage collection whenever it deems garbage collection is appropriate. C++ automatically cleans up stack allocated variables when they go out of scope, but variables allocated via new must be physically killed off by a call to delete.
Qt tries to aid in this by telling you it deletes all objects which are derived from QObject. I have seen this statement in a lot of documentation, and I'm not exactly buying it. Actually, it is mostly true, but telling people this is causing more problems than it is solving. The main problem occurs with QSqlDatabase. You have to clean up the references yourself. Earlier I showed you some code where I had to reassign our pointer to the default instance in order to free the reference, then I was able to remove the database from the list of databases.
Why do we find ourselves in this bind? Location, location, location. QSqlDatabase is global. The instances you declare of it tend to be on the stack rather than on the dynamic heap.
QSqlDatabase db = QSqlDatabase::addDatabase(driverName, qtDbName);
Only dynamically allocated objects can be removed by the delete operator in C++. Objects allocated on the stack are supposed to be cleaned up with the stack clean up prior to return, but you cannot remove them. Because you cannot physically delete stack allocated objects yourself, you cannot get rid of the references to global objects cleanly.
But when the object went out of scope it should have been garbage collected, right? No. It was made available to garbage collection, but each platform implements garbage collection at a time and place of its own choosing. The only way to clear the reference was to reassign my stack object to the default database connection prior to exiting the enclosing block.
db = QSqlDatabase::database(); // clear connection to avoid error
2.6 More Database Issues
Almost all of the examples I find floating around the Web show how to use the default database connection, not a named connection. Why should you avoid using the default database connection? Come on, think about it, you know the answer. Module reusability. If you always pass in a string with the name of the database as Qt has it stored, you can call your modules/classes from any application you develop. It doesn't matter if that connection is the first or the 50th connection.
You will find the xpnsqt application is coded quite differently from the mega_zillionare application used in other books in this series. The main.cpp for xpnsqt appears earlier in this chapter. It is the one we have been discussing with respect to database issues. I didn't notice it when working on Ubuntu 8.10, but when working on SuSE 11.x I was seeing error messages like the following:
roland@roland-desktop:~/megaqt$ ./megaqt
QSqlDatabasePrivate::database: unable to open database: Access denied for user
'roland'@'localhost' (using password: NO) QMYSQL: Unable to connect
Successfully connected to database
The instruction which tossed this error wasn't the open statement. It was the call to make a copy of the database instance.
QSqlDatabase db = QSqlDatabase::database( m_qtDbName);
I must say I was really taken aback by this. All of the information you could find online said you needed the main.cpp file to declare your first instance of QSqlDatabase so it would be owned by the last module to complete execution. Most examples I could find did it pretty much the way I was doing. It appears that when you attempt to get a copy or instance of an existing database connection, the method returning that instance automatically tries to open it for you. Quite the little helper isn't it. “The road to Hell is paved with good intentions.” A lot of truth in that statement.
There was a slightly larger shock waiting for me when I ran the xpxnqt application and saw the following in my terminal window:
roland@lgkl-destop:~/xpnsqt2$ ./xpnsqt2
QInotifyFileSystemWatcherEngine::addPaths: inotify_add_watch failed: No such file or directory
QFileSystemWatcher: failed to add paths: /home/roland/.config/ibus/bus Bus::open: Can not get ibus-daemon's address. IBusInputContext::createInputContext: no connection to ibus-daemon QSqlDatabasePrivate::database: unable to open database: "FATAL: database
"roland" does not exist QPSQL: Unable to connect" About to delete dialog
Dialog deleted, now clearing database
When you look through the various support forums and email lists on the subject of Qt programming, you will find quite a few hacks, suggestions, and work arounds. Everybody seems to be happy simply getting one to three lines of code which makes the error go away. They don't seem to be interested in actually fixing the problem. In this case, the problem was caused by someone trying to be helpful. It only became a problem because a lot of other people tried to be helpful telling people they needed to get an instance of the database object in their main.cpp file to keep it from going out of scope during the life of their program.
What we really need to make things work cleanly is our main.cpp to declare a driver for the default database connection and get a copy of the instance. We need a driver which will not throw an error trying to connect when a method tries to get a copy of a partially configured entry.
I guess it shouldn't be any surprise that Qt comes with its own version of SQLite on most platforms. I could not find an example of this anywhere I searched, but this is basically what you need to do. We will see this done in the megaqt (mega_zillionare) example, not the xpnsqt example.
If you make a typo in your driver name, and you happen to have launched your application from a terminal window, you will see a helpful error message along the lines of the following:
roland@roland-desktop:~/megaqt$ ./megaqt
QSqlDatabase: QMYSQL driver not loaded
QSqlDatabase: available drivers: QSQLITE QMYSQL3 QMYSQL QPSQL7 QPSQL
roland@roland-desktop:~/megaqt$
Of course you might have the driver typed correctly and even see the name of it in the list of available drivers. In that case, you are using a static Qt library. In the source file which is actually making reference to the driver you need to type the following macro:
Q_IMPORT_plugin(qsqlmysql)
and in the .pro file you need to add:
QTPLUGIN +=qmysql
Here is another beautiful error message which will have your head slamming against the keyboard:
QSqlDatabasePrivate::removeDatabase: connection 'blah' is still in use, all queries will cease to work.
I loved seeing this message. It took a long time to figure it out. This is the message you get when you forgot to remove your last reference to a database before calling removeDatabase(). I've already shown you the fix for it. Simply assign your database variable to the default database before calling remove. That will clear the reference in most cases. You get this error even if the database is closed because the reference counter is greater than zero.
2.7 Pitfalls of Mandatory Passwords
You have already seen a few examples from the world of PostgreSQL where it will let a properly configured user connect to a database without forcing a password prompt. The database itself has to be told to trust local system security before such things are allowed.
roland@lgkl-destop:~$ psql -d postgres psql (8.4.4)
Type "help" for help.
postgres=#
Most of your security type geeks make it a point to turn this feature off on any newly created system. The last thing they want is someone with access to a sensitive database (like one full of credit card info) walking away from their terminal without locking it. Sometimes the user community wins and sometimes it doesn't.
I'm pointing this out in the cautionary section because it can really screw your design. Actually, it can make entire applications impossible to develop. In the next chapter you will encounter our XPNSQT application. While it does not try to download, install, and configure PostgreSQL automatically for the user, it does try to create a missing database.
You should not be surprised to learn that creating a year specific expense database will require user input. It should surprise you even less to learn that the application builds a SHELL command on the fly using the tax year and other information to create the database.
cmd = "createdb " + db.databaseName() + " 'Expense Information';";
int x = system( cmd.toAscii());
The system() command in C/C++ does not create a shiny new terminal window where a user could view and respond to prompts that might be encountered when the command runs. If someone needs or wants that capability, they will have to modify the application to use QProcess provided by Qt.
I need to point out that “createdb” is a helper program provided with the PostgreSQL database package. It is expected to reside in a directory which is in everyone's path. There are actually quite a few of these helper programs. Where they get installed depends upon your platform and database version.
2.8 Database Frailty
If you work in IT long enough, you will find out there are reasons for most everything. Many times you will find that none of those reasons were good, but a course of action was chosen anyway. This is especially true of database products.
On the OpenVMS platform, we have a wonderful database called RDB. It was originally developed by DEC (Digital Equipment Corporation) and is now owned by Oracle. No, it isn't the Oracle database you now see running on useless Windows boxes, I'm talking about a good database. RDB is completely integrated with the operating system. It uses the ACLs (Access Control Lists) and rights identifiers from the same UAF (User Authorization File) as the operating system. In short, it doesn't have to provide some hokey rollyourown user administration and rights identifier system like almost every other database on the face of the planet.
OpenVMS is the only operating system which truly clusters, many claim it, but only OpenVMS actually delivers it. The OS kernel understands the concept of a record along with the concepts of ACLs, rights identifiers, logicals, and user privileges. This enables the kernel to support and utilize a lock manager which has complete recovery built into the system boot process. This enables the cluster to have a distributed lock manager that works in unison with every lock manager on every node in the cluster. The lock managers provide the foundation for the distributed transaction manager which allows a transaction to span multiple databases, indexed files, journaled flat files, and message queues with full rollback/commit capabilities. It may not be sexy, but it is data Nirvana.
RDB allows you some choices with database creation. You can create a single file database, or you can spread your database across multiple drives and directories. Single file databases aren't speed demons, but they are pretty cool. You can lose the system disk which had RDB running, create a shiny new system disk with RDB on it, boot, and tell RDB to open your database. All of the information needed is stored in the single file database and its directory. You have complete recovery. It's a bit trickier with the multifiledisk version.
I'm bringing this up to you now to let you know that none of the databases we will be covering have that capability. Most PC users think about the cheapness or ease of use when choosing a database, not about what happens when things go bad. I guess that is kind of understandable since a lot of those people are using a computer with a single hard drive they rarely back up and never take the time to create a full image backup. When the spindle spits a platter onto the floor they are well and truly pooched.
Professionals, on the other hand, have made a living designing systems on midrange and mainframe computers. We've had to build in fault tolerance to levels which would make the average PC user faint if they tried to wrap their heads around it.
I'm on my soapbox now so that you will understand just how shaky things are. Every database we will be covering has the exact same flaw:
If you lose the disk containing the database engine directory and have to reinstall on a new drive, all of your data is toast. It doesn't matter how many drives you spread the data across or how religiously you backed it up with some third party tool.
The flaw is caused by the database engine keeping all of the schema, user, access, space, and proprietary information in the directory with the engine, or some subdirectory thereof. When you lose that information, you lose touch with the data, no matter where it is stored. MySQL, as well as most of the others, tries to hide this fact from users by making the default location for database creation the same directory tree as the database engine which puts it on the same spindle as the operating system. This way you lose it all together and don't realize the flaw. Of course, creating a production database on the operating system disk drive is a systems management nono of biblical proportions.
I found this out about PostgreSQL the hard way.
No binary backup made with any backup tool is of any use if it cannot restore your database engine directory to exactly what it needs to be so database information isn't
lost. The “backup method” used by all of these databases is for you to write a script (or use a provided utility) which dumps the data into a text only CSV (Comma Separated Value) file. Once you recreate your database from the script files (you did use a script instead of just keying the definitions in at the interactive prompt didn't you?) the CSV files are to be imported into their corresponding tables.
If you aren't an IT person by trade, that pitfall probably doesn't seem so bad. I am an IT person by trade. The first two versions of my expense tracking software used standalone indexed files which I kept in different directories for different years on a separate disk drive. I also backed that drive up religiously. I could easily recover from any situation prior to moving to an Open Source relational database. Now, in order to be able to recover, I have to quintuple my storage requirements. Once all of the data is entered and verified, I must export each table to a CSV and backup that expanded version along with the scripts to create and restore the CSV.
What I'm showing you will be good enough for your own personal development. It may even be good enough for a small company as long as you are religious about CSV export and backup. Some companies have been burned pretty bad by this frailty. I have seen some companies map the system boot disk (which contains the database engine and proprietary information) to a RAID 6 volume on their SAN so they could be protected from two spindles burning out at the same time.
Only you can decide what your data is worth. Don't make that decision after you've developed the application
It's difficult for a regular PC user or even the average Open Source developer to grasp just how critical this issue is in a production environment. If your database is currently approaching 1TB in size and is filled with binary data (mostly numbers, not much text) you will need 5TB of contiguous free disk space to create your CSV file. Right now the biggest SATA drive you can buy for your PC is 2TB so unless you link three of them together with a RAID controller, you cannot backup.
2.9 Main.cpp for What?
One of the larger issues I have with the “accepted” method of Qt development and its “one directory tree” philosophy is the concept that every application should have a main.cpp file. I find this in all of the books I look at and most of the online examples. It appears to be the will of the Qt programming community.
Let me be the first to tell you, this is a disaster waiting to happen!
Yes, I really wanted one sentence all by itself like that. You need to pay attention to this issue. If every project has to have a main.cpp and they are all custom for that project, you have a ticking time bomb on