Currently having a couple of issues with laptop and server. Hmm.. sratching head, thinking cap on, etc..
Problem 1 – laptop swap On my laptop, I recently resized my root partition (lv) and removed/recreated my swap partition (also a logical volume). On my new swap, I used mkswap, added an entry in fstab and turned it on. All rudimentary stuff.
But when the system came to using it, it hung. No response in X whatsoever, although there seemed to be disk polling going on, suggesting the kernel was still operational. I couldn’t flip to another console or SSH in to find out, though.
I created a new partition directly on the disk, not using LVM, and made that a swap too. Same set up procedure as before, then activated it. This time, when the system needed to swap, it did – as you would expect it to. Bizarre. I can’t think why this might be happening, apart from something going wrong through the device mapper, maybe.
Problem 2 – server dump The second problem I’m having is backing up the server using dump. In short, when I dumped out a level 0 backup, not all my files were copied. Strangely, also, directory sizes on the tape, and when restored, seemed padded/boundary-aligned – e.g. 4kb, 8kb or 16kb. I’m trying to solve this one too, and am using tar in the meantime (which, if testing proves positive, may stick with).
The full title of this blog should really be ‘SELinux is preventing mysqld (mysqld_t) “search” to ./tmp (public_content_rw_t)’ as that is the problem I’ve been having with CentOS recently (and hence my searches on the web for a solution).
The cause of the problem
I use SugarCRM for customer and project management data – and very good it is too! (Gratuitous plug – I can help your company install and use this fine software ). Except that recently, when listing my Accounts within Sugar, I would not see all of the account context. Only the account data itself would be displayed and none of the subpanels/links.
The query to retrieve more data was failing, with this error message displayed in the browser window: mysqld: Can't create/write to file '/tmp/#08y2jw' (Errcode: 13) In my system log (/var/log/messages), I also got multiple SELinux errors like this: Oct 13 09:07:50 server setroubleshoot: SELinux is preventing mysqld (mysqld_t) "read" to ./tmp (public_content_rw_t). For complete SELinux messages. run sealert -l 1762c478-f3a2-4eeb-be09-bd3dc037d945 Clearly, the reason for “Errcode: 13″ was due to SELinux.
Incidentally. if you have seen a similar error on your web site, but with (Errcode: 28) instead, this is likely due to shortage of disk space. A great way of determining operating system errors like this, is to use ‘PError’, thus: # perror 28 OS error code 28: No space left on device # perror 13 OS error code 13: Permission denied
So there we are – two distinct and different issues.
With SELinux, resolving the permission issue can be difficult. By issuing # sealert -l 1762c478-f3a2-4eeb-be09-bd3dc037d945, as suggested above, I got the following output (trimmed and highlighted for clarity):
Summary: SELinux is preventing mysqld (mysqld_t) “search” to ./tmp (public_content_rw_t). Allowing Access: Sometimes labeling problems can cause SELinux denials. You could try to restore the default system file context for ./tmp, restorecon -v ‘./tmp’ Additional Information: Source Context root:system_r:mysqld_t Target Context system_u:object_r:public_content_rw_t
First things first: issuing # restorecon -v './tmp' didn’t fix it for me. I was also surprised to see that the path to /tmp was relative to the current working directory, so I tried a slightly modified # restorecon -v '/tmp', but to no avail. After restarting mysqld, the problem persisted: MySQL was simply being refused access to /tmp. Somewhere, a policy is disallowing this.
It’s a mistake to assume the the source context and target context should be the same; they don’t have to be, as it’s entirely policy-driven. I made bold those aspects (the file Type) above to highlight this incorrect assumption (that I previously held).
Find and fix a policy?
Although finding the troublesome policy and analysing it is a Good Thing, it’s also time-consuming and requires significant knowledge of SELinux, chiefly to avoid creating security holes. A better way, I found, was simply to relocate where mysqld tries to store temporary data.
Thanks to Surachart Opun’s blog, I learned that you can specify a new location for temporary files. In /etc/my.cnf, add or edit the following: [mysqld] tmpdir=/tmp # # e.g. tmpdir=/var/lib/mysql/tmp
Now do the legwork to set up the directory properly:
First, create directory with appropriate permissions # cd /var/lib/mysql # mkdir tmp # chown mysql:mysql tmp # chmod 1750 tmp
Now set the SELinux context up: # chcon --reference /var/lib/mysql tmp
and make the SELinuiux context permanent: # semanage fcontext -a -t mysql_db_t "/var/lib/mysql/tmp(/.*)?"
Finally, restart mysql: # service mysqld restart
Closing thoughts: optimisation
The methods above fixed the particular problem I was having. They didn’t, however, actually pinpoint the cause. This is one of the good things about Linux and SELinux in particular: you are forced to rethink what the system is doing and work out a solution that sits within the predefined security context – or learn how to write SELinux policies. Personally, I prefer the former
There is an additional benefit to the solution above – namely, optimisation. Because we have specified the security context with semanage, we are free to mount an external file system and use that instead for MySQL’s temporary files. In other words, we can maintain the security but increase the performance. One such filesystem could be tmpfs. tmpfs is actually a RAM Disk, uses a fixed amount of RAM to provide file storage. It is much quicker than an on-disk filesystem and thus perfectly optimised for storing temporary, caching data. There are many resources about tmpfs on the web. A good introduction to tmpfs can be at Planet Admon.
That’s right. Your old tat (or, you could say, my old tat) is just about as worthless to everyone else as it is to me. I’ve spent ages on ebay and sold almost nothing. And what I have sold, I sold for 99p.
Give to charity instead, that’s what I should have done! Pah!