Category Archives: howto

Sparse Files – Commands Overview

I was just checking which commands are supporting sparse files. I post hereafter a short overview of what I found out (working on Linux).

  • Create a sparse file of 20 GiB:
    dd if=/dev/zero of=foo bs=1 count=1 seek=20G

  • Check that a file is sparse:
    ls -alsh
    Compare the first versus second size column (the first one is the space taken on disk).

  • Copy a sparse file:
    cp foo bar
    Copy already detects and handles correctly sparse files.

  • Make a non-sparse file sparse (works only if it contains blocks filled with zeros):
    cp --sparse=always foo bar

  • Expand a sparse file to a non-sparse one:
    cp --sparse=never foo bar

  • Copy remotely a sparse file: scp does not support sparse files, it will expand them to non-sparse ones. Rsync does support them. Just use the -S or --sparse option. Example:
    rsync -vS foo root@someserver:/some/path/

How to Know if a File on Linux is Sparse?

A sparse file is a file which does not take more space on disk than needed. Such a file is usually used to store a partition image on disk, for instance with a virtualization solution like Xen.

It’s super easy, to know if a file is sparse or not. Just use the ‘s’ option of ls.

ls -alsh

will yield:

4.0K drwxr-xr-x 9 root root 4.0K 2010-03-23 18:22 .
4.0K drwxr-xr-x 23 root root 4.0K 2009-01-09 19:47 ..
12G -rw-r----- 1 root root 24G 2007-01-10 19:55 dompat.data
3.7G -rw-r--r-- 1 root root 7.1G 2009-01-06 21:09 dompat-hardy.sys
501M -rw-r----- 1 root root 501M 2007-01-07 16:40 dompat.swap

Where the first size column is the effective space taken on disk while the second size column is the max space of that file. We see that dompat.data is sparse, since its max size is 24 GB while it takes only 12 GB on disk.

References

  • More information on command which can handle sparse files in this article

GPGMail Compatible with Snow Leopard

I had posted that some times ago on Twitter. There is a new version of the excellent GPGMail plug-in for Mail (OS X). Grab it, if you haven’t already.

GPGMail for OS X 10.6.2

Just put it inside ~/Library/Mail/Bundles and restart Mail. (More information on GPGMail’s web page.)

Edit: This is not an official version of GPGMail. There is still no official version of GPGMail compatible with Snow Leopard.

NFS Automount with Snow Leopard

Before upgrading to Snow Leopard, I was using the NFS automount capability of Mac OS X. It was a handy way for my laptop to connect to my home NAS server (a linux box running Ubuntu).

Unfortunately I had to redo the NFS config after upgrading and I noticed that Directory Utility was not present anymore in Snow Leopard.

Here is what I found out after some research:

  • the NFS automount configuration is now done directly in Disk Utility (under File > NFS Mounts…) and not in Directory Utility like in Leopard (which does not exist anymore).
  • the configuration that worked for me was the following one (for accessing a read-write NFS share):
    Remote NFS URL: nfs://[server]/[path]
    Mount location: [path to local mount folder]
    Advanced Mount Parameters: -i,-s,-w=32768,-r=32768
    
  • for automount to reload its configuration, I had to run the following command:
    sudo automount -vc
  • That’s it. The NFS share should now be accessible.

    References:

  • http://discussions.apple.com/thread.jspa?threadID=2137675
  • Time Machine: Handy but Bitchy!

    I have been spending the last couple of days of my spare time performing a simple operation with my mac, yet slightly more complicated than expected: merge the two partitions of my hard drive into one. This was necessary since I did a mapping, quite usual on Linux, where my system was sitting on the first partition (50 GB) while my user data was on the second one (the rest of 300 GB). But this mapping was not making me happy, since the system partition was always almost full (Mac OS X is not well suited for a multi partition disk drive, I find).

    Although you can resize a live partition with Disk Utility (great feature, even if you booted with it), you cannot move the base position of a partition. You can merely resize it, if more space is available. So I had to back my second partition up for later restoring it. I went for Time Machine, since I was already using it for backing up my laptop.

    Time Machine Caveats

    Unfortunetely I noticed a couple of caveats during the restore operation:

  • When using the Time Machine restore utility shipped with the OS X install DVD, you can only restore the partition where the system is, but not other ones. This was quite of a surpise to me. Although you can choose on which partition you want to restore your system (the target, not the source).
  • Although your data is actually sitting on the Time Machine backup drive, you cannot use it directly (i.e. copy it back using the Finder or a shell). The reason for that is the ACLs that Leopard is putting on each and every file and folder to protect changing the Time Machine backup. In addition the ACL system (which is a parralel access control to the Unix one, which I did not know the existence of beforehand) is deeply flawed: you cannot reliably remove recursively the ACLs from a directory structure (you will still find files scattered within the structure having ACLs) and you even cannot remove the ACL permissions on symbolic links (this seems to be a bug of chmod on Leopard, although ACLs on a symlink do not seem to have any effect)
  • Solution for merging two partitions

    I finally found a solution to restore the second partition or to merge them: I had to fiddle directly with the Time Machine backup, copying manually the files I wanted from the second partition to the first, and did a restore of the first partition using Time Machine restore utility.

    Here are the details of the steps I took:

  • Plug my external drive where I have my Time Machine backup, open a terminal and sudo as root:
    sudo -s
  • Deactivate on that drive the ACL checks, so that I can modify directly the Time Machine backup:
    fsaclctl -p /Volumes/[backup drive] -d
  • Move the folders from the second partition to the first one with
    mv [from] [to]
  • Reactivate the ACL checks on the external drive:
    fsaclctl -p /Volumes/[backup drive] -e
  • Do the restore of the first partition with the Time Machine restore utility (located on the Leopard install DVD).
  • This worked for me. You should now have a merged partition on your drive, the restore utility having removed the ACLs during the operation. Phew! This was not a straight forward action! I still cannot believe that Apple did not take into account that people can have more than 1 partition on their drive (I might still have missed the way to do it, did not find it so far though).

    Links/Info

  • Macosxhint: reconnect Time Machine backup after a drive swap
  • Inspect the ACLs of your file with ls:
    ls -le

    For the record, this will yield the following listing when ACLs are present (pay attention to the lines starting with ’0: ‘ and ’1: ‘:
    drwxrwxr-x@ 138 root admin 4692 Sep 13 2008 Applications
    0: group:everyone deny add_file,delete,add_subdirectory,delete_child,writeattr,writeextattr,chown
    1: group:everyone deny delete
    drwxr-xr-x@ 2 pat staff 68 May 9 22:12 DeveloperSDK3
    0: group:everyone deny add_file,delete,add_subdirectory,delete_child,writeattr,writeextattr,chown
    drwxr-xr-x@ 5 pat staff 170 Oct 23 2008 VirtualBox
    0: group:everyone deny add_file,delete,add_subdirectory,delete_child,writeattr,writeextattr,chown

    You can also do that recursively using the following command:
    ls -lateR > filelist

  • You can remove ACLs on a file with this command (not working properly for a recursive operation with the option -R)
    chmod -a# 0 [file]
    where 0 is the ACL entry to remove.

  • Annoying ‘screen’ Backspace Problem

    Already a long time that this issue is bugging me. The great ‘screen’ unix command has a problem, at least on ubuntu: the backspace key produces the same effect than delete (first char on the right of the cursor gets deleted). Very annoying. Hopefully the solution is easy. Edit your file ‘~/.bashrc’ and add the following line:

    alias screen='TERM=screen screen'

    Pidgin Status Update from Twitter

    Just figured out how to get Twitter update the pidgin status (e.g. GTalk, Jabber, ICQ, MSN, etc.). For that, you need to install a pidgin plug-in written in Perl. Here is a small howto about it (as the Perl stuff was a bit tricky):

  • Download and install the multi-network Pidgin client
  • Install Perl. On Windows (yeah, I know. It’s at work. ;) ), install Active Perl version 5.8.8.X. Don’t take the last one, the 5.10.0, as Pidgin does not support it yet.
  • Install the Package XML:XPath, either using the CPAN console (install XML:XPath) or using the package manager of Active Perl.
  • Restart Pidgin and check that Perl is enabled. (Help -> About -> at the bottom of the page, you should see ‘Perl: Enabled’).
  • Download the twitter plugin and put it in the plugin folder of Pidgin. (For instance on Windows in ‘C:\Program Files\Pidgin\plugins’)
  • Configure the plug-in and enjoy!
  • Tomato on WRT54GL: LED shows WIFI status

    Something I wanted to do since ages. My wireless router WRT54GL running the Tomato firmware has a button on the front, which can be used to toggle WIFI on and off. The router has also a rather visible LED indicator, behind the button, that is unfortunately not used to show the WIFI status. (Originally, the wifi status is just shown by a tiny LED, that is just visible when watching closely the device).

    It’s easy to fix that. Using the web interface, go to the ‘Administration’ > ‘Buttons / LED’ page. Replace the custom script by the following code:


    # status: 1: wifi on, 0: wifi off
    status=$(wl -a eth1 dump | grep associated | cut -d " " -f 2);
    # toggle wifi & amber led
    wl -a eth1 $([ $status -eq 0 ] && echo up || echo down)
    led amber $([ $status -eq 0 ] && echo on || echo off)

    Then change the setting for ‘0 – 2 Seconds’ to ‘Run Custom Script’. That’s it. Now when we press the button, the wireless is toggled and the LED switches itself accordingly.

    Tested with Tomato 1.19.

    Green Server

    Last week I bought a consumption measurement device for 20 bucks. I wanted to know how much current is needed for various electronic devices I use at home, some of them on a 24/7 basis. I have got a home server running Linux, that I use as a NAS, print server, slim server (for streaming mp3 files to an mp3 device) and for experimenting a bit with Xen and Ubuntu. I have got also some other devices like a wireless access point (WRT54GL running Tomato), a cable modem, an external backup hard disk, a b&W laser printer and a few other electronic devices.

    Of course, one of the most energy hungry device I have is the server, which is always switched on. I was expecting a consumption of about 60 W, because I tried to choose efficient components, some of which were listed on a review regarding a green computer consuming 54 W when idle. My hardware configuration is the following:

    • Processor AMD 64 x2 EE 3800+, dual core and energy efficient version (which is actually quite cheap)
    • Micro-ATX Mainboard M2NPV-VM with AM2 socket
    • 2 GB of RAM (DDR2 PC4300)
    • 2 Seagate 7200.8 250 GB hard disks used as a software RAID-1 array
    • Power supply ElanVital 500W, ATX, with 80% efficiency.
    • Plextor Plexwriter DVD burner

    So I shut downed my server, plugged it in the measurement device and booted it. The value I could read on the device, after the reboot was finished and the server was idle, was higher than expected: 85 W. I was not very happy with that value and began to try to optimize it.

    • I first tried to remove the DVD burner, but I got just 2 W of reduction, which was not good enough.
    • Then yesterday I tried to install the last stable release of Ubuntu, which is Ubuntu 7.10 gutsy 32 bits version (I add previously Ubuntu feisty with a XEN enabled kernel). I was pleased to see that the server was only consuming about 68 W (DVD burner included). I then tried to re-install XEN and after rebooting saw that I was again at 85 W. So the XEN kernel had different options enabled that were not optimized for a good power management.
    • I got back to the stock kernel and tried to activate the automatic spin-down of the hard disks. I did this using the command
      hdparm -S 60 /dev/sda
      hdparm -S 60 /dev/sdb

      which spins down the two disks after being idle for 5 minutes. I was able to grab a few watts and get down to 60 W, which is not too bad. I added the two commands in /etc/rc.local so that it gets executed automatically on boot.

    • I then tried to activate the automatic CPU frequency scaling, so that when the processor has a low usage, its frequency gets automatically reduced, hence saving some energy as well. I was getting some problems using the module powernow-k8, so I updated the bios to the latest version (1201), which resolved that issue. I could then install correctly the cpu scaling daemon. I just had to run
      apt-get install cpufreqd

      and then edit the file /etc/default/cpufreqd and insert “powernow-k8″ in the option CPUFREQ_CPU_MODULE (you can check which module you need to use here). With these two operations, the BIOS upgrade and the cpu frequency scaling, I was able to grab a few watts, getting down, when the server is idle, to about 53 W, which is already a lot better than the 85 W I was getting without tuning.

    The options I might consider now are the following:

    • I could reduce significantly the power consumption if I automatically switch off the server every night. I am using it also to do a distant backup of another computer, but I could setup the BIOS to switch it on every night to do that, if it is not already the case. The only thing I would need to do is just to switch the server on, whenever I want to print something or hear mp3′s, which is not too bad. The server would then switch itself off at night, after doing the backup.
    • If I want to use XEN, I should compile my own patched kernel (for XEN) and tune the power management options to get the same consumption as on the stock kernel.
    • I could try to tune further the kernel options, which I am not sure that I am going to do, as it is much more convenient to use the stock kernel than to use a self compiled one (e.g. for updates).
    • I could try to find a way to cut off the consumption of some other electrical devices, which are most of the time not used or in standby mode, but do consume power anyway. Like for instance the laser printer (standby 14 W), the external backup hard disk Seagate 500 GB 7200.10 (standby 12 W), the wireless access point WRT54GL (always on, 14 W) or the cable modem (always on, 12 W). Not sure how I can do that. Of course I could plug some of these devices in a multi socket with a switch, but I would need to turn it on each time I want to access the internet using my laptop and turn it off afterwards (in addition it’s not compatible with my automatic distant backup). Not very elegant. I guess I will have to think about that issue.

    Additional links: