Category: Software

Posts about either the running, compiling, or writing of software.

  • My beets Configuration (Nov. 2016 Edition)

    This is mostly for my own convenience. I recently rebuilt a host for managing my beets library, and these are the packages (both deb and pip) that I needed to install for my particular beets config to work. And since that’s not really very helpful to anyone else without my beets config, I’ve included that as well.

    This should work for both ubuntu trusty and xenial.

    Requirements for beets:

    $ sudo apt-get install python-dev python-pip python-gi libchromaprint-tools imagemagick mp3val flac lame flac gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0 plugins-ugly gir1.2-gstreamer-1.0 libxml2-dev libxslt1-dev zlib1g-dev
    $ sudo pip install beets pyacoustid discogs-client pylast requests bs4 isodate lxml

    I use two plugins not included by default:
    bandcamp
    rymgenre

    Beets config:

    ############################################################################
    ## Beets Configuration file.
    ## ~./config/beets/config.yaml
    #############################################################################
    
    ### Global Options
    library: ~/.config/beets/library.blb
    directory: /mnt/music/
    pluginpath: ~/.config/beets/plugins
    ignore: .AppleDouble ._* *~ .DS_Store
    per_disc_numbering: true
    threaded: yes
    
    # Plugins
    plugins: [
    # autotagger extentions
    chroma,
    discogs,
    # metadata plugins
    acousticbrainz,
    embedart,
    fetchart,
    ftintitle,
    lastgenre,
    mbsync,
    replaygain,
    scrub,
    # path format plugins
    bucket,
    inline,
    the,
    # interoperability plugins
    badfiles,
    # misc plugins
    missing,
    info,
    # other plugins
    bandcamp
    ]
    
    # Import Options
    import:
    copy: true
    move: false
    write: true
    delete: false
    resume: ask
    incremental: false
    quiet_fallback: skip
    none_rec_fallback: skip
    timid: false
    languages: en
    log: ~/beets-import.log
    
    # Path options
    paths:
    # Albums/A/ASCI Artist Name, The/[YEAR] ASCI Album Name, The [EP]/01 - Track Name.mp3
    default: 'Albums/%bucket{%upper{%left{%the{%asciify{$albumartist}},1}}}/%the{%asciify{$albumartist}}/[%if{$year,$year,0000}] %asciify{$album} %aunique{albumartist album year, albumtype label catalognum albumdisambig}/%if{$multidisc,$disc-}$track - %asciify{$title}'
    # Singles/ASCII Artist Name, The - Track Name.mp3
    singleton: 'Singles/%%the{%asciify{$artist}} - %asciify{$title}'
    # Compilations/[YEAR] ASCI Compilation Name, The/01-01 - Track Name.mp3
    comp: 'Compilations/[%if{$year,$year,0000}] %the{%asciify{$album}%if{%aunique, %aunique{albumartist album year, albumtype label catalognum albumdisambig}}}/%if{$multidisc,$disc-}$track - %asciify{$title}'
    # Sountracks/[YEAR] ASCI Soundtrack Name, The/01 - Track Name.mp3
    albumtype:soundtrack: 'Soundtracks/[%if{$year,$year,0000}] %the{%asciify{$album}%if{%aunique, %aunique{albumartist album year, albumtype label catalognum albumdisambig}}}/%if{$multidisc,$disk-}$track - %asciify{$title}'
    
    ### Plugin Options
    
    # Inline plugin multidisc template
    item_fields:
    multidisc: 1 if disctotal > 1 else 0
    
    # Collects all special characters into single bucket
    bucket:
    bucket_alpha:
    - _
    - A
    - B
    - C
    - D
    - E
    - F
    - G
    - H
    - I
    - J
    - K
    - L
    - M
    - N
    - O
    - P
    - Q
    - R
    - S
    - T
    - U
    - V
    - W
    - X
    - Y
    - Z
    bucket_alpha_regex:
    '_': ^[^A-Z]
    
    # Per album genres selected from a custom list
    # My genre-tree.yaml is ever so slightly custom as well
    # I found per-album genres in last.fm could be very misleading.
    lastgenre:
    canonical: ~/.config/beets/genres-tree.yaml
    whitelist: ~/.config/beets/genres.txt
    min_weight: 20
    source: artist
    fallback: 'Unknown'
    
    # rymgenre doesn't run on import, so I use it as a backup
    # for when lastgenre is giving me garbage results.
    rymgenre:
    classes: primary
    depth: node
    
    # Fetch fresh album art for new imports
    fetchart:
    sources: coverart itunes amazon albumart
    store_source: yes
    
    # I want the option to scrub, but don't feel the need to scrub automatically
    scrub:
    auto: no
    
    # Gstreamer is a pain, but still the correct backend
    replaygain:
    backend: gstreamer
    overwrite: yes
    
    acoustid:
    apikey: <API_KEY>
    
    echonest:
    apikey: <API_KEY>
    auto: yes
  • Fixing Recently Added In XBMC After Re-Importing Music

    I did some rejiggering of how XBMC accesses my NAS which required re-importing all my media. The video library sets the ‘date added’ field to the file modification date of a video, which means the ‘Recently Added’ view of my movies is correct, even though everything only just got re-imported. But the music library takes the literal approach and sorts by the order in which albums are scanned. After re-importing my recently added list was just a reverse alphabetical sort by artist. The video library behaviour seems to me to be the intuitively correct behaviour, and I care because when I’m at home I primarily use the recently added view to listen to music.

    After a bit of digging around I determined that the music library sets an idAlbum field for each album which is an integer that increments upwards as albums are scanned, and this is how the recently added view sorts itself. So the solution was to blow the music library away and re-import (again), but this time forcing the order in which XBMC scans the music collection.

    Step 1) Get an album list sorted by modification date

    My music collection is well organized (I use beets) and all of my albums are in directories that have the release year in square brackets, so the following command gave my a list of all my albums sorted by modification date:

    $ find . -type d -name *[* -printf '%T@ %p\n' | sort -k 1n
    1407992390.0000000000 ./Albums/T/Talking Heads/[1987] More Songs About Buildings and Food
    1407992431.0000000000 ./Albums/T/Talking Heads/[1983] Remain in Light
    1407992516.0000000000 ./Albums/T/Talking Heads/[1984] Stop Making Sense
    1407992594.0000000000 ./Albums/T/Talking Heads/[1986] True Stories

    Except that when I reviewed the output I remembered that I’d recently done some updates to the tags on a bunch of albums and so the modification dates were way too recent on a lot of them. I was almost ready to start getting bummed out.

    The Real Step 1) Get an album list sorted by creation date

    POSIX doesn’t require that a filesystem keep track of file creation date and the XFS filesystem my music resides on dutifully doesn’t bother to track it. However, my music collection is synced with rsync every night from a server I colo downtown, and that host uses an ext4 filesystem which does in fact track the creation date. It’s not obvious, because the standard stat command does not return a creation date on linux but debugfs can.

    Now, here’s where things get complicated, mostly because by this time it was late and while I’m sure there’s a better way to do everything that follows my brain was starting to get sleepy. Please feel free to send me more efficient regexes, or versions of scripts that cut out extra steps, or whatever. I’ll update and give credit.

    debugfs is easier to work with if you’re just working with inodes so first generate a list of directories and their inodes:

    $ find . -type d -name *[* -exec ls -di {} \; > inodelist
    $ head -n 5 inodelist
    26763273 ./Albums/A/Andy Stott/[2012] Luxury Problems
    26763279 ./Albums/A/Arca/[2013] &amp;&amp;&amp;&amp;&amp;
    27189257 ./Albums/A/Arcade Fire/[2010] The Suburbs
    27189258 ./Albums/A/Arcade Fire/[2004] Funeral
    27189256 ./Albums/A/Arcade Fire/[2007] Neon Bible

    I found it easiest to generate a separate list of timestamps and then merge the two files (again, I’m sure there’s a better way to do this but I didn’t bother for a one-time use process). First we need a short script that can determine the crtime of the directory:

    #!/bin/bash
    inode=$1
    fs="/dev/md0"
    crtime=$(sudo debugfs -R 'stat <'"${inode}"'>' "${fs}" 2>/dev/null | grep crtime | cut -d ' ' -f 2 | cut -d ':' -f 1)
    printf "%d\n" "${crtime}"#!/bin/bash<br>inode=$1<br>fs="/dev/md0"<br>crtime=$(sudo debugfs -R 'stat &lt;'"${inode}"'>' "${fs}" 2>/dev/null | grep crtime | cut -d ' ' -f 2 | cut -d ':' -f 1)<br>printf "%d\n" "${crtime}"

    And use that to get yourself a nice sorted list:

    $ for i in `cat inodelist | cut -d ' ' -f 1`; do ./get_crtime.sh $i >> crtimelist; done
    $ paste -d " " crtimelist inodelist | sort | cut -f2- -d '/' > albumlist
    $ head -n 5 albumlist
    Albums/P/Prince/[1987] Dirty Mind
    Albums/P/Public Enemy/[1988] It Takes a Nation of Millions to Hold Us Back
    Albums/M/mum/[2002] Finally We Are No One
    Albums/A/Amon Tobin/[1997] Bricolage
    Albums/B/Blonde Redhead/[2000] Melody of Certain Damaged Lemons

    Congratulations. You now have a list of albums sorted by their creation date. I generated the file with paths relative to the root of my music directory, because it’s easier to work with that way. You’ll clearly need to play with the cut options to get something that works for your particular library of music.

    Step 2) Scan your directories in the correct order

    XBMC has a JSON-RPC API, which includes the helpful AudioLibrary.Scan method, and even a wiki page on how to use it to trigger scans, though we need to pass the optional directory parameter:

    $ curl --data-binary '{ "jsonrpc": "2.0", "method": "AudioLibrary.Scan", "params": { "directory": "/mnt/music/Albums/F/FKA twigs/[2013] EP2" }, "id": "1"}' -H 'content-type: application/json;' http://localhost:8080/jsonrpc
    {"id":"1","jsonrpc":"2.0","result":"OK"}$ curl --data-binary '{ "jsonrpc": "2.0", "method": "AudioLibrary.Scan", "params": { "directory": "/mnt/music/Albums/F/FKA twigs/[2013] EP2" }, "id": "1"}' -H 'content-type: application/json;' http://localhost:8080/jsonrpc<br>{"id":"1","jsonrpc":"2.0","result":"OK"}

    In testing I found that a lot of the jsonrpc calls were silently failing – I’d get the OK result for all of them, but watching the logs I could see that not all of the requests were actually being executed. In my first test I used a while loop and fired over 700 requests in a couple seconds and saw that only about 30% of them executed (I didn’t even bother to check if they were in the correct order). I watched the import notification on screen when I imported a single album and saw it took roughly ten seconds to import the album. With that in mind for the second test I waited 20 seconds between each request and I still saw only 80-90% of them executed. I doubt it’s because the previous request was still running because then I’d expect the first test to have only resulted in a single (maybe two) successfully executed requests.

    By this time it’s really late and I didn’t care enough to troubleshoot further – I decided to just brute force the matter:

    $ while read album; do
    	echo $album
    	curl -s --data-binary '{ "jsonrpc": "2.0", "method": "AudioLibrary.Scan", "params": { "directory": "/mnt/music/'"$album"'" }, "id": "1"}' -H 'content-type: application/json;' http://localhost:8080/jsonrpc > /dev/null
    	sleep 20
    	curl -s --data-binary '{ "jsonrpc": "2.0", "method": "AudioLibrary.Scan", "params": { "directory": "/mnt/music/'"$album"'" }, "id": "1"}' -H 'content-type: application/json;' http://localhost:8080/jsonrpc > /dev/null
    	sleep 20
    	curl -s --data-binary '{ "jsonrpc": "2.0", "method": "AudioLibrary.Scan", "params": { "directory": "/mnt/music/'"$album"'" }, "id": "1"}' -H 'content-type: application/json;' http://localhost:8080/jsonrpc > /dev/null
    	sleep 20
    done < albumlist

    It’s about as elegant as a sledgehammer, but it works. The extra calls to the RPC method are redundant at worst since it does no harm to scan the directory repeatedly but at least you can be reasonably sure the album will get scanned successfully. Run it in screen overnight and when you return in the morning you should have all of your albums imported, in the order in which you acquired them.

    UPDATE: Even the sledgehammer wasn’t enough for three albums. Evidently it’s easy for the AudioLibrary.Scan method call to be skipped. So I blew away the music library again and this time used a script to scan each album, this time checking the XBMC logs. You need to enable debug logging for this script to work, but I’ll leave enabling that as an exercise for the user since there’s a few ways to do it. Anyway, here’s a better solution for importing:

    #!/bin/bash
    
    while read album; do
      while true; do
        IMPORTED=`grep -F "$album" ~/.xbmc/temp/xbmc.log`
        if [ $? == 0 ]; then
          break
        fi
        echo $album
        curl -s --data-binary '{ "jsonrpc": "2.0", "method": "AudioLibrary.Scan", "params": { "directory": "/mnt/music/'"$album"'" }, "id": "1"}' -H 'content-type: application/json;' http://localhost:8080/jsonrpc > /dev/null
        sleep 15
      done
    done < $1
    $ import_albums.sh albumlist#!/bin/bash<br><br>while read album; do<br>  while true; do<br>    IMPORTED=`grep -F "$album" ~/.xbmc/temp/xbmc.log`<br>    if [ $? == 0 ]; then<br>      break<br>    fi<br>    echo $album<br>    curl -s --data-binary '{ "jsonrpc": "2.0", "method": "AudioLibrary.Scan", "params": { "directory": "/mnt/music/'"$album"'" }, "id": "1"}' -H 'content-type: application/json;' http://localhost:8080/jsonrpc > /dev/null<br>    sleep 15<br>  done<br>done &lt; $1<br>$ import_albums.sh albumlist

    Areas for improvement:
    1) That crap with get_crtime.sh and the steps around it, in particular generating a separate crtimelist file and merging it back in. Maybe something that can be called directly from find -exec?
    2) The sledgehammer import. Perhaps check the log after making a request and seeing if the DoScan event shows up there before moving on?

  • Netatalk 3.1.3 For Debian Wheezy

    I’ve found the netatalk available in Wheezy (version 2.2.2) to be flakey for a while now. Even Jessie only has 2.2.5, while the latest from netatalk.sourceforge.net is 3.1.3. My specific problem was intermittently failing time machine backups from the Macs in my house. Netatalk helpfully includes directions on compiling netatalk from source on Wheezy here, but I don’t like to have all those dev packages on my fileserver, which means spinning up a build host, creating a deb, yada yada yada. Anyway, no point in going to all that bother and not sharing it.

    First make sure you remove the old version: # apt-get remove –purge netatalk

    Install the pre-requisites: # apt-get install libdbus-glib-1-2 libmysqlclient18 mysql-common libcrack2 avahi-daemon

    Download this: netatalk_3.1.3-1_amd64.deb

    And install it: # dpkg -i netatalk_3.1.3-1_amd64.deb

    Netatalk v3 uses an entirely new config format, so you’ll have to recreate your config files (hence why we used –purge above). It’s actually way easier to configure now though, so don’t fret too much.

    Big important note: I’m providing this purely as a convenience. I accept no liability and provide no guarantees. Use at your own risk.

  • NFS Exports And XFS’s inode64 Mount Option

    I recently turned up a new RAID array and plopped an XFS filesystem down on it. I didn’t bother setting any specific tunings when I created the filesystem. However I couldn’t for the life of me export any subdirectories from the volume over NFS. Local access was fine and I could export via netatalk and samba.

    On the server I saw messages like this in the logs:

    Feb 14 13:08:43 monolith rpc.mountd[3092]: authenticated mount request from 192.168.1.50:1003 for /mnt/music (/mnt/music)
    Feb 14 13:08:57 monolith rpc.mountd[3092]: authenticated mount request from 192.168.1.50:1002 for /opt/music (/opt/music)
    Feb 14 13:15:19 monolith rpc.mountd[3092]: authenticated mount request from 192.168.1.20:717 for /mnt/music (/mnt/music)
    Feb 14 13:15:20 monolith rpc.mountd[3092]: authenticated mount request from 192.168.1.20:1001 for /mnt/music (/mnt/music)
    Feb 14 13:15:22 monolith rpc.mountd[3092]: authenticated mount request from 192.168.1.20:1002 for /mnt/music (/mnt/music)
    Feb 14 13:15:26 monolith rpc.mountd[3092]: authenticated mount request from 192.168.1.20:801 for /mnt/music (/mnt/music)
    Feb 14 13:15:34 monolith rpc.mountd[3092]: authenticated mount request from 192.168.1.20:967 for /mnt/music (/mnt/music)
    Feb 14 13:15:44 monolith rpc.mountd[3092]: authenticated mount request from 192.168.1.20:794 for /mnt/music (/mnt/music)
    Feb 14 13:15:54 monolith rpc.mountd[3092]: authenticated mount request from 192.168.1.20:855 for /mnt/music (/mnt/music)
    Feb 14 13:16:04 monolith rpc.mountd[3092]: authenticated mount request from 192.168.1.20:863 for /mnt/music (/mnt/music)
    Feb 14 13:16:14 monolith rpc.mountd[3092]: authenticated mount request from 192.168.1.20:932 for /mnt/music (/mnt/music)
    Feb 14 13:16:24 monolith rpc.mountd[3092]: authenticated mount request from 192.168.1.20:830 for /mnt/music (/mnt/music)

    On the client I would get two different behaviours, depending on whether it was NFSv4 or NFSv3 that was being used. With NFSv4 it would mount the directory, but any attempt to read from it would give a ‘Stale NFS handle’ error:

    root:~# mount -t nfs -v 192.168.1.10:/mnt/music /mnt/
    mount.nfs: timeout set for Fri Feb 14 16:49:39 2014
    mount.nfs: trying text-based options 'vers=4,addr=192.168.1.10,clientaddr=192.168.1.20'
    root:~# ls /mnt/
    ls: cannot open directory /mnt/: Stale NFS file handle

    With NFSv3 it would just time out, although if I used the -v switch I could see that it was timing out because it was getting ‘Stale NFS handle’ errors during the attempt to mount the directory:

    root:~# mount -t nfs -o nfsvers=3 -v 192.168.1.10:/mnt/music /mnt/
    mount.nfs: timeout set for Fri Feb 14 16:51:29 2014
    mount.nfs: trying text-based options 'nfsvers=3,addr=192.168.1.10'
    mount.nfs: prog 100003, trying vers=3, prot=6
    mount.nfs: trying 192.168.1.10 prog 100003 vers 3 prot TCP port 2049
    mount.nfs: prog 100005, trying vers=3, prot=17
    &amp;lt;SNIP&amp;gt;
    mount.nfs: prog 100003, trying vers=3, prot=6
    mount.nfs: trying 192.168.1.10 prog 100003 vers 3 prot TCP port 2049
    mount.nfs: prog 100005, trying vers=3, prot=17
    mount.nfs: trying 192.168.1.10 prog 100005 vers 3 prot UDP port 60925
    mount.nfs: mount(2): Stale NFS file handle
    mount.nfs: Connection timed out

    After much frustrated googling I finally stumbled across the answer: inode64. By default NFS still exports 32bit inodes, and XFS by default uses 64bit inodes. There are a few workarounds that worked for me:

    1) Export the root of the filesystem and not subdirectories. If you’re in a high trust environment this might be a reasonable solution. However you might not want to export everything on your filesystem, in which case you should look at option 2.

    2) You can also manually set the fsid option for each exported subdirectory like this:

    /mnt/music 192.168.1.0/24(rw,async,wdelay,insecure,no_root_squash,no_subtree_check,fsid=1)
    /mnt/pictures 192.168.1.0/24(rw,async,wdelay,insecure,no_root_squash,no_subtree_check,fsid=2)

    I’ve tested both of the above with the following clients: Ubuntu 13.10, FreeBSD 9.2, and Mac OS X 10.9. I don’t actually have any older clients to test with at the moment.

  • FreeBSD 9.2 on Jetway NF9H-525

    I wanted a new system to run as my router, so I set out to find a small system with at least three gigabit NICs. I settled on a Jetway NF9H-525 (also referenced as a NF9HQL-525), which is a mini-ITX board geared for networking applications with four onboard gigabit NICs and a dual core atom 1.8Ghz processor.  I also purchased an LGX MC500 case, two gigabytes of RAM, and pulled a drive out of a recently retired laptop. Bottom line: everything worked great and I would recommend it.

    The board has four onboard realtek gigabit NICs (RTL8111/8168B) which use the re driver. They’re not the beloved Intel em NICs, but they’ll do. Especially for the types of applications this board is likely to be used for – like my home networking. Same deal with the intel atom processor. The system can be a bit louder than I would have really liked, but the fan speed stepping does work so it’s only loud when it’s under a bit of load.

    I’m not going to bother including any build notes, mostly because I didn’t encounter any gotchas or oddities during the installation or configuration of the system. I read reports online that the NICs require a recent version of FreeBSD, but I didn’t have any issues with 9.2-RELEASE.