MusicBrainz Server Installation Log

This log records my efforts to get my own MusicBrainz test server. Unfortunately, this log is incomplete - at some point I managed to delete my log.

The story so far was basically that I'd downloaded the server software and was going through its INSTALL file. At some point I decided that this wasn't going to work, and that I needed to work from the CVS version. My first task was to get the remaining dependencies installed...

As always, everything I do is made 10 times more complicated by the fact that I insist on using Debian package management for everything.

Oh, by the way, the trigger for all of this was Winfried's complaint that MusicBrainz couldn't perform transcoding on imports from FreeDB, making imports of Russian CDs basically impossible. As it happened, the MB developers fixed that before I had a chance to work on it. So I moved on to fixing a few little bugs.

2004-08-09 21:10

Reading the INSTALL file, I have a bunch of perl modules to get through:

One way or another, I've got all except:

I don't know what's up with MIME-Base64. I do a full grep through all the sources and can't find any reference to it. I can't find it on CPAN. I'll leave it for the moment.

Net::Amazon was one of the ones I prepared earlier. I have no idea how any more, but it does appear to work when I install it with dpkg.

2004-08-10 23:11

RDFStore was also one I prepared earlier. Installed.

For String::Unicode::Similiarity, I remember that this was in some intermediate state. It's possible that this has been fixed by now, so I'm going to CVS update just in case. For that, I need the instructions. Hmm, they still have that annoying typo in which they talk about musicbrainz_sshanoncvs instead of musicbrainz_anoncvs :-(

OK, no change to the status of this module, so I'm going to have to do some work. There are two bits to get - the original module, and the patch. The patch is no problem. But the module itself seems to be missing from CPAN! It might have changed name (maybe dropping the "Unicode"). I'm going to scour the internet to see if I can find the actual version 0.2.

Ah, no, I was misreading the installation instructions. It's been renamed from String::Similarity. Unfortunately, the version on CPAN is version 1.0. The patch is for 0.2. So I'm going to have to find the old version.

Well, that wasn't too hard; I found it on the backperl page for the author. Now to make a Debian package out of it. I should make a mental note that it'd be good to update the patch so it works with the 1.0 version.

I need to go right back and read the Debian New Maintainers' Guide. Sigh. I think I'll do this in two steps - first get the original version running, then do the patched version.

mexon@aeon:~/musicbrainz/similarity$ tar xzf String-Similarity-0.02.tar.gz
mexon@aeon:~/musicbrainz/similarity$ mv String-Similarity-0.02 libstring-similarity-perl-0.02
mexon@aeon:~/musicbrainz/similarity$ cd libstring-similarity-perl-0.02/
mexon@aeon:~/musicbrainz/similarity/libstring-similarity-perl-0.02$ dh_make -e 17132580@exon.dyndns.org -f ../String-Similarity-0.02.tar.gz

Type of package: single binary, multiple binary, library, or kernel module?
 [s/m/l/k] s

Maintainer name : Matthew Exon
Email-Address   : 17132580@exon.dyndns.org
Date            : Wed, 11 Aug 2004 00:19:35 +0200
Package Name    : libstring-similarity-perl
Version         : 0.02
Type of Package : Single
Hit  to confirm:
Currently there is no top level Makefile. This may require additional tuning.
Done. Please edit the files in the debian/ subdirectory now. You should also
check that the libstring-similarity-perl Makefiles install into $DESTDIR and not in / .

    

2004-08-16 13:57

First job, the control file. I changed the section to "interpreters", and added descriptions based on the perldoc. I didn't change the dependencies at all.

The copyright file was tricky because I don't know exactly what to put in the "Copyright:" field. IANAL. But it's GPL, so I guess no-one will sue.

I decided not to do anything with the changelog.

For the rules, I'm just copying what's in rdfstore. How did I come up with that stuff? I wish I hadn't deleted my old log :-(

Just a docs file, and now it's time to build it. Here's what happens:

mexon@aeon:~/musicbrainz/similarity/libstring-similarity-perl-0.02$ dpkg-buildpackage -rfakeroot
dpkg-buildpackage: source package is libstring-similarity-perl
dpkg-buildpackage: source version is 0.02-1
dpkg-buildpackage: source maintainer is Matthew Exon <17132580@exon.dyndns.org>
dpkg-buildpackage: host architecture is i386
 fakeroot debian/rules clean
dh_testdir
dh_testroot
rm -f build-stamp configure-stamp
/usr/bin/make clean
make[1]: Entering directory `/home/mexon/musicbrainz/similarity/libstring-similarity-perl-0.02'
make[1]: *** No rule to make target `clean'.  Stop.
make[1]: Leaving directory `/home/mexon/musicbrainz/similarity/libstring-similarity-perl-0.02'
make: [clean] Error 2 (ignored)
dh_clean
 dpkg-source -b libstring-similarity-perl-0.02
dpkg-source: building libstring-similarity-perl using existing libstring-similarity-perl_0.02.orig.tar.gz
dpkg-source: building libstring-similarity-perl in libstring-similarity-perl_0.02-1.diff.gz
dpkg-source: building libstring-similarity-perl in libstring-similarity-perl_0.02-1.dsc
 debian/rules build
dh_testdir
/usr/bin/perl Makefile.PL INSTALLDIRS=vendor
Checking if your kit is complete...
Looks good
Writing Makefile for String::Similarity
/usr/bin/make
make[1]: Entering directory `/home/mexon/musicbrainz/similarity/libstring-similarity-perl-0.02'
cp Similarity.pm blib/lib/String/Similarity.pm
/usr/bin/perl /usr/share/perl/5.8/ExtUtils/xsubpp  -typemap /usr/share/perl/5.8/ExtUtils/typemap  Similarity.xs > Similarity.xsc && mv Similarity.xsc Similarity.c
cc -c   -D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBIAN -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2   -DVERSION=\"0.02\" -DXS_VERSION=\"0.02\" -fPIC "-I/usr/lib/perl/5.8/CORE"   Similarity.c
Running Mkbootstrap for String::Similarity ()
chmod 644 Similarity.bs
rm -f blib/arch/auto/String/Similarity/Similarity.so
LD_RUN_PATH="" cc  -shared -L/usr/local/lib Similarity.o  -o blib/arch/auto/String/Similarity/Similarity.so
chmod 755 blib/arch/auto/String/Similarity/Similarity.so
cp Similarity.bs blib/arch/auto/String/Similarity/Similarity.bs
chmod 644 blib/arch/auto/String/Similarity/Similarity.bs
Manifying blib/man3/String::Similarity.3pm
make[1]: Leaving directory `/home/mexon/musicbrainz/similarity/libstring-similarity-perl-0.02'
touch build-stamp
 fakeroot debian/rules binary
dh_testdir
dh_testroot
dh_clean -k
dh_installdirs
/usr/bin/make install DESTDIR=/home/mexon/musicbrainz/similarity/libstring-similarity-perl-0.02/debian/libstring-similarity-perl
make[1]: Entering directory `/home/mexon/musicbrainz/similarity/libstring-similarity-perl-0.02'
Installing /home/mexon/musicbrainz/similarity/libstring-similarity-perl-0.02/debian/libstring-similarity-perl/usr/lib/perl5/auto/String/Similarity/Similarity.soInstalling /home/mexon/musicbrainz/similarity/libstring-similarity-perl-0.02/debian/libstring-similarity-perl/usr/lib/perl5/auto/String/Similarity/Similarity.bsFiles found in blib/arch: installing files in blib/lib into architecture dependent library tree
Installing /home/mexon/musicbrainz/similarity/libstring-similarity-perl-0.02/debian/libstring-similarity-perl/usr/lib/perl5/String/Similarity.pm
Installing /home/mexon/musicbrainz/similarity/libstring-similarity-perl-0.02/debian/libstring-similarity-perl/usr/share/man/man3/String::Similarity.3pm
make[1]: Leaving directory `/home/mexon/musicbrainz/similarity/libstring-similarity-perl-0.02'
dh_testdir
dh_testroot
dh_installchangelogs Changes
dh_installdocs
dh_installexamples t
dh_perl
dh_installdeb
dh_shlibdeps
dh_gencontrol
dpkg-gencontrol: warning: unknown substitution variable ${misc:Depends}
dh_md5sums
dh_builddeb
dpkg-deb: building package `libstring-similarity-perl' in `../libstring-similarity-perl_0.02-1_i386.deb'.
 signfile libstring-similarity-perl_0.02-1.dsc
gpg: skipped `Matthew Exon <17132580@exon.dyndns.org>': secret key not availablegpg: [stdin]: clearsign failed: secret key not available
    

I don't remember seeing that failed to sign thing before. Should be able to easily make a key for signing though. So I just did that, and here's how it ends now:

dpkg-deb: building package `libstring-similarity-perl' in `../libstring-similarity-perl_0.02-1_i386.deb'.
 signfile libstring-similarity-perl_0.02-1.dsc

You need a passphrase to unlock the secret key for
user: "Matthew Exon <17132580@exon.dyndns.org>"
1024-bit DSA key, ID 1EB5504C, created 2004-08-16


 dpkg-genchanges
dpkg-genchanges: including full source code in upload
 signfile libstring-similarity-perl_0.02-1_i386.changes

You need a passphrase to unlock the secret key for
user: "Matthew Exon <17132580@exon.dyndns.org>"
1024-bit DSA key, ID 1EB5504C, created 2004-08-16


dpkg-buildpackage: full upload (original source is included)
    

So that all looks good, and it installed OK. Next, I'll have to do the patched version. I'll do this as a separate package with a different name, but I'll do it in the same directory as the "untouched" version. First I extract it, then apply the patch:

mexon@aeon:~/musicbrainz/similarity$ tar xzf String-Similarity-0.02.tar.gz
mexon@aeon:~/musicbrainz/similarity$ patch -p0 < String-Unicode-Similarity.patch
patching file String-Similarity-0.02/Changes
patching file String-Similarity-0.02/fstrcmp.c
patching file String-Similarity-0.02/fstrcmp.h
patching file String-Similarity-0.02/Makefile.PL
patching file String-Similarity-0.02/README
patching file String-Similarity-0.02/Similarity.pm
patching file String-Similarity-0.02/Similarity.xs
patching file String-Similarity-0.02/t/00_load.t
mexon@aeon:~/musicbrainz/similarity$ mv String-Similarity-0.02 libstring-unicode-similarity-perl-0.02
mexon@aeon:~/musicbrainz/similarity$ cd libstring-unicode-similarity-perl-0.02
mexon@aeon:~/musicbrainz/similarity/libstring-unicode-similarity-perl-0.02$ dh_make -e 17132580@exon.dyndns.org -f ../String-Similarity-0.02.tar.gz

Type of package: single binary, multiple binary, library, or kernel module?
 [s/m/l/k] s

Maintainer name : Matthew Exon
Email-Address   : 17132580@exon.dyndns.org
Date            : Mon, 16 Aug 2004 14:40:46 +0200
Package Name    : libstring-unicode-similarity-perl
Version         : 0.02
Type of Package : Single
Hit  to confirm:
Currently there is no top level Makefile. This may require additional tuning.
Done. Please edit the files in the debian/ subdirectory now. You should also
check that the libstring-unicode-similarity-perl Makefiles install into $DE
    

So I did all the same debian files, built the package, and installed it. But the actual module didn't make it into the package! I guess something must still be referring to String::Similarity.

2004-08-17 17:41

OK, I think the problem was this line in rules, that I'd copied over from the non-unicode version:

        $(MAKE) install DESTDIR=$(CURDIR)/debian/libstring-similarity-perl
    

So I think I've got to do this from scratch, without cheating.

OK, that worked much better. I've now got String::Unicode::Similarity installed. Next, on to UUID. First pass failed with these errors:

cc -c   -D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBIAN -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2   -DVERSION=\"0.02\" -DXS_VERSION=\"0.02\" -fPIC "-I/usr/lib/perl/5.8/CORE"   UUID.c
UUID.xs:5:23: uuid/uuid.h: No such file or directory
UUID.xs: In function `do_generate':
UUID.xs:13: error: `uuid_t' undeclared (first use in this function)
UUID.xs:13: error: (Each undeclared identifier is reported only once
UUID.xs:13: error: for each function it appears in.)
UUID.xs:13: error: parse error before "uuid"
UUID.xs:14: error: `uuid' undeclared (first use in this function)
UUID.xs: In function `do_unparse':
UUID.xs:20: error: `uuid_t' undeclared (first use in this function)
UUID.xs:20: error: parse error before "uuid"
UUID.xs: In function `do_parse':
UUID.xs:29: error: `uuid_t' undeclared (first use in this function)
UUID.xs:29: error: parse error before "uuid"
UUID.xs:33: error: `uuid' undeclared (first use in this function)
make[1]: *** [UUID.o] Error 1
make[1]: Leaving directory `/home/mexon/musicbrainz/uuid/libuuid-perl-0.02'
make: *** [build-stamp] Error 2
    

This is because I didn't have the package uuid-dev installed. Once I did that, it worked fine.

String::ShellQuote is in the package libstring-shellquote-perl, so I just have to install that.

Tie::STDERR was straightforward. I'm getting pretty used to making these perl packages now.

Text::Unaccent is already installed. The INSTALL file is worried that the normal version is 1.01 and it needs 1.05. However, the version that is installed seems to be 1.07, so I think that should be OK. Hmm - no wait. It does need to be patched after all. More work then :-(. The INSTALL file really should be updated to get rid of the "latest versoin ... is ... 1.01".

2004-10-07 02:24

OK, I downloaded Text::Unaccent from it's download area. I removed the old libtext-unaccent-perl package to make room for the new one.

I'm now building the package. Are you supposed to use latin1 or UTF8 in package descriptions? Just curious. One thing to note. You can't do the dh_make thing, then patch, then build the package:

dpkg-source: building libtext-unaccent-perl using existing libtext-unaccent-perl_1.07.orig.tar.gz
dpkg-source: building libtext-unaccent-perl in libtext-unaccent-perl_1.07-1.diff.gz
dpkg-source: cannot represent change to t/unac.t: binary file contents changed
dpkg-source: building libtext-unaccent-perl in libtext-unaccent-perl_1.07-1.dsc
dpkg-source: unrepresentable changes to source
    

I tried to cheat and do it that way round, because I forgot to apply the patch before setting things up, but no go. I'll have to do it the proper way.

Hmm, just tried that but it didn't make any difference. After looking on the web for that error message, I found a message that suggests using the command fakeroot debian/rules binary. I have no idea what that does, but I just tried it, and it seemed to work.

So I think that's the last of the perl! I can check with the script admin/depend.pl. However, running that, I discovered a thing about the location of the MusicBrainz runtime data. This is defined by the variable MB_SERVER_ROOT. It seems I previously set this to /usr/share/mb_server. I assume that's OK. Anyway, this means that depend.pl expects to find perl in /usr/share/mb_server/cgi-bin/perl. I'm going to set up a temporary symlink there to bootstrap myself into things. OK. So now it runs, but there's no output. I guess that's good?

2004-10-09 01:00

I have to decide where to put the musicbrainz data in my filesystem. The recommended areas, in /home/httpd, definitely aren't appropriate since I've got server software that belongs under /usr, and runtime data that belongs under /var. So I think it'll break down like this:

I considered using /var/locks/musicbrainz, but decided that I don't really know what it's for. BTW, I have actually been reading the LHS. Look at me - I'm making design decisions! It's almost like being a real engineer again!

2004-10-11 00:30

Let's edit some files, as described in section 4 of INSTALL.

mexon@aeon:~/musicbrainz/mb_server/cgi-bin$ diff DBDefs.pm.dist DBDefs.pm
33c33
< sub MB_SERVER_ROOT    { "/home/httpd/musicbrainz/mb_server" }
---
> sub MB_SERVER_ROOT    { "/usr/share/musicbrainz/mb_server" }
37c37
< sub DATA_IMPORT_DIR   { "/home/httpd/musicbrainz/data-import" }
---
> sub DATA_IMPORT_DIR   { "/var/lib/musicbrainz/data-import" }
41,42c41,42
< sub LOCK_DIR          { "/home/httpd/musicbrainz/locks" }
< sub SESSION_DIR               { "/home/httpd/musicbrainz/sessions" }
---
> sub LOCK_DIR          { "/var/lib/musicbrainz/locks" }
> sub SESSION_DIR               { "/var/lib/musicbrainz/sessions" }
45c45
< sub MASON_DIR         { "/home/httpd/musicbrainz/mason" }
---
> sub MASON_DIR         { "/usr/share/musicbrainz/mason" }
112c112
< sub SMTP_SECRET_CHECKSUM { "" }
---
> sub SMTP_SECRET_CHECKSUM { "2529629470321146" }
mexon@aeon:~/musicbrainz/mb_server/admin$ diff startup.pl.dist startup.pl
8c8
< use lib "/home/httpd/musicbrainz/mb_server/cgi-bin";
---
> use lib "/usr/share/musicbrainz/mb_server/cgi-bin";
mexon@aeon:~/musicbrainz/mb_server/admin$ diff vh_httpd.conf.dist vh_httpd.conf
3c3
<     DocumentRoot "/home/httpd/musicbrainz/mb_server/htdocs"
---
>     DocumentRoot "/usr/share/musicbrainz/mb_server/htdocs"
5c5
<     Alias /cgi-bin/ "/home/httpd/musicbrainz/mb_server/cgi-bin/"
---
>     Alias /cgi-bin/ "/usr/share/musicbrainz/mb_server/cgi-bin/"
9,10c9,10
<     ServerName www.musicbrainz.example.com
<     ServerAlias rdf.musicbrainz.example.com
---
>     ServerName www.musicbrainz.exon.dyndns.org
>     ServerAlias rdf.musicbrainz.exon.dyndns.org
15c15
<     <Directory /home/httpd/musicbrainz/mb_server/htdocs>
---
>     <Directory /usr/share/musicbrainz/mb_server/htdocs>
30c30
<     PerlRequire /home/httpd/musicbrainz/mb_server/admin/startup.pl
---
>     PerlRequire /usr/share/musicbrainz/mb_server/admin/startup.pl
    

2004-10-12 23:58

Two... minutes... to miiiidniiiight...

The configuration of Apache 2 under Debian works by simply putting the new configuration chunk into /etc/apache2/conf.d. This is really convenient. So I've just put it in there.

I then copied the htdocs directory into /usr/share/musicbrainz/mb_server. I don't yet know what to do with startup.pl.

I then restarted Apache, and went to http://www.musicbrainz.exon.dyndns.org/, but it didn't work like it's supposed to. Damn.

2004-10-14 17:55

OK, I think that doesn't work, and I realised that nothing has yet set up the database. So I'm going to undo what I did and work on the assumption that I need to use the live data. This is a pain because it probably means I'll have to check out the musicbrainz stuff from scratch. Boo.

Incidentally I've just discovered that Apache 2 isn't actually running. And it isn't present in /usr/sbin like it's supposed to. I'm investigating that. Hmm. Turns out it wasn't actually installed. Not sure now if I should be using Apache 1 or 2. I'm going with 2 for the moment, then see how it works out and possibly switch.

One thing I've come across is that all the scripts in admin reference /home/httpd/musicbrainz. I really don't want stuff to go there, so I'm going to substitute them all with the real path to perl. Hmm, there's a lot of them. Boo!

2004-10-15 19:58

Another problem. Under Debian, by default, only the user "postgres" is allowed access to the database. But the InitDb.pl script expects to be able to do that. I'm going to have to explore options for fixing that.

2004-10-26 08:37

I have to think about the database permissions problem from the point of view of the superuser installing this as a Debian package. That means they'll be running as root. I think the best thing to do is su to postgres and do it that way. Let's see what happens.

OK, this is what's happening for the moment.

postgres@aeon:/home/mexon/musicbrainz/mb_server$ perl ./admin/InitDb.pl --createdb --echo --import ../data/mbdump*.tar.bz2
Tue Oct 26 08:40:14 2004 : InitDb.pl starting
Tue Oct 26 08:40:15 2004 : Creating database 'musicbrainz_db'
Tue Oct 26 08:40:20 2004 : Creating tables ... (CreateTables.sql)
Tue Oct 26 08:40:20 2004 : psql: FATAL:  IDENT authentication failed for user "musicbrainz_user"
Error during sql/CreateTables.sql at ./admin/InitDb.pl line 65.
Tue Oct 26 08:40:20 2004 : InitDb.pl failed
postgres@aeon:/home/mexon/musicbrainz/mb_server$ dropdb musicbrainz_db DROP DATABASE
    

The postgres user is fine, but it still wants to connect as musicbrainz_user for the majority of stuff, and that account doesn't exist. I should probably do what the INSTALL file suggests and map the web user to musicbrainz_user. However, I'm not sure how to do that.

Using ps, I can find that the user that the Apache runs as is www-data. So I think just doing exactly what the INSTALL file suggests is the best way to go.

I just spent a long, frustrating time trying to get to the stage where I can connect as musicbrainz_user. I used exactly what was specified in the INSTALL file, but it failed to make it clear how important it is that you get the pg_hba.conf file in exactly the right order. It's very frustrating because until that and the pg_ident.conf file are set up exactly right, all you get is an endless sequence of "IDENT authentication failed" messages, which don't tell you anything. In my case I had forgotten to change www-user to www-data in pg_ident.conf.

I eventually added this to pg_hba.conf:

aeon:/etc/postgresql# diff pg_hba.conf.old pg_hba.conf
86a87,90
>
> # MusicBrainz account information
> local musicbrainz_db musicbrainz_user ident mb_map
>
    

...and this to pg_ident.conf:

aeon:/etc/postgresql# diff pg_ident.conf.old pg_ident.conf
35a36,40
>
> # MusicBrainz account information
> mb_map www-data musicbrainz_user
> mb_map postgres musicbrainz_user
>
    

The INSTALL file suggests only adding the www-data line, but then there'd be no single user that can run the whole InitDb.pl script.

So I'm running it now and it's taking a long, long time. Excellent! Here's some abbreviated logs:

postgres@aeon:/home/mexon/musicbrainz/mb_server$ perl ./admin/InitDb.pl --createdb --echo --import ../data/mbdump*.tar.bz2
Tue Oct 26 09:29:47 2004 : InitDb.pl starting
Tue Oct 26 09:29:49 2004 : Creating database 'musicbrainz_db'
Tue Oct 26 09:29:53 2004 : Creating tables ... (CreateTables.sql)
Tue Oct 26 09:29:53 2004 : BEGIN;
Tue Oct 26 09:29:53 2004 : BEGIN
Tue Oct 26 09:29:53 2004 : CREATE TABLE album
Tue Oct 26 09:29:53 2004 : (
Tue Oct 26 09:29:53 2004 :     id                  SERIAL,
Tue Oct 26 09:29:53 2004 :     artist              INTEGER NOT NULL,
Tue Oct 26 09:29:53 2004 :     name                VARCHAR(255) NOT NULL,
Tue Oct 26 09:29:53 2004 :     gid                 CHAR(36) NOT NULL,
Tue Oct 26 09:29:53 2004 :     modpending          INTEGER DEFAULT 0,
Tue Oct 26 09:29:53 2004 :     attributes          INTEGER[] DEFAULT '{0}',
Tue Oct 26 09:29:53 2004 :     page                INTEGER NOT NULL
Tue Oct 26 09:29:53 2004 : );
Tue Oct 26 09:29:53 2004 : psql:/home/mexon/musicbrainz/mb_server/admin/sql/CreateTables.sql:28: NOTICE:  CREATE TABLE will create implicit sequence "album_id_seq" for "serial" column "album.id"
Tue Oct 26 09:29:54 2004 : CREATE TABLE
Tue Oct 26 09:29:54 2004 : CREATE TABLE album_amazon_asin
Tue Oct 26 09:29:54 2004 : (
Tue Oct 26 09:29:54 2004 :     album               INTEGER NOT NULL,
Tue Oct 26 09:29:54 2004 :     asin                CHAR(10),
Tue Oct 26 09:29:54 2004 :     coverarturl         VARCHAR(255),
Tue Oct 26 09:29:54 2004 :     lastupdate          TIMESTAMP WITH TIME ZONE DEFAULT NOW()
Tue Oct 26 09:29:54 2004 : );
    

...

Tue Oct 26 09:29:56 2004 : COMMIT
Tue Oct 26 09:29:57 2004 : tar -C /tmp/MBImport-ZZOq4INq --bzip2 -xvf ../data/mbdump-artistrelation.tar.bz2
TIMESTAMP
COPYING
REPLICATION_SEQUENCE
SCHEMA_SEQUENCE
mbdump/artist_relation
Tue Oct 26 09:29:58 2004 : tar -C /tmp/MBImport-gnrMQJoH --bzip2 -xvf ../data/mbdump-closedmoderation.tar.bz2
TIMESTAMP
COPYING
REPLICATION_SEQUENCE
SCHEMA_SEQUENCE
mbdump/moderation_closed
    

...

mbdump/replication_control
mbdump/track
mbdump/trm
mbdump/trmjoin
Tue Oct 26 09:41:23 2004 : Validating snapshot
Tue Oct 26 09:41:23 2004 : Snapshot timestamp is 2004-10-13 07:44:05.196194+00
Tue Oct 26 09:41:23 2004 : Schema sequence mismatch - codebase is 2, snapshot files are 3

Failed to import dataset.
Tue Oct 26 09:41:28 2004 : InitDb.pl failed
    

Oops. Back to the drawing board I guess. I'll sync up the codebase, and download fresh snapshots, and hope that that works.

Hmm. I do the following:

mexon@aeon:~/musicbrainz/mb_server$ export CVSROOT=:ext:cvs@cvs.musicbrainz.org:/var/cvs
mexon@aeon:~/musicbrainz/mb_server$ export CVS_RSH=~/musicbrainz/musicbrainz_anoncvs
mexon@aeon:~/musicbrainz/mb_server$ cvs status -v INSTALL | grep RELEASE_ | sort
        RELEASE_20030121                (revision: 1.30)
        RELEASE_20030302                (revision: 1.30)
...
        RELEASE_20041009-BRANCH         (branch: 1.57.4)
        RELEASE_20041009-ROOT           (revision: 1.57)
   Sticky Tag:          RELEASE_20041009-BRANCH (branch: 1.57.4)
    

That looks like there hasn't been a release of the software since I last checked it out. But the error message implies that my data dumps are later than the code. That's weird. I'm checking out another copy of the code from tip to see if maybe CVS just doesn't print tags later than the sticky tag it's synced to.

Nope, that gives the same tag list. And looking at the live MusicBrainz site, they have "server version: 2004-10-10" at the bottom of each page, so that tallies. Maybe I just try downloading a new set of data dumps. I originally downloaded them 2004-10-13 01:01, but maybe there was just some delay going on such that the dumps are actually from before the new version was rolled out.

mexon@aeon:~/musicbrainz/data$ wget -r ftp://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/20041023-080322
    

Nope, that didn't work either :-(

2004-10-27 14:45

I figured out why it didn't work when I went through the musicbrainz-devel mails from the last few weeks. This email form Robert Kaye:

No, I think that is correct. I had to do that the other day too -- I think we forgot to bump up the schema number. I fixed that now -- cvs HEAD has schema 3 now.

So I think I just have to sync to HEAD. OK, I did that, and I reran everything, and it's going well so far. So far being about two hours at this stage. I'm gonna leave it for a bit...

2004-10-27 18:03

OK, it's done. Here's the tail of the output:

Wed Oct 27 17:49:04 2004 : CREATE TRIGGER
Wed Oct 27 17:49:04 2004 : CREATE TRIGGER a_del_album_amazon_asin AFTER DELETE ON album_amazon_asin
Wed Oct 27 17:49:04 2004 :     FOR EACH ROW EXECUTE PROCEDURE a_del_album_amazon_asin();
Wed Oct 27 17:49:04 2004 : CREATE TRIGGER
Wed Oct 27 17:49:04 2004 : Optimizing database ...
WARNING:  skipping "pg_shadow" --- only table or database owner can vacuum it
WARNING:  skipping "pg_database" --- only table or database owner can vacuum it
WARNING:  skipping "pg_group" --- only table or database owner can vacuum it
VACUUM
Wed Oct 27 17:58:28 2004 : Initialized and imported data into the database.
Wed Oct 27 17:58:28 2004 : InitDb.pl succeeded
    

2004-11-02

I've just tried installing MusicBrainz and running the site. I copied the vh_httpd.conf file into /etc/apache2/conf.d, and the htdocs and cgi-bin directories into /usr/share/musicbrainz/mb_server. Then I restarted memcached and Apache. When I restarted Apache, I got this error:

aeon:/etc/init.d# ./apache2 start
Starting web server: Apache2Syntax error on line 18 of /etc/apache2/conf.d/vh_httpd.conf:
Invalid command 'PerlHandler', perhaps mis-spelled or defined by a module not included in the server configuration
    

I think this means that I don't have modperl properly installed.

2004-11-03

After thinking about things, I decided the error message above probably doesn't indicate a problem with modperl, but rather the fact that I should be using Apache 1 instead of 2. So I changed my install appropriately, copied the vh_httpd.conf into /etc/apache-perl/conf.d and tried again. This time, I got:

Configuration syntax error detected. Not reloading.

Syntax error on line 28 of /etc/apache-perl/conf.d/vh_httpd.conf:
Can't locate Apache/DBI.pm in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.8.4 /usr/local/share/perl/5.8.4 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.8 /usr/share/perl/5.8 /usr/local/lib/site_perl . /etc/apache-perl/ /etc/apache-perl/lib/perl) at (eval 5) line 3.
    

So I'm installing libapache-dbi-perl. We'll see if that makes a difference. Well, yes. Here's the error:

aeon:/etc/init.d# ./apache-perl start
Configuration syntax error detected. Not reloading.

[Wed Nov  3 14:12:51 2004] [error] Can't locate /usr/share/musicbrainz/mb_server/admin/startup.pl in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.8.4 /usr/local/share/perl/5.8.4 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.8 /usr/share/perl/5.8 /usr/local/lib/site_perl . /etc/apache-perl/ /etc/apache-perl/lib/perl) at (eval 7) line 1.\n
Syntax error on line 30 of /etc/apache-perl/conf.d/vh_httpd.conf:
Can't locate /usr/share/musicbrainz/mb_server/admin/startup.pl in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.8.4 /usr/local/share/perl/5.8.4 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.8 /usr/share/perl/5.8 /usr/local/lib/site_perl . /etc/apache-perl/ /etc/apache-perl/lib/perl) at (eval 7) line 1.
    

I attempted to solve this by copying the admin directory from mb_server to /usr/share/musicbrainz/mb_server. The result was this:

aeon:/etc/init.d# ./apache-perl start
Configuration syntax error detected. Not reloading.

Loading Apache::Request at runtime.  You could increase shared memory between Apache processes by preloading it in your httpd.conf or handler.pl file
[Wed Nov  3 14:16:35 2004] [error] Can't locate Apache/Request.pm in @INC (@INC contains: /usr/share/musicbrainz/mb_server/cgi-bin /etc/perl /usr/local/lib/perl/5.8.4 /usr/local/share/perl/5.8.4 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.8 /usr/share/perl/5.8 /usr/local/lib/site_perl . /etc/apache-perl/ /etc/apache-perl/lib/perl) at /usr/share/perl5/HTML/Mason/ApacheHandler.pm line 664.\nCompilation failed in require at /usr/share/musicbrainz/mb_server/admin/startup.pl line 77.\nBEGIN failed--compilation aborted at /usr/share/musicbrainz/mb_server/admin/startup.pl line 77.\nCompilation failed in require at (eval 7) line 1.\n
Syntax error on line 30 of /etc/apache-perl/conf.d/vh_httpd.conf:
Can't locate Apache/Request.pm in @INC (@INC contains: /usr/share/musicbrainz/mb_server/cgi-bin /etc/perl /usr/local/lib/perl/5.8.4 /usr/local/share/perl/5.8.4 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.8 /usr/share/perl/5.8 /usr/local/lib/site_perl . /etc/apache-perl/ /etc/apache-perl/lib/perl) at /usr/share/perl5/HTML/Mason/ApacheHandler.pm line 664.
Compilation failed in require at /usr/share/musicbrainz/mb_server/admin/startup.pl line 77.
BEGIN failed--compilation aborted at /usr/share/musicbrainz/mb_server/admin/startup.pl line 77.
Compilation failed in require at (eval 7) line 1.
    

OK then, I'm installing libapache-request-perl. And it still doesn't work, and this one doesn't look too good:

aeon:/etc/init.d# ./apache-perl start
Configuration syntax error detected. Not reloading.

/usr/sbin/apache-perlctl: line 80: 10351 Segmentation fault      $HTTPD -t
    

2005-01-04 04:47

OK, so it's been a while since my last entry. Sue me.

Using the old gdb on Apache, it seems pretty clear who the culprit is:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 1077754336 (LWP 15509)]
0x406b5a20 in XS_Text__Unaccent_unac_string ()
   from /usr/lib/perl5/auto/Text/Unaccent/Unaccent.so
    

I think at this stage I should just go for the latest CVS version and work from there. Incidentally, they've fixed their typo. Good show. Hmm. Looking at the INSTALL file, it seems that nothing's changed in Text::Unaccent. So I might have to do some digging and try to debug it myself.

I've just added some silly debugging statements to Unaccent.xs. I've just got to figure out how to compile, install, and run it...

OK, I did this by doing a make clean, then building a new Makefile with perl Makefile.PL, then a cheeky little make, and finally running fakeroot debian/rules binary.

Well blow me down:

About to call unac_string
About to call unac_string
Syntax OK

Program exited normally.
(gdb)
aeon:/home/mexon/musicbrainz/textunaccent# /etc/init.d/apache-perl start
About to call unac_string
About to call unac_string
Starting web server: apache-perl.
aeon:/home/mexon/musicbrainz/textunaccent#
    

Well, I guess I'll remove the debugging and try again. Yep, still appears to work. Apache now runs. Yay. I guess that means I've got to populate the bloody database again now. Damn. I'm going to have to delete some stuff to make room :-(

2005-02-04 1:58

Well, not only does Apache run, but it's now serving up a neat little copy of MusicBrainz! I can see it on http://aeon.exon.dyndns.org, but not http://musicbrainz.exon.dyndns.org. Good enough. Next thing I want to do is figure out how to manually edit the database using Postgres. Unfortunately, it's been a very long time since I've used psql. Hmm, don't seem to be able to log in from a mexon prompt, so I guess I'll just do it as postgres. OK, I've done that.

I think the first step for making my "import Russian CDs" dialog is trying to do an import and seeing what the data looks like. I just tried to create an account on my MusicBrainz server, and got this error:

error:   	Could not open file (/var/lib/musicbrainz/locks/Apache-Session-f6198d1cdc93c2bd19ef3e11bdd93e80.lock) for writing: Permission denied at /usr/share/perl5/Apache/Session/Lock/File.pm line 62.
context:  	
...  	
58:  	
59:  	my $LockDirectory = $session->{args}->{LockDirectory} ||
60:  	$Apache::Session::Lock::File::LockDirectory;
61:  	
62:  	open($fh, "+>".$LockDirectory."/Apache-Session-".$session->{data}->{_session_id}.".lock") || die "Could not open file (".$LockDirectory."/Apache-Session-".$session->{data}->{_session_id}.".lock) for writing: $!";
63:  	
64:  	$self->{fh} = $fh;
65:  	$self->{opened} = 1;
66:  	}
...  	
code stack:  	/usr/share/perl5/Apache/Session/Lock/File.pm:62
/usr/share/perl5/Apache/Session.pm:584
/usr/share/perl5/Apache/Session.pm:512
/usr/share/perl5/Apache/Session/File.pm:40
/usr/share/perl5/Apache/Session/Lock/File.pm:62
    

I have to admit, everything does seem to be owned by root. BTW, I just tried adding musicbrainz and musicbrainz.exon.dyndns.org as aliases of Aeon in Charly's hosts file. That does seem to work, for some reason, in lynx, but not with Firefox. Oh well. I just tried a little chown -R www-data.www-data /var/lib/musicbrainz/. After a little confusion, this does appear to have worked.

Now, my next step is to import one of my Russian CDs from FreeDB. I don't know the TOC key, so I'm going to have to use Grip and a packet sniffer to watch for the ID. OK, done that, and found the culprit.

OK, just tried that in MusicBrainz, and: bugger! Some bastard has gone and implemented my feature already! Of all the nerve! Hmm, wonder what I'll do now?

2005-02-06 21:37

Well, I guess I could try implementing a simple bugfix, for example this one about slashes in titles.

A bit of info about the database. I'm currently using it by logging in as postgres, and then connecting with psql musicbrainz_db. Once there, the \d command is the way to navigate around all the tables. I was able to find out who I'd logged in as with select * from moderator order by membersince desc limit 10;. At first I was shocked to learn that all the moderator details are available for download, until I saw that the password is mb for all of them. I don't know if this is just a leftover from a previous version of MusicBrainz, and the passwords are kept somewhere else, or if this is just something that happens in the dump for testing purposes. The passwords from my test accounts are stored in plaintext. Which is lame. Might as well encrypt them, even if you have to store the stuff in the database.

Well, having tried to reproduce the bug, I've found that this too appears to be fixed in the code I've got. It's still possible that this was in fact broken in recent releases of the code, so maybe I should update CVS to tip, check it out again, and if it still works move onto another bug.

Just playing around with the system a bit. I found that I wasn't allowed to edit the database because I didn't have an email address - it's been deleted from the database dumps to protect my privacy. But I can't put it back because the reply-to address is on the real musicbrainz. That's really going to have to be fixed for development to work properly. However, in the meantime I found that I could just add my email address to the database manually with psql.

For this bug about entering albums with no tracks, I wasn't able to reproduce the problem. That is, it did allow me to enter "0" as the number of tracks, but then I had numerous problems, probably due to screens expecting a non-zero number of tracks. For example:

You've chosen to enter a "single artist" album, but looking at the track names it seems that maybe this should be a "various artists" album. (This message appears because all track names contain either "-" or "/"). What do you want to do?

I guess there should be a check that the number of tracks is greater than zero. Hmm, I'm even allowed to enter "-6". That's probably not very good. OK, let's have a go at fixing this. Of course, it's probably already been fixed in the latest version, but this should be a good learning exercise regardless. If I get it working, I can then examine the latest version to see if it's still relevant.

Now, the URL for this page is addalbum.html. Doing a locate for that, I find that there's one in the mason directory, and one in the mb_server directory. I don't know if they're different or not... Well, they certainly are very different. Actually, the former seems like a pre-compiled version of the latter. That means, if I edit it, I might have to manually remove the one in the mason directory. We'll see.

The Mason stuff seems relatively straightforward. Of course, the file I want is not addalbum.html, but the one that should be checking the results, cdi/enter.html. BTW, I discovered that I can cheerfully enter 3000 as the number of tracks. I then have to wait for the page with 3000 tracks in the form to load :-)

OK, that was quite simple to do. I add a javascript script in mb_server/htdocs/scripts/addalbumvalidate.js that looks like this:

function validate(f)
{ 
    if ((f.tracks.value < 1) || (f.tracks.value > 99))
    {
        alert("Please enter a number of tracks between 1 and 99");
        return false;
    }

    return true;
}
    

I then just add a call to this script in addalbum.html like so:

<script type="text/javascript" src="/scripts/addalbumvalidate.js">
</script>
                        onsubmit="return validate(this);"
    

So, having done that, I'm now going to sync to tip and do it again. After dinner...

2004-02-06 23:35

In order to upgrade:

BTW, I think that my mason directory should be in /var instead of /usr. In vh_httpd.conf I've gone from www.musicbrainz... to just musicbrainz..., because that's my alias. Maybe I'll be able to use that address now.

Well, that didn't work:

aeon:/usr/share/musicbrainz/mb_server# /etc/init.d/apache-perl restart
Configuration syntax error detected. Not reloading.

[Sun Feb  6 23:49:52 2005] [error] Can't locate Time/ParseDate.pm in @INC (@INC contains: /usr/share/musicbrainz/mb_server/cgi-bin /etc/perl /usr/local/lib/perl/5.8.4 /usr/local/share/perl/5.8.4 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.8 /usr/share/perl/5.8 /usr/local/lib/site_perl . /etc/apache-perl/ /etc/apache-perl/lib/perl) at /usr/share/musicbrainz/mb_server/admin/startup.pl line 30.\nCompilation failed in require at (eval 7) line 1.\n
Syntax error on line 32 of /etc/apache-perl/conf.d/vh_httpd.conf:
Can't locate Time/ParseDate.pm in @INC (@INC contains: /usr/share/musicbrainz/mb_server/cgi-bin /etc/perl /usr/local/lib/perl/5.8.4 /usr/local/share/perl/5.8.4 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.8 /usr/share/perl/5.8 /usr/local/lib/site_perl . /etc/apache-perl/ /etc/apache-perl/lib/perl) at /usr/share/musicbrainz/mb_server/admin/startup.pl line 30.
Compilation failed in require at (eval 7) line 1.
    

2005-02-07 02:48

Searching for Time::ParseDate, it seems that it's part of a group of modules. I guess I'll try putting all of them in a package. Ah, wait a moment, I found it. I'm after libtime-modules-perl. OK, and now I have a problem with it not having Algorithm::Diff. Tiresome.

And do you know, bugger it. I'm getting just a little pissed off with people doing work for me. Bastards! This, I just found in the INSTALL file:

New (Jan 2005):
If you can dedicate a whole machine to use as a MusicBrainz server, you might find the following helpful:

http://wiki.musicbrainz.org/wiki.pl?DebianServerSetup

It describes how to take a fresh machine, install Debian on it, and then install the MusicBrainz server on top of that. Even if you don't have a whole machine for MusicBrainz, you might find the above document a useful source of reference.

This is annoying because of course I've completely ignored it up until now, and my setup is rather different. Oh well. It's pretty clear that it installs perl modules the perl way, not the Debian way, and just generally isn't as nice as what I've been trying to achieve.

By the way, depend.pl doesn't produce any output for me, which indicates that it is a damned liar. The package I'm after is libalgorithm-diff-perl. Having installed that, I can get the front page, but I can't search for an artist. Looks like the database has changed. That means I'm going to have to download the database again. What a pain. Still, musicbrainz.exon.dyndns.org does work now, so that's a bonus.

2005-02-07 14:15

Well, after upgrading, it seems that the feature isn't yet implemented, and my change does work when I reapply it. That was too easy, so I think I'll find something else on the todo list and do that. Then I might contact the developers and see if there's something chunkier I could look at, like the random voters report or something.

2005-02-08 21:44

Dave Evans suggested several bugs I could fix, including this one about editing the track order. Seems like a more interesting case, so I'll give it a go. Unfortunately, my server isn't working any more, presumably because of the OS upgrade I did yesterday. Seems to have that segfault again :-(

I just did the thing of re-compiling libtext-unaccent-perl, and as I was installing I noticed that I must have upgraded it to 1.08:

aeon:/home/mexon/musicbrainz/textunaccent# dpkg --install libtext-unaccent-perl_1.07-1_i386.deb
dpkg - warning: downgrading libtext-unaccent-perl from 1.08-1 to 1.07-1.
(Reading database ... 168283 files and directories currently installed.)
Preparing to replace libtext-unaccent-perl 1.08-1 (using libtext-unaccent-perl_1.07-1_i386.deb) ...
Unpacking replacement libtext-unaccent-perl ...
Setting up libtext-unaccent-perl (1.07-1) ...
    

Now we're back in business...

2005-02-12 04:58

Well I've fixed a few bugs in MusicBrainz and I'm feeling pretty good about things. This update is just to mention a few things. Many of the images can't be found by the server. I get errors like this:

[Sat Feb 12 04:57:44 2005] [error] [client 192.168.64.1] File does not exist: /usr/share/images/remove.gif
    

Looks like a misconfiguration somewhere, but I can't see where. It doesn't matter anyway.

Just thought I'd mention that when adding releases, you need Date::Calc, which is in package libdate-calc-perl. libbit-vector-perl and libcarp-clan-perl came along for the ride.

2005-02-12 11:08

A further note. I fixed the images problem by adding an alias in vh_httpd.conf next to the one for cgi-bin:

    Alias /images/ "/usr/share/musicbrainz/mb_server/htdocs/images/"
    

2005-02-18 12:54

I was having problems testing some bugs because my moderations weren't being accepted, no matter how many votes they had. I think I've figured out the problem now - moderations are "run" by the script CheckVotes.pl. I'm running it now. It's having to churn through the entire list of open moderations, of which there are thousands, but I only need to do this once. It's taking a long time, but I can tell it's doing something useful because I can see in psql the number of entries in open_moderations going down.

One problem I had though is that the script as is wouldn't work, complaining that I couldn't use "0" as an arrayref on line 230 of ModBot.pm. Looking at the code for GetSubscribersForArtist, I couldn't figure out how I'm supposed to signal to that function that I want an array. So in the end I just commented out that piece of functionality:

    my $sql = Sql->new($self->{DBH});

#    return $sql->SelectSingleValue(
#		"SELECT COUNT(DISTINCT moderator) FROM moderator_subscribe_artist WHERE artist = ?",
#		$artist,
#    ) if not wantarray;

    $sql->SelectSingleColumnArray(
    

Ugly, but seems to work.


Matthew Exon
Last modified: Sat May 28 15:36:32 CEST 2005