Migrating Asterisk Voicemail to IMAP

Leo Moll, January 19, 2014, 13:51:22

This article describes the migration of an active standard voicemail system running on Asterisk PBX from the default file storage to IMAP server based storage while preserving all existing voicemails and recorded user announcements (greet, busy and unavail)

There are several good articles that describe how to configure an Asterisk PBX for using IMAP server based storage for voicemails. Unfortunately it is quite difficult to find any information about how to migrate an existing Asterisk voicemail installation to IMAP without losing all recorded user announcements and voicemails. This article will explain the whole procedure and provide some useful tools that will support you in doing the task.

Prerequisites: You should have a decent understanding of bash scripting and be familiar with common tasks on Linux systems in order to understand this guide and perform such a migration.

How Asterisk stores Voicemails

In order to understand the migration process, we will take a look on how Asterisk stores all voicemail related data in a standard file based configuration. In our example we will assume the following very simple mailbox configuration from voicemail.conf:

[default]
80 => 1234,User 1,user1@example.com
81 => 1234,User 2,user2@example.com
82 => 1234,User 3,user3@example.com

Asterisk keeps the whole voicemail data under /var/spool/asterisk/voicemail in the following structure:

default/80/busy.wav
default/80/greet.wav
default/80/unavail.wav
default/80/INBOX/msg0000.txt
default/80/INBOX/msg0000.wav
default/80/Old/msg0000.txt
default/80/Old/msg0000.wav
default/80/Old/msg0001.txt
default/80/Old/msg0001.wav
...
default/80/Old/msg0025.txt
default/80/Old/msg0025.wav
default/80/unavail.wav
default/81/busy.wav
default/81/greet.wav
default/81/Old/msg0000.txt
default/81/Old/msg0000.wav

The three possible voicebox announcements are stored directly in the voicebox subdirectory (defined as <context>/<boxnumber>) in the file format specified within the configuration file (format=wav in the [general] section). However the voicemails are stored in two subdirectories (INBOX and Old) depending upon they have been already played or not. Every voicemail is accompanied by a sidecar file containing meta information about the call:

;
; Message Information file
;
[message]
origmailbox=80
context=macro-dispatch
macrocontext=target-moll-home
exten=result-NOANSWER
rdnis=unknown
priority=2
callerchan=SIP/sipgate.leo.3.0-00000121
callerid="01711234567" <01711234567>
origdate=Sat Nov 30 08:48:34 AM UTC 2013
origtime=1385801314
category=
flag=
duration=9

The challenge is to transfer all these data to the IMAP server without losing anything. In order to achieve this, we must understand how Asterisk stores announcements and voicemails in an IMAP server environment.

How Asterisk stores Voicemails on IMAP Servers

To find this out, I created a test environment, recorded all three announcements and a few voicemails. Afterwards I looked what had been stored on the IMAP server.

Basically there is no magic in this: Asterisk stores every entity as a normal mail object. The usual email attributes (sender name and address, recipient name and address, subject and mail body) are taken from what has been configured in voicemail.conf but they have no real effect on functionality. It seems that they are only provided for presentation purposes. The media file is included as a standard base64 encoded attachment named exactly as in file storage. The relevant meta data is instead stored in some extended header fields:

X-Asterisk-VM-Message-Num: 0
X-Asterisk-VM-Server-Name: Voicemail System
X-Asterisk-VM-Context: default
X-Asterisk-VM-Extension: 80
X-Asterisk-VM-Flag: (null)
X-Asterisk-VM-Priority: 1
X-Asterisk-VM-Caller-channel: SIP/office-leo-00000025
X-Asterisk-VM-Caller-ID-Num: 19
X-Asterisk-VM-Caller-ID-Name: Office Leo
X-Asterisk-VM-Duration: 16
X-Asterisk-VM-Category:
X-Asterisk-VM-Message-Type: unavail
X-Asterisk-VM-Orig-date: Saturday, 18 January 2014 at 18:37:37
X-Asterisk-VM-Orig-time: 1390066657
X-Asterisk-CallerID: 19
X-Asterisk-CallerIDName: Office Leo

The announcement itself is included as an attachment:

------voicemail_0805640663319705
Content-Type: audio/x-wav; name="unavail.wav"
Content-Transfer-Encoding: base64
Content-Description: Voicemail sound attachment.
Content-Disposition: attachment; filename="unavail.wav"

UklGRuTrAwBXQVZFZm10IBAAAAABAAEAQB8AAIA+AAACABAAZGF0YcDrAwAEAAQA/v/6//z/
...
TAAwAEYAIAAaAA4ABgDk/8z/

------voicemail_0805640663319705--
.

The only difference between a voicemail and an announcement, is the message type encoded into the extended header attributes, and the filename of the attachment:

...
X-Asterisk-VM-Message-Type: Message
...
------voicemail_18111680494858773
Content-Type: audio/x-wav; name="msg0025.wav"
Content-Transfer-Encoding: base64
Content-Description: Voicemail sound attachment.
Content-Disposition: attachment; filename="msg0025.wav"
...

Armed with this knowledge, we may now start with the migration of the existing voicemail boxes.

Migrating the Voicemail Boxes

Initial Considerations

One important decision you shall take, when migrating voicemail storage to IMAP, is whether you want to store the voicemails directly into the user's mail folders or if you prefer to create dedicated mailboxes for each voicemail box.

Although the first solution seems more straightforward, the second one has some indubitable advantages:

  1. Security: The credentials of the user specific e-mail account will not be stored in voicemail.conf. Instead the voicemail only credentials will be stored there. Also if the configuration file ends up in the wrong hands, only the voicemail box will be compromised.
  2. Consistency: If the user changes the password of his mail account, no intervention on voicemail.conf is required.
  3. Performance: Asterisk must poll at regular intervals all mailboxes in order to provide MWI (Message Waiting Indication). This operation is quite expensive, especially if the checked folders are stuffed with thousands of other emails. By keeping all data in a dedicated account, the resource usage is minimised as much as possible.

Also if voicemails are stored in a separate mailbox, there are several ways to map the voicemail box into the mail accounts of the owners (if wanted) but this is not in the scope of this article. In the majority of the cases, the method of symlinking mailboxes should do the job.

In our guide, we will create a mailbox for each user, store his voicemails into the inbox, and the announcements in a folder called "Greetings".

Since I run my Asterisk on Debian, you should check if the file paths on your system are the same as in this article, and if not, you should adapt them accordingly. Also the commands for installing/uninstalling modules are dependent upon the used Linux distribution.

Step 1: Create the Mailboxes

The first step is to create the mailboxes for each user on the IMAP server. Since the users do not need direct access to the mailboxes (unless you want them to add the box as a secondary mail account in their mail client), you can use the same password for all mailboxes (but obviously it would be better to generate them randomly).

Empty Mailbox

Create the folder "Greetings" in each mailbox by logging in into each mailbox.

Step 2: Backup all relevant files

You can do this with one command on your Asterisk:

tar -cvzf /wherever/you/want/voicemail-backup.tar.gz /etc/asterisk/voicemail.conf /var/spool/asterisk/voicemail

Step 3: Modify the Voicemail Configuration

In the general section of voicemail.conf the following values will be added (or modified):

[general]
; Asterisk needs to poll all mailboxes since there may be status changes coming
; from outside (like read/unread flags from reading on a mail client)
pollmailboxes=yes

; If the "pollmailboxes" option is enabled, this option sets the polling
; frequency.  The default is once every 30 seconds.
pollfreq=30

; Specify here the address of your imap server
imapserver=myimapserver.example.com

; Specify here the name of the folder where voicemails will be stored
; imapfolder=

; These flags are optional.
imapflags=notls

; Store the greetings on the imap server?
imapgreetings=yes

; If imapgreetings=yes, then specify which folder to store your greetings in. If
; you do not specify a folder, then INBOX will be used
greetingsfolder=Greetings

; Some IMAP server implementations store folders under INBOX instead of
; using a top level folder (ex. INBOX/Friends).  In this case, user
; imapparentfolder to set the parent folder. For example, Cyrus IMAP does
; NOT use INBOX as the parent. Default is to have no parent folder set.
;imapparentfolder=INBOX

For each mailbox, we will specify the IMAP credentials to use:

[default]
80 => 1234,User 1,user1@example.com,,imapuser=user1|imappassword=SuperSecret
81 => 1234,User 2,user2@example.com,,imapuser=user2|imappassword=SuperSecret
82 => 1234,User 3,user3@example.com,,imapuser=user3|imappassword=SuperSecret

Step 4: Replace the Voicemail Module

In a standard Debian installation of Asterisk, there are 3 different voicemail modules available:

  1. asterisk-voicemail - simple voicemail support for the Asterisk PBX
  2. asterisk-voicemail-imapstorage - IMAP voicemail storage support for the Asterisk PBX
  3. asterisk-voicemail-odbcstorage - ODBC voicemail storage support for the Asterisk PBX

Now we will replace the default (file based) voicemail module with the module providing IMAP support:

apt-get --yes --purge remove asterisk-voicemail
apt-get --yes install asterisk-voicemail-imapstorage
service asterisk restart

That's all. Let's see if everything is working:

root@tuxpoldo ~ $ asterisk -r
Asterisk 1.8.13.1~dfsg1-3+deb7u3, Copyright (C) 1999 - 2012 Digium, Inc. and others.
Created by Mark Spencer <markster@digium.com>
Asterisk comes with ABSOLUTELY NO WARRANTY; type 'core show warranty' for details.
This is free software, with components licensed under the GNU General Public
License version 2 and other licenses; you are welcome to redistribute it under
certain conditions. Type 'core show license' for details.
=========================================================================
Connected to Asterisk 1.8.13.1~dfsg1-3+deb7u3 currently running on tuxpoldo (pid = 8303)
Verbosity is at least 1
tuxpoldo*CLI> voicemail show users
Context    Mbox  User                      Zone       NewMsg
default    80    User 1                                    0
default    81    User 2                                    0
default    82    User 3                                    0
3 voicemail users configured.
tuxpoldo*CLI>

Great! Everything is doing fine. Now we will transfer all the old stuff into the IMAP mailboxes.

Step 5: Convert Announcements and Voicemails

In order to generate the mails to store in the IMAP accounts, I prepared a nice shell script, able to generate the RFC mail files out of the source files stored in /var/spool/asterisk/voicemail:

root@tuxpoldo ~ $ ./vmtoimap --help
usage: vmtoimap [-hneNEvfubd] <inputfile>

Options:
 -n, --destination-name       set the name of the mailbox owner
 -e, --destination-addr       set the e-mail address of the mailbox owner
 -N, --source-name            set the name of the voicemail system
 -E, --source-addr            set the e-mail address of the voicemail system
 -d, --duration               force duration (seconds)
 -v, --voicemail              force message type voicemail
 -f, --greet                  force message type greeting prompt
 -u, --unavail                force message type unavailable prompt
 -b, --busy                   force message type busy prompt

This script creates an RFC mail from an asterisk file based voicemail. It
tries to autodetect if the speicifed file is a bare media file (wav, alaw,
ulaw, g722, etc) or a sidecar file of a voicemail (msgXXXX.txt) and tries
to guess all related parameters

This script takes an announcement media file or a voicemail sidecar file as input, and outputs an RFC mail body on STDOUT:

root@tuxpoldo ~ $ ./vmtoimap -N "Voicemail System" -E 'voicemail@example.com' -n "User 1" -e 'user1@example.com' /var/spool/asterisk/voicemail/default/80/unavail.wav 
Date: Sun, 17 Jun 2007 17:15:06 +0200
From: "Voicemail System" <voicemail@example.com>
To: "User 1" <user1@example.com>
Subject: =?ISO-8859-1?Q?Voicemail_received_from_=22Unknown=22_=3CUnknown?=
 =?ISO-8859-1?Q?=3E_-_=280=3A10_min=29?=
Message-ID: <Asterisk-1-1182093306-80-11680@>
X-Asterisk-VM-Message-Num: 1
X-Asterisk-VM-Server-Name: Voicemail System
X-Asterisk-VM-Context: default
X-Asterisk-VM-Extension: 80
X-Asterisk-VM-Flag:
X-Asterisk-VM-Priority: 1
X-Asterisk-VM-Caller-channel: SIP/default
X-Asterisk-VM-Caller-ID-Num: Unknown
X-Asterisk-VM-Caller-ID-Name: Unknown
X-Asterisk-VM-Duration: 10
X-Asterisk-VM-Category:
X-Asterisk-VM-Message-Type: unavail
X-Asterisk-VM-Orig-date: Sunday, 17 June 2007 at 17:15:06
X-Asterisk-VM-Orig-time: 1182093306
X-Asterisk-CallerID: Unknown
X-Asterisk-CallerIDName: Unknown
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="----voicemail_18111680494858773"

This is a multi-part message in MIME format.

------voicemail_18111680494858773
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit

Dear User 1:

just wanted to let you know you were just left a 0:10 long message (number 1)
in mailbox 80 from Unknown (Unknown), on Sunday, 17 June 2007 at 17:15:06, so you might
want to check it when you get a chance.  Thanks!

                                --Asterisk
------voicemail_18111680494858773
Content-Type: audio/x-wav; name="unavail.wav"
Content-Transfer-Encoding: base64
Content-Description: Voicemail sound attachment.
Content-Disposition: attachment; filename="unavail.wav"

UklGRiTBAgBXQVZFZm10IBAAAAABAAEAQB8AAIA+AAACABAAZGF0YQDBAgAAAAAAAAAAAAAA
...
wP8=

------voicemail_18111680494858773--
.

Not every information here is correct (like the duration), but for announcements it's OK. If you instead specify a voicemail sidecar file, all relevant information is correctly extracted from the sidecar file:

root@tuxpoldo ~ $ ./vmtoimap -N "Voicemail System" -E 'voicemail@example.com' -n "User 1" -e 'user1@example.com' /var/spool/asterisk/voicemail/default/80/Old/msg0000.txt
Date: Sun, 22 Jul 2007 11:02:13 +0200
From: "Voicemail System" <voicemail@example.com>
To: "User 1" <user1@example.com>
Subject: =?ISO-8859-1?Q?Voicemail_received_from_=22messagenet.it=22_=3C000390583419052?=
 =?ISO-8859-1?Q?=3E_-_=280=3A15_min=29?=
Message-ID: <Asterisk-1-1185094933-80-11680@>
X-Asterisk-VM-Message-Num: 1
X-Asterisk-VM-Server-Name: Voicemail System
X-Asterisk-VM-Context: default
X-Asterisk-VM-Extension: 80
X-Asterisk-VM-Flag:
X-Asterisk-VM-Priority: 2
X-Asterisk-VM-Caller-channel: SIP/5324907-b6f06058
X-Asterisk-VM-Caller-ID-Num: 000390583123456
X-Asterisk-VM-Caller-ID-Name: messagenet.it
X-Asterisk-VM-Duration: 15
X-Asterisk-VM-Category:
X-Asterisk-VM-Message-Type: Message
X-Asterisk-VM-Orig-date: Sunday, 22 July 2007 at 11:02:13
X-Asterisk-VM-Orig-time: 1185094933
X-Asterisk-CallerID: 000390583123456
X-Asterisk-CallerIDName: messagenet.it
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="----voicemail_18111680494858773"

This is a multi-part message in MIME format.

------voicemail_18111680494858773
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit

Dear User 1:

just wanted to let you know you were just left a 0:15 long message (number 1)
in mailbox 80 from 000390583123456 (messagenet.it), on Sunday, 22 July 2007 at 11:02:13, so you might
want to check it when you get a chance.  Thanks!

                                --Asterisk
------voicemail_18111680494858773
Content-Type: audio/x-wav; name="msg0000.wav"
Content-Transfer-Encoding: base64
Content-Description: Voicemail sound attachment.
Content-Disposition: attachment; filename="msg0000.wav"

UklGRqR3AwBXQVZFZm10IBAAAAABAAEAQB8AAIA+AAACABAAZGF0YYB3AwAAAAAAAAAAAAAA
...
AAAAAAAAAAAAAAAA

------voicemail_18111680494858773--
.

This output can be piped directly to sendmail in order to forward it into the mailboxes. Let's send the announcements of User 1 into the IMAP box:

root@tuxpoldo ~ $ ./vmtoimap -N "Voicemail System" -E 'voicemail@example.com' -n "User 1" -e 'user1@example.com' /var/spool/asterisk/voicemail/default/80/greet.wav | sendmail user1@example.com
root@tuxpoldo ~ $ ./vmtoimap -N "Voicemail System" -E 'voicemail@example.com' -n "User 1" -e 'user1@example.com' /var/spool/asterisk/voicemail/default/80/busy.wav | sendmail user1@example.com
root@tuxpoldo ~ $ ./vmtoimap -N "Voicemail System" -E 'voicemail@example.com' -n "User 1" -e 'user1@example.com' /var/spool/asterisk/voicemail/default/80/unavail.wav | sendmail user1@example.com

After sending the announcements, they have to be manually moved into the "Greetings" folder, since the voicemail system will find them only there. You can test the announcements by calling the voicemail box. If everything was fine, you should now hear the greeting recorded by the owner of the voicemail box.

The next step is to import all old messages:

root@tuxpoldo ~ $ ./vmtoimap -N "Voicemail System" -E 'voicemail@example.com' -n "User 1" -e 'user1@example.com' /var/spool/asterisk/voicemail/default/80/Old/msg0000.txt | sendmail user1@example.com
...
root@tuxpoldo ~ $ ./vmtoimap -N "Voicemail System" -E 'voicemail@example.com' -n "User 1" -e 'user1@example.com' /var/spool/asterisk/voicemail/default/80/Old/msg0025.txt | sendmail user1@example.com

Since those messages have already been played, they must all be flagged as read using the mail client, otherwise all the old voicemails will be presented again as new the next time the user calls his voicemail box to listen to new messages.

Now the new messages can be imported:

root@tuxpoldo ~ $ ./vmtoimap -N "Voicemail System" -E 'voicemail@example.com' -n "User 1" -e 'user1@example.com' /var/spool/asterisk/voicemail/default/80/New/msg0000.txt | sendmail user1@example.com

Since this is a very annoying and repetitive procedure, it is a good advice to automate it using a script. The subsequent script can be adapted to fit your needs:

#!/bin/bash

# constants
VOICEMAIL_PATH="/var/spool/asterisk/voicemail/"

# CHANGE HERE:
# customize the following variables to your needs
ORIG_NAME="Voicemail System"
ORIG_ADDR="voicemail@example.com"
MEDIA_EXT="wav"
VOICEMAIL_CONTEXT="default"

# CHANGE THERE:
# go to the end of the script, and customize it to your needs by
# following the instructions in the comments

# $1: Mailbox number
# $2: Name of the User
# $3: E-Mail Address of the user
function migrate_mailbox() {
        echo "Migrating voicebox $1 to $3"
        if [ ! -d ${VOICEMAIL_PATH}${VOICEMAIL_CONTEXT}/${1} ]; then
                echo >&2 "Mailbox $1 not found"
                return
        fi

        echo "Migrating annoucements..."
        LOCATION=${VOICEMAIL_PATH}${VOICEMAIL_CONTEXT}/${1}
        echo ${LOCATION}/'*'.${MEDIA_EXT}
        echo ${LOCATION}/*.${MEDIA_EXT}
        for FILE in ${LOCATION}/*.${MEDIA_EXT}; do
                if [ -f "${FILE}" ]; then
                        echo "=> $(basename ${FILE})"
                        ./vmtoimap \
                                -N "${ORIG_NAME}" \
                                -E "${ORIG_ADDR}" \
                                -n "${2}" \
                                -e "${3}" \
                                "${FILE}" | sendmail ${3}
                fi
        done
        echo "Please log in and move all voice prompts to the"
        echo "Greetings folder. Hit ENTER to continue"
        read VAR

        echo "Migrating old voicemails..."
        LOCATION=${VOICEMAIL_PATH}${VOICEMAIL_CONTEXT}/${1}/Old
        for FILE in ${LOCATION}/*.txt; do
                if [ -f "${FILE}" ]; then
                        echo "=> $(basename ${FILE})"
                        ./vmtoimap \
                                -N "${ORIG_NAME}" \
                                -E "${ORIG_ADDR}" \
                                -n "${2}" \
                                -e "${3}" \
                                "${FILE}" | sendmail ${3}
                fi
        done
        echo "Please log in and flag all received voicemails as read."
        echo "Hit ENTER to continue"
        read VAR

        echo "Migrating new voicemails..."
        LOCATION=${VOICEMAIL_PATH}${VOICEMAIL_CONTEXT}/${1}/INBOX
        for FILE in ${LOCATION}/*.txt; do
                if [ -f "${FILE}" ]; then
                        echo "=> $(basename ${FILE})"
                        ./vmtoimap \
                                -N "${ORIG_NAME}" \
                                -E "${ORIG_ADDR}" \
                                -n "${2}" \
                                -e "${3}" \
                                "${FILE}" | sendmail ${3}
                fi
        done
}

# for every mailbox you want to migrate, insert one
# function call HERE

# $1: mailbox number
# $2: Name of the User
# $3: E-Mail Address of the user
migrate_mailbox 80 "User 1" "user1@example.com"
migrate_mailbox 81 "User 2" "user2@example.com"
migrate_mailbox 82 "User 3" "user3@example.com"

With this step, the migration should be finished. The scripts presented in this article can be downloaded from this site.

Author:
This page was last updated on January 21, 2014, 23:16:41