Matt Florell: [0:00] OK, well, it’s 15:32. Time for the presentation about VICIDIAL. How many out there have ever heard of VICIDIAL? That’s good. Can I see? [laughs] Now I can. How many of you have used VICIDIAL? Probably a smaller number. OK, a few. [laughs] [0:26] Well, this is basically a presentation about VICIDIAL, about how it evolved, some of the issues that ran into it, reasons that we made some of the choices we did with the things we did. If Neer was here, I am sure he would be sneering at some of the AGI stuff I am going to talk about, but luckily he’s not.
[0:49] What is VICIDIAL? Well, it is a call center suite, open source. It actually was GPL licensed. We switched to the AGPL, because it is primarily a web-based application on the agent and the admin side. We could get into a very long discussion about the differences or the reason for doing AGPL, but, I don’t think we will go into that today. We only have a half hour.
[1:17] It does inbound, outbound, call handling as well as blended, putting them all altogether; a lot of different options involving all those. It runs on top of Asterisk open source PBX, and yes, only Asterisk. No, I have no plans to develop it for FreeSWITCH or Yate or anything else, just because of how integrated it is into Asterisk. It would take a very long time to get everything running under a different platform. So yeah, I realize that creates inherent inefficiencies, but we have gotten it to work very well the way it is, and unless somebody comes up with a large quantity of cash, we probably won’t be developing for FreeSWITCH anytime soon.
[2:08] As I mentioned, web-based user interfaces, available in multi languages. The Amoocon site actually mentions ten languages. Well, we have added two since then just in the last couple of months; Dutch and Russian to the agent side. All of our languages originally come from a contributor in the native tongue, except for the first version of the Spanish one, and we will talk about that later. [laughs]
[2:40] Why was VICIDIAL created? Well, two very simple reasons; because Asterisk was available, which I will get into. It was a wonderful discovery, and the lack of millions of dollars to buy a commercial outbound dialer solution for a 200 seat call center. Which at the time, 2003, was pretty much it. If you wanted a 200 seat you were going to spend a million dollars, at least, in the first year, just on hardware, software, support, licenses. And, that was a bit daunting to a couple of my clients at the time. So, we started out slowly.
[3:18] In 2003, my frustrations in dealing with Dialogic and Bayonne gave me a very good example of what’s a difficult platform to work with. Just trying to get the Linux Dialogic drivers working was an exercise in frustration in itself, and dealing with Bayonne’s interfaces. It’s taking the level of control you have in Asterisk, and making it even more detailed, so where you have to take 20 steps to do what would be one step in Asterisk.
[3:52] So, then I discovered Asterisk. I bought a single T1 card from Digium, and had a working system on a T1 in two hours and was blown away. And, this was pre-0.7 release, but it still was more stable and more of a good experience than the weeks that I had spent working, or attempting to work with Bayonne. Then, I discovered that Asterisk was not only easy to use, but it had these powerful programming options of the Asterisk manager interface and AGI; the gateway interface that Neer was just speaking about.
[4:35] So, release the AST GUI client GPL project, on SourceForge. It was Perl applications only, some AGI, some constantly running daemon scripts that would access the AMI, and a Perl/TK client. And at this point, it didn’t even do list styling or anything that resembles a call center application. It was purely an extension. It was kind of an operator panel at the time. It was able to do very basic click-to-dial type applications. You put in a number and then it dials it. You could do conferencing, you could do recording, some very basic stuff.
[5:21] But then 2004 came along. VICIDIAL, the client wanted to be able to just do list dialing. So, we were able to have a very, very basic web interface, where you could load in lists, and it would click-to-dial, and the agent would get the next lead and dial it. You couldn’t really do much more than that.
[5:45] You were able to do a few more things as time went along. And by the end of the year, you were able to do conferencing and stuff like that in the application. Auto-dialing was added towards the end of the year as well, where you can dial two lines for every agent but the managers, whoever was running the room, had to adjust manually the dialing ratio. So, that was a bit painful, and with the United States FTC regulations coming into force, that was also illegal.
[6:24] Inbound call handling actually used a separate web-based application where the agents had to click on phone calls to get them, which funny enough as it is, some of our inbound clients would prefer that kind of action. Because there are some utilities, or some ACDs out there, that still function that way, that are out on the market where agents actually have to choose the calls that they take.
[6:48] The ability to send calls to other groups was added towards the end of the year as well. I presented it, the first AstriCon in Atlanta, about basically dealing with the manager interface and some of the inherent flaws that are in it. But, we will talk about that later.
[7:06] This is the lovely first screen shot. Look at all the buttons, pretty colors. [laughs] So, you can see what the agents had to look forward to. It worked, it functioned. This was client-server, everything was in Perl/TK. It ran on Linux, it ran on Windows, it ran on Mac, so it was multi-platform. But, it was lacking in several ways, and we will go over the conversion from Perl/TK to web later on.
[7:43] In 2005 is when that conversion happened. We switched to Ajax-based web user agent. Added blended call handling, so an agent logged into a session, could take inbound and outbound at the same time. Added a script tab so you could do scripting; populate the name, address and that kind of stuff inside of an agent script. Hot keys so that you can press a number and it would hang up, dis-position a call and move onto the next one; again just adding to the efficiency. This is also when multi server load balancing was added which added another level of complexity, but also a very good ability to be able to scale the system across multiple servers.
[8:34] And, our first alternative language, Spanish, we released as a translation. This is the web-based screen shot. Again, not very different, it gets better. [laughs] But, at this point in time, it really helped with the adoption of VICIDIAL to be web-based because it removed a lot of barriers.
[9:00] The next year, 2006, we won the SourcForge.net project of the year for VoIP applications. We added scheduled callbacks, alternative phone number dialing for leads. Finally, got down using a lot of data that had been gathered over several months, and created a pretty good dialing algorithm.
[9:24] And we moved to SVN and became a real project using a code control. Which I was reluctant to do at first but now I’d never go back. Because having the ability to look at changes and be able to log changes in your development environment as well as all of the other things that Subversion lets you do is absolutely vital to the project — thriving, in my opinion.
[9:51] Here’s the next screenshot. A little better. Not finished but getting there. 2007, we added skill-based routing. QueueMetrics logging added. This is our first integration with a proprietary technology. Added List Mix which gives way too much control to the manager.
[10:15] So we actually had it disabled by default. Another list lead ordering options. We did our first training class. We founded a company to do commercial support for VICIDIAL. And we launched a hosted service where companies could come and basically just pay us money to have access to a server that had VICIDIAL on it.
[10:41] They weren’t paying per agent, which is a very different concept in the hosting call center business. Most of them you pay per agent. And that’s led to some very big installations that we’ve done in our co-location facility. And last year, we got the second prize for voice applications at VoIP Germany.
[11:05] We added queue prioritization, a separate time clock tracking application, DID call routing from the web interface, in-bound features like estimated hold time, place in line, adding a threshold with different options on wait time.
[11:23] And we released a live CD which, we have a new version right here. There’s a stack of them sitting there. They’re free. Feel free to take one. This is a…it boots off of the CD, it won’t overwrite your hard drive.
[11:38] It’s got a fully functional Asterisk, MySQL, Apache PHP, VICIDIAL, Ubuntu desktop, Firefox, Zoiper, and another softphone, all on the CD. And yeah, it can be a little slow on slower older machines, but it’s a very good way to demo it and actually see everything. Because it is fully functional you can actually connect to it from an outside phone or use an outside trunk if you would.
[12:11] But we do not recommend using it in production because for that, we enter this year, we integrated with Sangoma’s call progress detection, which I’m actually going to be talking about tomorrow in a presentation at, I think, 5:15 PM, in the other room.
[12:36] We added in depth vTiger integration. A client of ours needed a CRM application that was able to…their qualification was that it had to be a open source, purely open source. So we integrated with vTiger, and they’ve made a lot of customizations to it.
[12:55] But, user synchronization, accounts synchronization, call logging, click-to-call from vTiger, is all enabled. Agent shift enforcement, we added. Web-based Asterisk configuration. This was a pretty big one that was a lot of fun to deal with. But basically the phones and the servers that you configure inside of the VICIDIAL interface you can set the system so that it will create .conf files: extensions, SIP, IAX and it will do the reloads in a system-friendly way. It actually attaches to a screen that Asterisk was launched in, so you don’t get the nasty Asterisk freezes through doing a reload from Asterisk -r.
[13:47] We released 2.0.5, which is our current release. We have a VICI-box server installation ISO which I didn’t bring with me but you can download, that will do a production quality install of Ubuntu server with everything. Downloads, compiles Asterisk, so you have the source code. All that fun stuff.
[14:09] And this is the current screenshot. See we’ve gone to gray, to clean things up a bit. Some of the planned features that we’re actually working on now, we have a functional text-to-speech integration with Cepstral. We’ve done some work on our web-based IVR configuration.
[14:30] And we are planning on adding dynamic data forms and an optional Java phone, which we’ve had some issues with in the past, mostly due to inconsistencies on the client side. But we have several potential clients and a couple of actual clients of ours now that are interested in the Java phone option.
[14:50] Issues working with Asterisk. I am going to keep this kind of small because there are a lot of them that we’ve had over the years. Some of the biggest ones that led to some of our design decisions. Asterisk’s Queues was unstable, unreliable, and inconsistent. You’d go from point release to point release within 1.0 and Asterisk’s Queues had all sorts of issues.
[15:14] One behavior wouldn’t be in the next version and then it would be in the version after. But if you did it, it would crash. So we needed something that was more stable. Also there was, and still is in, 1.2 and 1.4 no ability to just take two channels and bridge them.
[15:33] I wrote a patch for that. And it worked pretty well, but there were some issues with doing native transfer, natively bridged channels. There were a lot of issues with that so I didn’t pursue it much further than that. Plus, 1.6 ended up coming out with a proper way to bridging with channels. So instead of doing all that we went with a known quantity that was stable and had been stable for quite some time, which is MeetMe.
[16:07] It also allows you to do a lot of other interesting things, like raise and lower volume of different calling parties, mute different calling parties, doing three way calls, four way calls, DTMF macros which is a way of playing DTMF, the actual audio, so that they pass out over the audio stream.
[16:29] MeetMe did introduce a certain amount of inefficiency to it all. But the stability that we were given and the ability to do those extra features, pretty much in my opinion, outweighs the inefficiencies.
[16:45] Not to mention with VICIDIAL you are not tied down to a single server, you can keep adding servers in a cluster. We have one client that uses some rather old machines at times and they’ve got 16 servers in a 260 seat installation; heavily outbound.
[17:05] I believe they have 26 T-1s plugged into them. So that’s the reason for so many. The queuing is done through AGI and that is something that Neer would probably not liked to hear. Yeah, my AGIs for call routing run longer than five seconds and they’re designed to do that.
[17:27] When the customer picks up the phone they go into the AGI and the call is immediately attempted to be sent to an agent. If there is no agent available then all sorts of different things happen, just like they should in a queuing application. But because it’s done in AGI, all the settings are database driven. There’s really no issue involved with doing that.
[17:59] Bugs and crashing on high load. Well, this is still an issue with Asterisk. We had standardized on 1.2 for several years, and in our very high volume outbound dialing applications we still recommend using one of the higher 1.2s, 1.2.27 all the way up to 30.4. I think there’s, I don’t even know what’s out now — 33, 31.something.
[18:29] We’ve started doing a lot more installations with 1.4. Really, nothing was usable up until 1.4.18. 1.4.19 was pretty good. 184.108.40.206 is what we’ve been using for the last several months. 1.4.22 and 23 both had critical bugs in them. 1.4.24 introduced some rather annoying errors if the Asterisk manager interface connection is terminated too quickly, or what Asterisk now deems is too quickly. It spews a bunch of errors on the screen, which aren’t really errors because nothing is wrong. It’s operating exactly as it has been.
[19:12] Aside from that, 220.127.116.11 seems to be fairly stable in our experience. We haven’t done that many recent checks on 1.6. We do need to do some work to be 1.6 compatible, and I’m sure we’ll get there eventually. But as of yet, we haven’t really wanted to dive into that; haven’t had the time.
[19:37] Some of the issues we had with Asterisk manager interface. In terms of bugs, doing channel lists where you would actually have some of the channels not show up. Let’s say you had Zap channels, SIP channels, and IAX channels. You do a "show channels, " and you’ll see 20 Zap, 20 SIP channels, 20 IAX channels.
[20:01] Then the next time you do it, and this happened a lot in the earlier versions of Asterisk 1.2, you’d see 20 Zap channels, 0 IAX, and 20 SIP channels. They just wouldn’t be there. So we had to write logic into some of our applications to be able to ignore output from time to time, because sometimes it just didn’t show up, and you knew if in a half a second you went from 20 IAX channels being there to just disappearing, something is probably wrong. So you should probably try to grab it.
[20:34] There were little inconsistencies like that in different functions of the manager API. There were a lot of other little issues like that that we’ve had to deal with over the years with Asterisk. That one’s gotten much better, especially in 1.4. They’ve done a lot to help fix that.
[20:56] The Agent evolution, as you saw, started with Perl/TK. This was a bit cumbersome, requiring client software installation on every machine. Upgrades were very time-consuming, because you had to make sure everything worked — the configuration file that was on the machine to connect to the database and all that.
[21:54] The code is on the server so no upgrades are needed. So this really, as I said, helped with the adoption of VICIDIAL to the point where now VICIDIAL is installed at over 1,000 company locations in over 70 countries around the world.
[22:51] We have clients running this on Pentium II, 500 MHz machines. So you really don’t need any kind of speed demon on the agent side. We even had some clients running over a dozen machines off of an old Linux terminal server.
[23:09] Multi-language builds. This one was a lot of fun. UTF-8, if anybody has ever worked with that and seen the lovely little things that it outputs on your screen on a terminal session, knows what UTF-8 is like. There are over 500 phrases in the agent interface and over 2,000 phrases in the admin interface that had to be translated.
[23:34] Static builds are created with a translation utility, and I want to mention that it’s very important when doing multi-language builds to get input from a native speaker, because when we were doing these literal translations, the Spanish one, hang-up on customer literally meant hang the customer and kill them; which we were told the agents found out was just hilarious, because when they got off the phone, they got to click, “Yes, kill the customer.”
[24:07] So that’s a little experience you might want to — we had the same kind of experience in German. We were trying to do literal translations, and we kept getting told, “No, people understand that’s English, and you can use the English. It’s OK.” So we were doing these literal translations of hang-up the customer, and it would mean take the customer and put them up on a hook and decorate them, or something, was what we ended up with. [laughs] So we just changed it to ‘hang up’.
[24:41] How do you make money? I don’t know how many times I’m asked this, especially at commercial call center shows. Like, your software is free? You don’t charge licenses? You don’t charge for add-ons like recording? Recording is not an add-on. Well, it is to us.
[24:58] That’s how commercial call center software companies will charge $1200 per agent seat. Then you have an add-on of recording, which is another $1200 per seat. Then you have another $800 to $1200 per seat for workforce management. They’re all separate utilities that have these separate tie-ins together that need their separate servers. You pay each company separately for support, and it ends up costing you, as I mentioned, millions of dollars to have a large call center.
[25:29] So, VICIDIAL is a bit of a foreign concept to them, at least the model. So this is how we make money. We sell manuals. We have two manuals that have over 200 pages of material in them that just go over how to use the interface. Pretty much everything that’s mentioned in the manuals has been mentioned at one point in time or another in our forums, which we freely support — the VICIDIAL forums. We post on them. I post on them daily. And basically everything is tested out there or posted there before it even makes it into the manual.
[26:07] We do training classes, just like a lot of other open source companies of a certain size — Asterisk has them all over. We just have them in the US, but it’s another way to make money.
[26:20] We sell hardware, usually just to companies in the US. Other than that we’ve done some shipping of TDM cards out to other countries, but again, it’s another way to make money. Hosted VICIDIAL service makes money. Installation of VICIDIAL systems where we install on new hardware, bill hourly for the installation depending on the client needs.
[26:45] Upgrades of existing or improperly installed VICIDIAL systems. This is always a big one, because there are people that take it and they don’t know how to configure it. They set it up; somebody got paid $200 to do a complete system install, and then they got the money through the software on there, and nothing worked, and then they never heard back from them. So then they would call us.
[27:09] Customization. This one is a big one — customized programming to suit client needs. Just consulting, basically.
[27:16] So, all together, this is how we stay in business and pay everybody and make money and make profit.
[27:25] So why do big companies choose to use something from a small company that’s free, when they can pay a lot of money for the same kind of thing? [laughs] No end of life — this is how we’ve gotten some of our biggest clients.
[27:40] If any of you have ever heard of the call center company Davox, they were around for many years. They were one of the first, if not the first predictive dialer company in the world. They were gobbled up by the Aspect monster, which was leveraging itself in debt to go and buy a bunch of companies.
[28:00] We retained a couple of clients based upon the fact that they were told that their dialer that they had spent millions of dollars on is no longer supported. They couldn’t get the hardware. One of them was buying replacement parts on eBay, and [laughs] they basically needed an option.
[28:21] Because both of these large companies were so frustrated at how they’d been treated with this buyout and they couldn’t do anything about it, they turned to open source. One of them actually approached several commercial call center software companies and wanted to know if there was any way they could get source code put in escrow in case something happened to the company.
[28:45] They were basically told, “No, you have to buy us for $50 million.” So in walks VICIDIAL. [laughs] Somebody at these companies found VICIDIAL and contacted us, and we went through the whole process of helping them install.
[29:03] One of them is still our client after three years. Another one internalized it. They hired programmers, and they maintain their system entirely themselves. So the cost of paying for six IT staff personnel is still much, much less than they ever would have been paying a commercial software provider. So they’re all very happy with how it worked out for them.
[29:29] No per-seat licensing costs, of course, money. That’s always an important factor, especially when you have hundreds of seats and you’re facing millions, or at least hundreds of thousands, of dollars a year to support your call center.
[29:46] A wide feature set. As I mentioned, VICIDIAL’s got built-in integrated recording with the ability to link to recordings, amongst a whole bunch of other standard call center features.
[29:59] Internal controlled code base; as I mentioned one of the larger clients is doing that. We have several clients that are making alterations on their own as well.
[30:11] A higher degree of customization is possible because you have the source code. It’s written in Perl and PHP. Interpretive languages are not that difficult. You don’t need a higher-level programmer to make simple alterations to it.
[30:27] Commercial support is available. Of course, that’s important with a company to stand behind it. If we were to dissolve or get swallowed up in a tsunami because we are in Florida, or a hurricane, there are consultants around the world. We have a preferred partner list of companies around the world that do VICIDIAL installations and support. So it is supportable.
[30:52] And that’s it for the presentation part. Anybody have any questions? Step out from the blinding lights. Yes?
Man: [31:03] Why did you [inaudible] ?
Matt: [31:05] I started in Perl because in my opinion Perl was more well suited to doing AGIs, especially AGIs that, as Neer said, never end. So you don’t have to use PHP. It was more efficient in my tests back in 2003 when I started. [31:24] There were also daemons running on the back end. So everything on the back end that’s not a user interface is Perl, and everything that is a user interface is PHP. They’re very similar languages. Some of the code is actually the same between them, shared code, because you can do that. Perl is easy.
Man: [31:44] [inaudible]
Matt: [31:46] Both. We use FastAGI for some of the very repetitive processes, like logging. And when we switched that to FastAGI from AGI, we saw a 50% cut in processor load, which was pretty dramatic; because the logging runs several times per call. [32:05] The call routing and some of the other AGIs are run just as AGIs. I’m sure we could optimize them more and turn them into FastAGIs which we’ve talked about. But at this point it’s not really worth the investment of time because everything works very well as it is. Yes?
Man: [32:25] [inaudible]
Matt: [32:42] We have lots of logs. [laughs] Actually, I can bring up a live call center in the US, in Florida, if you want to see it. If we have time, I don’t know when the next person’s coming on. In about ten minutes? [32:58] This is actually a live call center. This is reporting. Actually, if I were logged into it, you can click to monitor, where you register your phone. You enter your phone into here, and then you can actually click to listen. Click and it calls your phone with the agent in the session.
[33:19] Because we use MeetMe, you have virtually unlimited whatever-your-system-can-support monitoring phones. This client actually has a training room that has 20 seats in it. Agents have phones. They can all dial into the same live agent conference session and listen in blindly.
Man: [33:41] [inaudible]
Matt: [33:46] Yes, it’s through MeetMe Admin, which is part of Asterisk 1.4. We back ported it to 1.2 as well. It sends a command that raises or lowers the volume, as well as you can do mute and unmute. So you can adjust volume to suit whatever the caller on the other end needs; if you need to be louder or you need to hear them louder. [34:11] That’s pretty difficult to do through an Asterisk connection. For anything else it’s actually impossible, [laughs] unless you alter the source code.
[34:22] Any other questions? Yes, go ahead.
Man: [34:27] [inaudible]
Matt: [34:35] Yeah. I believe it’s IAX. It doesn’t really matter to me what it is. I’d prefer IAX because I love IAX. But we use IAX for server-to-server communication because it’s very efficient. It doesn’t have baggage left over when the call ends; and we’ve actually tried SIP and it doesn’t work. It’s not fast enough. It doesn’t kill fast enough because SIP hangs around to wait for messages, and IAX doesn’t need to. [35:03] To the agent interface we’d add the tab, and the softphone would be in the tab. It would be optional. We’ve tested this in the past with another Java softphone, and we had very inconsistent results and audio quality and compatibility for agent systems. So we kind of shied away from it.
[35:24] But there’s a commercial one, and I can’t remember the name of it off the top of my head, that was just released. The source was released open source. That sounds familiar. It was recently released, so now there’s an open source quality Java softphone that is probably what we’ll choose to do the integration.
[35:47] But as I said, since it’s pretty open, you could probably pop in any softphone you wanted. But just because of firewalls and efficiency, we prefer to use IAX.
Man: [35:58] [inaudible]
Matt: [36:08] Sure, if it can be triggered or launched in some way. QueueMetrics actually has a sidebar plugin. We haven’t integrated with it directly, but you can actually launch it through calls that you can trigger from a web page. If it can be triggered, we could do just about anything like that. [36:39] Any other questions? No? Thank you very much.