ConfBridge Beta support!

[mflorell]

We've recently added patches and documentation for ConfBridge support for VICIdial in Asterisk 16 and higher.

To test this, you will need to read the ConfBridge documentation, and upgrade your system to svn/trunk revision 3628 or higher,
https://www.vicidial.org/docs/ConfBridg ... tation.txt

Activation of ConfBridge requires multiple manual steps, including patching existing VICIdial files, so it should be done by someone familiar with Linux command-line.

Please test it out if you are able to and post your results here!

[carpenox]

I was abele to get this working rather easily with matts notes. If anyone wants to test it out, go to www.cyburdial.com and login as an agent, the user i 0000 and the pass is demo


call yourself from the dialer.....no more bloop.


I will be creating an article for this sometime this week but if u have any questions, feel free to message me or stop by our skype live support channel which now has nearly 200 visitors and clients just chilling and helping each other out n shit, great vibes.


Chris

[carpenox]

Here is the article I wrote for all of you guys to be able to get this done easily:

https://cyburdial.net/how-to-switch-vic ... onfbridge/


Chris

[mcargile]

Please keep in mind that this is all still changing. As an example it was pointed out that I missed the RINGALL code. As the agent entering the conference for a RINGALL is neither a regular agent nor an admin user it makes sense to create a ConfBridge user profile and an extension for that entry as well. Please pay careful attention to the SVN commits and re-read through the documentation every time something is added to the ConfBridge section.

[jamiemurray]

I've been using this extensively and have discovered a few bugs all of which have been posted on the issue tracker most of them with fixes.
It seems there's been little movement this week, the guys must be busy so if you're using SVN 3638 you may be experiencing issues with some of the steps
Here's a list of pending issues as of 3638, with the suggested fixes.

1. Various stats don't reset with end of day process (eg. Agent call count on the realtime report).
Caused by typo on ADMIN_keepalive_ALL.pl patch - fix explained.
http://www.vicidial.org/VICIDIALmantis/view.php?id=1380

2. Agent API audio_playback broken.
Caused by typo on ADMIN_keepalive_ALL.pl patch - Updated Patch Provided
http://www.vicidial.org/VICIDIALmantis/view.php?id=1379

3. Blind monitor by manager causes entry/exit tones to play in conference.
Caused by the blind monitor profile having quiet=no set. Fix explained.
http://www.vicidial.org/VICIDIALmantis/view.php?id=1377

4. If in System Settings a custom login sound file is selected, the agent's phone doesn't ring on login.
Caused by vicidial.php switching to a context that was never set up for ConfBridge - No fix per say, but conf bridge allows you to change the only participant message relatively easily.
http://www.vicidial.org/VICIDIALmantis/view.php?id=1376

[jamiemurray]

Another one to add:

http://www.vicidial.org/VICIDIALmantis/view.php?id=1381
The AST_conf_update_screen.pl script fails to authenticate with AMI due to a username mismatch.
Fixed with changing line 218 to:

Code: Select all

$telnet_login = "$ASTmgrUSERNAME";

[mflorell]

I think we've fixed all of these issues(and a couple more we found this week) in the ConfBridge code and doc in svn/trunk revision 3640. Please update and test if possible.

Thank you very much for your work on this jamiemurray!

[jamiemurray]

You're more than welcome Matt.
I tested the fixes on a client who hasn't started yet, it all seems fine, the only one I didn't check yet is the Custom Login Sound due to time constraints before agents start logging in.

[jamiemurray]

Custom Login Sound is definately fixed too.

[jamiemurray]

Matt, is there anything else you are waiting on feedback for before including this in the standard code? I am happy to try out any remaining items to be checked to speed this process up.

[mflorell]

I think we are pretty good on feature testing, but we would like to get as much data on capacity/load as possible. We also would like more feedback on anyone running ConfBridge with VICIdial on any kind of Virtual-Machine platform.

So, any details on what kind of per-dialer load you are running at(agents/calls/concurrent-calls/etc...) as well as what kind of hardware or VM platform you are using would be great!

[jamiemurray]

Short lived for me so far. I've reverted to MeetMe after discovering a major audio issue with ConfBridge that I haven't quite managed to get pinned down.
We started getting random complaints about audio, those complaints when checked all had one thing in common, the customers were saying "you're breaking up", "every other word is cutting out" but on the recordings the agents (and customers) sound 100% fine with no audio issues apparent what so ever.
On our softswitch, MOS scores were all 4-4.5 for the calls reported, no/little packet loss and low latency/jitter.

After checking all carriers etc (examples provided also showed all carriers were affected), I tried doing a echo test via an external number via a phone on the system, that's fine. Then I tried it from a ConfBridge session (via dial override on the agent interface) and bingo, the audio on the other side when played back was terrible. I repeated the test with MeetMe and the issue isn't there.

I've read a few reports about bad audio on ConfBridge and it seems to be something to do with the mixing as far as I can see.

Agents are pretty much all on ulaw for their call to the ConfBridge.
Carriers mostly only connecting on alaw, with a handful of ulaw
Using chan_sip
No more than 15 agents per telephony server.
Load is always under 30%

I'm going to try a few experiments with codecs and different mixing and sampling settings to see if I can address it but just wondering if anyone got any ideas on where I should focus my efforts.

Asterisk 16.27.0-vici
SVN 3643
Original Install via ViciBox v.10.0.0 210901

[mflorell]

Thank you very much for the feedback!

I wonder if Asterisk 18, and/or using PJSIP instead of chan_sip would have any affect on this?

We have PJSIP support in Asterisk 16, and we've been testing Asterisk 18 compatibility, so it might be something else to try.

[mcargile]

@jamiemurray few questions:

What is the specific hardware are you running this on?
Is it virtualized?
Do you have physical DAHDI timers?
What does dahdi_test show when the audio is getting choppy?

To the best of my knowledge ConfBridge does not need a DAHDI timer, but the system over all benefits from having one. Basically it is pulling its timing from the underlying Asterisk timing system which will use the best timing source that is available. I do plan on getting clarification on this during AsteriskDev Con this year.

[jamiemurray]

Thank you very much for the feedback!

I wonder if Asterisk 18, and/or using PJSIP instead of chan_sip would have any affect on this?

We have PJSIP support in Asterisk 16, and we've been testing Asterisk 18 compatibility, so it might be something else to try.

I'll try PJSIP, also Asterisk 18 if that doesn't work. I do like the fact though asterisk is managed by zypper, it allows the automatic update script I have on the servers which automatically checks for updates to asterisk from your official repo and automatically performs svn updates overnight as well as applies any patches I've wrote to address specific setup requirements or bugs which haven't been resolved on official SVN version yet. Has the ast 18 you've been working with been made available on a zypper repo yet?

@jamiemurray few questions:

What is the specific hardware are you running this on?
Is it virtualized?
Do you have physical DAHDI timers?
What does dahdi_test show when the audio is getting choppy?

To the best of my knowledge ConfBridge does not need a DAHDI timer, but the system over all benefits from having one. Basically it is pulling its timing from the underlying Asterisk timing system which will use the best timing source that is available. I do plan on getting clarification on this during AsteriskDev Con this year.

This specific system is entirely virtual and has been working without incident on MeetMe with at times up to 20 agents per server although usually only around 15 per sever.
The 4 tel servers on this cluster all have the following resources:
AMD EPYC 2nd Gen. 8 cores.
16GB ECC RAM.
260GB NVMe SSD storage (no archiving done, recordings remain on the server for the required length of time the client is required by law to keep the calls).
QEMU based virtualization.
Bandwidth definately isn't an issue, there's around 3-4Gbps available up and down to each server, they also communicate internally via private vlan and are physically located in the same room but not the same host.

dahdi_test shows 99.9-100% during the calls where the customer leg is breaking up.

Some of the other things I saw around when googling, was playing around with mixing_interval depending on the codecs sometimes resolved the issue, so I'm going to attempt with 20 rather than 40 to see if that makes a difference.
Others have reported issues with dsp_drop_silence causing issues too.
It's strange how it's just on the carrier leg the audio from the conference is bad, if it's a internal phone or channel (such as recording), the audio is perfect.

[carpenox]

thanks for the plug but the diret link are:

https://cyburdial.net/how-to-switch-vic ... onfbridge/

and

https://cyburdial.net/how-to-remove-the ... -vicidial/

Chris aka carpenox

[mcargile]

Asterisk 18 is in very early testing. I have it compiled on my dev system and have had it place a few million simulated autodialed outbound calls to make sure it does not crash. That is it. I have not tested the AMI or AGI scripts to make sure they are working properly. I have not tested the agent interface. I have not tested manual dials, transfers, recordings, or a dozen other features. I dont even know if those autodialed calls have routed to where they are supposed to go. In other words I cannot promise that anything about Vicidial besides that it will place the calls to the customer is working properly.

I am attending AsteriskDevCon this year on October 24th and would like to be able to report to them about our ConfBridge experience. So if you could continue testing ConfBridge it would be greatly appreciated.

[jamiemurray]

I'm using it successfully on a few very small single server installations without issue but these machines are overkill for the load of the system if I'm being honest.
The ones I stopped using it on were clusters of 50 agents or more because of the issue I described above with the audio on the carrier leg.
I may try with just one server in a busy cluster, that way if the issue becomes too much and we can't get to the cause quickly, I can disable it as an agent server as a quick solution.

[mcargile]

So looking a bit further into the differences between ConfBridge and MeetMe I did find this post by Matt Jordon:

https://asteriskfaqs.org/2016/11/10/ast ... audio.html

It does look like ConfBridge uses the underlying timing interface provided by Asterisk to mix everything. Or at least it did in 2016. The priority of the timing interfaces in Asterisk are:

1. res_timing_timerfd.so
2. res_timing_kqueue.so (FreeBSD only)
3. res_timing_dahdi.so
4. res_timing_pthread.so

according to https://wiki.asterisk.org/wiki/display/ ... Interfaces. What is interesting is that page says, "The only functionality that requires internal timing is IAX2 trunking." which kind of implies that ConfBridge doesn't use timing.

@jamiemurray can you please try adding:

noload => res_timing_timerfd.so
noload => res_timing_kqueue.so
noload => res_timing_pthread.so

to /etc/asterisk/modules.conf and see if ConfBridge behaves better? That should force Asterisk to only use the DAHDI timing interface which is what MeetMe is using for mixing the audio.

EDIT: I forgot the => in the modules.conf noloads.

[mcargile]

So I have some bad news for ConfBridge on Virtualization. MeetMe does its audio mixing within the DAHDI driver. This means it gets done in kernel space. ConfBridge does its audio mixing in the Asterisk Bridge mechanism which is done is user space. As such on a Virtual Machine MeetMe is operating just one step above the hypervisor where as ConfBridge is operating two steps above the hypervisor. This means the ConfBridge is going to be considerably more impacted by performance on VMs than MeetMe.

EDIT: The Digium developers did say that the underlying Bridge mechanism in Asterisk used by things like ConfBridge and ARI Bridging is "pluggable" meaning the audio mixing COULD be offloaded to the DAHDI Driver like MeetMe, but it currently does not do that.

[mcargile]

@jamiemurray
The Asterisk developers recommended changing the ConfBridge mixing_interval to 20 from 40 and recommended switching over to res_timing_dahdi.so for the system timing.

With the mixing_interval set to 40ms, ConfBridge will send out two 20ms RTP packets at the same time every 40ms. This has been known to confuse some SIP endpoints and media gateways they are not buffering the audio properly. Especially if packets are arriving out of order. With the interval set to 20ms it is sending out one 20ms RTP packet every 20ms. This spaces them out which makes it less likely that the packets will arrive at the destination out of order.

res_timing_timerfd.so which is the default Asterisk timing interface can fail to trigger when Asterisk forks off a new thread. AGI scripts which Vicidial uses heavily forks off a new thread every time they get executed. res_timing_dahdi.so should not do that.

These two changes SHOULD make ConfBridge more consistent on physical servers that are not overloaded. One Virtual Machines or overloaded physical servers there is still a good chance that ConfBridge will have audio issues. As I said in my previous post MeetMe does its audio mixing inside the DAHDI kernel driver. This means the audio mixing is done in Kernel space and as such is done at a higher priority. ConfBridge does its mixing in user space. This means that it is less likely to work consistently when overloaded or when other VMs on the same hypervisor need execution time.

We are currently waiting to hear back from Sangoma about what it would cost to get the Asterisk Bridging mechanism used by ConfBridge and the ARI bridging system to use DAHDI for mixing.

[jamiemurray]

Trying this out today on the client that had the most issues with it. I'm sure I'll find out within 30 minutes of them dialing if it's successful or not. They start around 11am eastern, will post updates as soon as I know for sure.

[jamiemurray]

So far so good with 40 agents, another 15 agents start at 1pm eastern so will see if it continues then.

[mcargile]

How many agents per tel server?

[jamiemurray]

3 tel servers, 19 on one, 18 on each of the other 2. All still going well so far.

[jamiemurray]

The day completed without any complaints about audio quality. test calls throughout the day were all fine too, so looking like those few changes have resolved the issue for me!

[mcargile]

Still working well?

[jamiemurray]

Yes, it's been in production on all my clients' systems since then and no complaints about audio except for today on a new setup.
I discovered though fail2ban was causing high usage on one server and eventually started causing packet loss, stopping f2b for now has resolved it.

[okli]

Yes, it's been in production on all my clients' systems since then and no complaints about audio except for today on a new setup.

Thanks for sharing your experience!
Is this in virtual servers with same specs you posted earlier or physical servers?

[jamiemurray]

Working well on both physical and virtual, I will say however that virtual varies widely depending on the hardware used and config of the hypervisor. But my busiest entirely virtual cluster has 8 telephony servers with up to 20 agents per server, 100% recording and up to dial level 10, AMD, and I finally switched that one over a few days after I was happy the timing issue had been resolved and never looked back yet.