User:Sfiggins/Broadworks Controlled Registration

Verify RegistrationsPerMinute on AS1/2

If the RegistrationsPerMinute count on the active Application server is greater than a couple hundred, the platform will need help recovering.

Log into castrum.risebroadband.com, and then ssh to 172.16.18.3 or 10.1.75.181, depending on which AS is active.

$ ssh castrum.risebroadband.com
[sfiggins@castrum ~]$ ssh 172.16.18.3 -l bwadmin
bwadmin@172.16.18.3's password: 

bwadmin@as1.jabvoice.net$ bwcli


======================================================================
BroadWorks Command Line Interface
  Type HELP for more information
======================================================================
Reading initial CLI command file...

AS_CLI>

Then run the qcurrent command to see the current state of the queue.

AS_CLI> qcurrent
May 24, 2019 =====================================
 NbOfActiveCalls                               2.0
 SIPSetupSignalDelay                          16.0
 SIPAnswerSignalDelay                          2.0
 RegistrationsPerMinute                    12821.0
 SIPMsgRetryToNE::10.1.75.182                  0.0
 SIPMsgRetryToNE::172.16.18.4                  0.0
 SIPMsgRetryToNE::64.1.8.229                   0.0
 SIPMsgRetryToNE::172.16.18.5                  0.0
 SIPMsgRetryToNE::172.16.18.148               75.0
 SIPMsgRetryToNE::10.1.75.183                  0.0
 SIPMsgRetryToNE::172.16.18.3                  0.0
 SIPMsgRetryToNE::10.1.75.181                 83.0
 SIPMsgRetryToNE::172.16.18.130                0.0
 SIPMsgRetryPercentToOther                    91.0
 MGCPDialtoneDelay                             0.0
 MGCPSetupSignalDelay                          0.0
 MGCPAnswerSignalDelay                         0.0
 MGCPMessageRetryPercent                       0.0
 CallsPerSecond                                0.0

If this number is jumping around from from 6,000-18,000, the platform will not be able to recover itself, and you need to proceed with the rest of the document. If this number is steadily decreasing, you can monitor it to see if it will recover on its own.

Grab packet capture from cn10-inverness-co on eth1, and format into the top 110 hosts.

Log into cn10-inverness-co.suburbanbroadband.net and run the following commands:

# DATE=`date +%Y-%m-%d-%H%M`; sudo timeout 60s /usr/sbin/tcpdump -n -i eth1 port 5060 \
   and dst 64.1.10.130 > /tmp/sbc-$DATE.pcap
 
# echo "ip access-list extended SBC-Security-$DATE"; cat /tmp/sbc-$DATE.pcap \
   | awk '{print $3}' | perl -npe 's/^([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+).*$/$1/g' | sort \
   | uniq -c | sort -n | tail -n 110 | awk '{print " deny ip host "$2" host 64.1.10.130"}' \
   | sort -V  | uniq; echo " permit ip any any";echo '!'; echo "interface vlan80"; \
   echo " ip access-group SBC-Security-$DATE out"; rm -f /tmp/sbc-$DATE.pcap

This will generate a configuration that looks like this:

ip access-list extended SBC-Security-2019-05-26-1728
 deny ip host 12.44.24.2 host 64.1.10.130
 deny ip host 12.70.162.58 host 64.1.10.130
 deny ip host 12.119.159.66 host 64.1.10.130
 deny ip host 24.149.3.162 host 64.1.10.130
 deny ip host 24.196.104.14 host 64.1.10.130
 deny ip host 24.234.156.148 host 64.1.10.130
 deny ip host 35.130.74.43 host 64.1.10.130
 deny ip host 38.76.99.26 host 64.1.10.130
 deny ip host 40.131.51.194 host 64.1.10.130
 deny ip host 47.44.70.163 host 64.1.10.130
 deny ip host 50.234.168.98 host 64.1.10.130
 deny ip host 50.234.168.174 host 64.1.10.130
 deny ip host 63.225.119.2 host 64.1.10.130
 deny ip host 63.248.254.18 host 64.1.10.130
 deny ip host 63.249.33.34 host 64.1.10.130
 deny ip host 63.249.43.58 host 64.1.10.130
 deny ip host 64.6.11.61 host 64.1.10.130
 deny ip host 64.64.154.144 host 64.1.10.130
 deny ip host 64.64.154.205 host 64.1.10.130
 deny ip host 64.92.130.162 host 64.1.10.130
 deny ip host 65.114.218.19 host 64.1.10.130
 deny ip host 65.158.61.2 host 64.1.10.130
 deny ip host 66.160.212.190 host 64.1.10.130
 deny ip host 66.160.219.2 host 64.1.10.130
 deny ip host 66.160.223.26 host 64.1.10.130
 deny ip host 66.160.255.66 host 64.1.10.130
 deny ip host 66.185.12.38 host 64.1.10.130
 deny ip host 66.211.11.196 host 64.1.10.130
 deny ip host 67.202.159.242 host 64.1.10.130
 deny ip host 67.217.11.105 host 64.1.10.130
 deny ip host 67.237.218.163 host 64.1.10.130
 deny ip host 69.169.254.178 host 64.1.10.130
 deny ip host 69.170.67.222 host 64.1.10.130
 deny ip host 69.197.98.22 host 64.1.10.130
 deny ip host 70.166.203.100 host 64.1.10.130
 deny ip host 71.237.0.218 host 64.1.10.130
 deny ip host 72.19.129.42 host 64.1.10.130
 deny ip host 72.19.143.166 host 64.1.10.130
 deny ip host 72.19.143.190 host 64.1.10.130
 deny ip host 72.19.147.229 host 64.1.10.130
 deny ip host 72.19.179.138 host 64.1.10.130
 deny ip host 72.19.183.78 host 64.1.10.130
 deny ip host 72.19.183.86 host 64.1.10.130
 deny ip host 72.19.183.122 host 64.1.10.130
 deny ip host 72.19.184.211 host 64.1.10.130
 deny ip host 72.19.185.122 host 64.1.10.130
 deny ip host 72.164.199.218 host 64.1.10.130
 deny ip host 72.250.209.146 host 64.1.10.130
 deny ip host 72.250.209.147 host 64.1.10.130
 deny ip host 72.250.209.156 host 64.1.10.130
 deny ip host 72.250.212.203 host 64.1.10.130
 deny ip host 72.250.213.133 host 64.1.10.130
 deny ip host 72.250.219.56 host 64.1.10.130
 deny ip host 72.250.220.99 host 64.1.10.130
 deny ip host 72.250.221.41 host 64.1.10.130
 deny ip host 72.250.221.150 host 64.1.10.130
 deny ip host 72.250.221.194 host 64.1.10.130
 deny ip host 72.250.222.164 host 64.1.10.130
 deny ip host 73.8.213.51 host 64.1.10.130
 deny ip host 73.63.8.101 host 64.1.10.130
 deny ip host 74.84.74.74 host 64.1.10.130
 deny ip host 74.205.144.106 host 64.1.10.130
 deny ip host 74.205.144.211 host 64.1.10.130
 deny ip host 74.205.144.218 host 64.1.10.130
 deny ip host 74.205.145.122 host 64.1.10.130
 deny ip host 74.205.146.219 host 64.1.10.130
 deny ip host 74.205.147.42 host 64.1.10.130
 deny ip host 74.205.148.78 host 64.1.10.130
 deny ip host 74.205.148.126 host 64.1.10.130
 deny ip host 74.205.148.171 host 64.1.10.130
 deny ip host 75.87.37.174 host 64.1.10.130
 deny ip host 76.77.241.166 host 64.1.10.130
 deny ip host 96.66.68.142 host 64.1.10.130
 deny ip host 97.64.160.82 host 64.1.10.130
 deny ip host 98.158.33.10 host 64.1.10.130
 deny ip host 98.158.33.26 host 64.1.10.130
 deny ip host 104.201.67.26 host 64.1.10.130
 deny ip host 147.92.49.189 host 64.1.10.130
 deny ip host 162.17.54.34 host 64.1.10.130
 deny ip host 173.198.165.166 host 64.1.10.130
 deny ip host 173.198.166.66 host 64.1.10.130
 deny ip host 173.198.166.70 host 64.1.10.130
 deny ip host 173.225.234.10 host 64.1.10.130
 deny ip host 173.240.87.182 host 64.1.10.130
 deny ip host 173.244.141.22 host 64.1.10.130
 deny ip host 199.19.115.248 host 64.1.10.130
 deny ip host 199.168.68.171 host 64.1.10.130
 deny ip host 199.168.71.118 host 64.1.10.130
 deny ip host 204.28.241.58 host 64.1.10.130
 deny ip host 204.28.241.90 host 64.1.10.130
 deny ip host 204.28.241.150 host 64.1.10.130
 deny ip host 204.28.242.2 host 64.1.10.130
 deny ip host 204.28.242.38 host 64.1.10.130
 deny ip host 204.28.242.42 host 64.1.10.130
 deny ip host 204.28.242.62 host 64.1.10.130
 deny ip host 204.28.253.32 host 64.1.10.130
 deny ip host 204.235.44.3 host 64.1.10.130
 deny ip host 205.170.23.26 host 64.1.10.130
 deny ip host 205.185.94.42 host 64.1.10.130
 deny ip host 205.185.94.238 host 64.1.10.130
 deny ip host 206.248.58.243 host 64.1.10.130
 deny ip host 207.109.154.114 host 64.1.10.130
 deny ip host 208.73.252.226 host 64.1.10.130
 deny ip host 208.81.199.22 host 64.1.10.130
 deny ip host 208.123.252.72 host 64.1.10.130
 deny ip host 209.206.65.254 host 64.1.10.130
 deny ip host 216.73.236.30 host 64.1.10.130
 deny ip host 216.114.45.2 host 64.1.10.130
 deny ip host 216.114.62.146 host 64.1.10.130
 deny ip host 216.228.69.74 host 64.1.10.130
 permit ip any any
!
interface vlan80
 ip access-group SCB-Security-2019-05-26-1728 out

Apply ACL on sw4-inverness-co

Log into sw4-inverness-co.suburbanbroadband.net and run the following command:

w4-inverness-co# conf t
Enter configuration commands, one per line.  End with CNTL/Z.
sw4-inverness-co(config)#

Paste in the ACL generated from the cn10-inverness-co.

After it is pasted in, run the following command:

sw4-inverness-co(config-if)# end
sw4-inverness-co#

Verify om AS1/2 that the RegistrationsPerMinute has dropped

While still logged into AS1 or AS2, verify that the RegistrationsPerMinute count has dropped.

AS_CLI> qcurrent
May 24, 2019 =====================================
 NbOfActiveCalls                               5.0
 SIPSetupSignalDelay                          16.0
 SIPAnswerSignalDelay                          2.0
 RegistrationsPerMinute                      984.0
 SIPMsgRetryToNE::10.1.75.182                  0.0
 SIPMsgRetryToNE::172.16.18.4                  0.0
 SIPMsgRetryToNE::64.1.8.229                   0.0
 SIPMsgRetryToNE::172.16.18.5                  0.0
 SIPMsgRetryToNE::172.16.18.148               83.0
 SIPMsgRetryToNE::10.1.75.183                  0.0
 SIPMsgRetryToNE::172.16.18.3                  0.0
 SIPMsgRetryToNE::10.1.75.181                 83.0
 SIPMsgRetryToNE::172.16.18.130                0.0
 SIPMsgRetryPercentToOther                   100.0
 MGCPDialtoneDelay                             0.0
 MGCPSetupSignalDelay                          0.0
 MGCPAnswerSignalDelay                         0.0
 MGCPMessageRetryPercent                       0.0
 CallsPerSecond                                0.0

This should drop down to a very low number within a couple minutes of applying the ACL. If it has not, check your work.

AS_CLI> qcurrent
May 24, 2019 =====================================
 NbOfActiveCalls                               5.0
 SIPSetupSignalDelay                          16.0
 SIPAnswerSignalDelay                          2.0
 RegistrationsPerMinute                       39.0
 SIPMsgRetryToNE::10.1.75.182                  0.0
 SIPMsgRetryToNE::172.16.18.4                  0.0
 SIPMsgRetryToNE::64.1.8.229                   0.0
 SIPMsgRetryToNE::172.16.18.5                  0.0
 SIPMsgRetryToNE::172.16.18.148               66.0
 SIPMsgRetryToNE::10.1.75.183                  0.0
 SIPMsgRetryToNE::172.16.18.3                  0.0
 SIPMsgRetryToNE::10.1.75.181                 83.0
 SIPMsgRetryToNE::172.16.18.130                0.0
 SIPMsgRetryPercentToOther                    73.0
 MGCPDialtoneDelay                             0.0
 MGCPSetupSignalDelay                          0.0
 MGCPAnswerSignalDelay                         0.0
 MGCPMessageRetryPercent                       0.0
 CallsPerSecond                                0.0

If Enterprise Support team is in Loveland, and 72.19.129.42 is not in the ACL, they can check to see if their phones are working. If it is in the ACL, you can remove that line, and have Enterprise Support retest.

When the RegistrationsPerMinute has dropped to a substantially low number, like 39 shown above, you can start to remove lines from the ACL.

Controlled Registration / Removal of ACL

Log into sw4-inverness-co.suburbanbroadband.com and issue the following commands:

w4-inverness-co# conf t
Enter configuration commands, one per line.  End with CNTL/Z.
sw4-inverness-co(config)# ip access-list extended SBC-Security-2019-05-26-1728

Enter the following commands, 5-10 at a time, while monitoring the per-minute registration rate on AS1/2. If the rate raises, ensure that it is dropping back to less than 200 before continuing with the next set of commands. Total time should be less than 10 minutes.

no 10
no 20
no 30
no 40
no 50
no 60
no 70
no 80
no 90 
no 100
no 110
no 120
no 130
no 140
no 150
no 160
no 170
no 180
no 190
no 200
no 210
no 220
no 230
no 240
no 250
no 260
no 270
no 280
no 290
no 300
no 310
no 320
no 330
no 340
no 350
no 360
no 370
no 380
no 390
no 400
no 410
no 420
no 430
no 440
no 450
no 460
no 470
no 480
no 490
no 500
no 510
no 520
no 530
no 540
no 550
no 560 
no 570
no 580
no 590
no 600
no 610
no 620
no 630
no 640
no 650
no 660
no 670
no 680
no 690
no 700
no 700
no 710
no 720
no 730
no 740
no 750
no 760
no 770
no 780
no 790
no 800
no 810
no 820
no 830
no 840
no 850
no 860
no 870
no 880
no 890
no 900
no 910
no 920
no 930
no 940
no 950
no 960
no 970
no 980
no 990
no 1000
no 1010
no 1020
no 1030
no 1040
no 1050
no 1060
no 1070
no 1080
no 1090
no 1100

After these commands are entered, the ACL should be empty. You can remove the ACL from the config and the interface with the following commands:

interface vlan80
 ip access-group SBC-Security-2019-05-26-1728 out
!
no ip access-list extended SBC-Security-2019-05-26-1728

Finish up by running these commands:

sw4-inverness-co(config)# end
sw4-inverness-co#