User:Sfiggins/Broadworks Controlled Registration
From Labrats.us
Jump to navigationJump to search
Verify RegistrationsPerMinute on AS1/2
If the RegistrationsPerMinute count on the active Application server is greater than a couple hundred, the platform will need help recovering.
Log into castrum.risebroadband.com, and then ssh to 172.16.18.3 or 10.1.75.181, depending on which AS is active.
$ ssh castrum.risebroadband.com [sfiggins@castrum ~]$ ssh 172.16.18.3 -l bwadmin bwadmin@172.16.18.3's password: bwadmin@as1.jabvoice.net$ bwcli ====================================================================== BroadWorks Command Line Interface Type HELP for more information ====================================================================== Reading initial CLI command file... AS_CLI>
Then run the qcurrent command to see the current state of the queue.
AS_CLI> qcurrent May 24, 2019 ===================================== NbOfActiveCalls 2.0 SIPSetupSignalDelay 16.0 SIPAnswerSignalDelay 2.0 RegistrationsPerMinute 12821.0 SIPMsgRetryToNE::10.1.75.182 0.0 SIPMsgRetryToNE::172.16.18.4 0.0 SIPMsgRetryToNE::64.1.8.229 0.0 SIPMsgRetryToNE::172.16.18.5 0.0 SIPMsgRetryToNE::172.16.18.148 75.0 SIPMsgRetryToNE::10.1.75.183 0.0 SIPMsgRetryToNE::172.16.18.3 0.0 SIPMsgRetryToNE::10.1.75.181 83.0 SIPMsgRetryToNE::172.16.18.130 0.0 SIPMsgRetryPercentToOther 91.0 MGCPDialtoneDelay 0.0 MGCPSetupSignalDelay 0.0 MGCPAnswerSignalDelay 0.0 MGCPMessageRetryPercent 0.0 CallsPerSecond 0.0
If this number is jumping around from from 6,000-18,000, the platform will not be able to recover itself, and you need to proceed with the rest of the document. If this number is steadily decreasing, you can monitor it to see if it will recover on its own.
Log into the SBC and check the CPU
Log into castrum again, then telnet into the SBC.
[sfiggins@castrum ~]$ telnet 64.1.10.132 Trying 64.1.10.132... Connected to 64.1.10.132. Escape character is '^]'. Password: Denver3820> en Password: Denver3820#
Look a the CPU.
Denver3820# show processes cpu Task Name Task Id Pri Status Total CPU Avg Now -------------- -------- --- ---------- -------------- ----- ----- tSipd 1c8c8144 80 READY 47:15:13.431 8.0 10.3 tNpDmaTx 0d3e59f0 60 PEND 3:13:26.747 0.5 1.0 tFlowGdTmr 0d43b030 62 PEND+T 4:06:15.324 0.6 0.6 tNpDmaRx 0d399220 61 READY 2:41:07.554 0.4 0.5 tMbcd 0347a64c 78 PEND+T 1:00:41.640 0.1 0.1 tIpFrag 17218adc 60 PEND 42:58.169 0.1 0.0 tNpwbNpmRx 0d3999c4 60 READY 29:47.101 0.0 0.0 tAlarm 1cbfc878 60 DELAY 26:57.329 0.0 0.0 BusM A 033460b4 100 READY 25:18.067 0.0 0.0 tNetTask 03350b10 50 PEND 24:47.240 0.0 0.0 tArpMgr 0d45388c 61 PEND 9:44.128 0.0 0.0 tAtcpd 03468c84 75 PEND+T 1:07.423 0.0 0.0 nPCSL_timer 03436cb4 100 DELAY 1:03.407 0.0 0.0 tSysmand 0d19dc50 75 PEND+T 1:02.914 0.0 0.0 tTaskCheck 0342a664 100 DELAY 44.885 0.0 0.0 tAlgd 1c788d2c 80 PEND+T 40.789 0.0 0.0 -------------- -------- --- ---------- -------------- ----- ----- Applications 60:35:47.922 10.3 System 587:19:12 13.3
The SBC should be between 5 and 15% CPU. If this is stable, the SBC will not need further remediation. It is it higher, check it again after the ACL is in place. Without traffic, the CPU on the SBC should be very low.
Additional commands that are of interest:
Denver3820# show sipd realms
11:57:51-57
----- Inbound ----- ---- Outbound ----- -- Latency -- Max
Realm Active Rate ConEx Active Rate ConEx Avg Max Burst
VGtoBSoft I 11 0.4 0 0 0.0 0 0.000 0.000 9
VGtoBSoft_private I 0 0.0 0 11 0.4 0 2.042 16.442 9
bw-as-access I 0 0.0 0 0 0.0 0 0.084 0.590 66
bw-as-core I 0 0.0 0 0 0.0 0 11.710 29.464 66
ils-t7k-access I 0 0.0 0 0 0.0 0 0.000 0.000 0
ils-t7k-core I 0 0.0 0 0 0.0 0 0.000 0.000 0
soho-o365-private I 0 0.0 0 0 0.0 0 0.000 0.000 0
soho-o365-public I 0 0.0 0 0 0.0 0 0.000 0.000 0
soho-um-private I 0 0.0 0 0 0.0 0 0.000 0.000 0
soho-um-public I 0 0.0 0 0 0.0 0 0.000 0.000 0
This will show which realms are in use. It does not seem to track registrations. The above no calls involving the phones.
Denver3820# show sipd realms bw-as-access
11:58:24-30
Realm bw-as-access() [In Service]
-- Period -- -------- Lifetime --------
Active High Total Total PerMax High
Inbound Sessions 0 0 0 259720 44 52
Rate Exceeded - - 0 0 0 -
Num Exceeded - - 0 0 0 -
Burst Rate 0 0 0 0 0 12
Reg Rate Exceeded - - 0 0 0 -
Reg Burst Rate 0 0 0 0 0 0
Outbound Sessions 0 0 0 496348 102 113
Rate Exceeded - - 0 0 0 -
Num Exceeded - - 0 0 0 -
Burst Rate 0 0 0 0 0 63
Reg Rate Exceeded - - 0 0 0 -
Out of Service - - 0 0 0 -
Trans Timeout 0 0 0 0 0 0
Requests Sent - - 9 10193974 1223 -
Requests Complete - - 8 8888161 1197 -
Seizure - - 0 508581 103 -
Answer - - 0 137077 68 -
ASR Exceeded - - 0 0 0 -
Requests Received - - 6818 274595943 17291 -
QoS Major Exceeded - - 0 0 0 -
QoS Critical Exceeded - - 0 0 0 -
Latency=0.609; max=2.482
QoS R-Factor Avg=0.00; max=0.00
You can see that there is not much going on in these counters.
Grab packet capture from cn10-inverness-co on eth1, and format into the top 110 hosts.
Log into cn10-inverness-co.suburbanbroadband.net and run the following commands:
# /usr/local/bin/sbc-acl.sh
This will generate a configuration that looks like this:
! ! - Starting TCPDUMP for 60 seconds. Please wait. ! ! ! Paste the following into sw4-inverness-co configuration. ! ! ip access-list extended SBC-Security-2019-05-26-1838 deny ip host 12.44.24.2 host 64.1.10.130 deny ip host 12.70.162.58 host 64.1.10.130 deny ip host 12.119.159.66 host 64.1.10.130 deny ip host 24.149.3.162 host 64.1.10.130 deny ip host 24.196.104.14 host 64.1.10.130 deny ip host 24.234.156.148 host 64.1.10.130 deny ip host 35.130.74.43 host 64.1.10.130 deny ip host 38.76.99.26 host 64.1.10.130 deny ip host 40.131.51.194 host 64.1.10.130 deny ip host 47.44.70.163 host 64.1.10.130 deny ip host 50.234.168.98 host 64.1.10.130 deny ip host 50.234.168.174 host 64.1.10.130 deny ip host 63.225.119.2 host 64.1.10.130 deny ip host 63.248.254.18 host 64.1.10.130 deny ip host 63.249.33.34 host 64.1.10.130 deny ip host 63.249.43.58 host 64.1.10.130 deny ip host 64.6.11.61 host 64.1.10.130 deny ip host 64.64.154.144 host 64.1.10.130 deny ip host 64.64.154.205 host 64.1.10.130 deny ip host 64.92.130.162 host 64.1.10.130 deny ip host 65.114.218.19 host 64.1.10.130 deny ip host 65.158.61.2 host 64.1.10.130 deny ip host 66.160.212.190 host 64.1.10.130 deny ip host 66.160.219.2 host 64.1.10.130 deny ip host 66.160.223.26 host 64.1.10.130 deny ip host 66.160.255.66 host 64.1.10.130 deny ip host 66.185.8.30 host 64.1.10.130 deny ip host 66.185.12.38 host 64.1.10.130 deny ip host 66.211.11.196 host 64.1.10.130 deny ip host 67.202.159.242 host 64.1.10.130 deny ip host 67.217.11.105 host 64.1.10.130 deny ip host 67.237.218.163 host 64.1.10.130 deny ip host 69.20.190.174 host 64.1.10.130 deny ip host 69.169.254.178 host 64.1.10.130 deny ip host 69.170.67.222 host 64.1.10.130 deny ip host 69.170.95.246 host 64.1.10.130 deny ip host 69.197.98.22 host 64.1.10.130 deny ip host 70.166.203.100 host 64.1.10.130 deny ip host 71.237.0.218 host 64.1.10.130 deny ip host 72.19.143.166 host 64.1.10.130 deny ip host 72.19.143.190 host 64.1.10.130 deny ip host 72.19.147.229 host 64.1.10.130 deny ip host 72.19.179.138 host 64.1.10.130 deny ip host 72.19.183.78 host 64.1.10.130 deny ip host 72.19.183.86 host 64.1.10.130 deny ip host 72.19.183.122 host 64.1.10.130 deny ip host 72.19.184.211 host 64.1.10.130 deny ip host 72.164.199.218 host 64.1.10.130 deny ip host 72.250.209.146 host 64.1.10.130 deny ip host 72.250.209.147 host 64.1.10.130 deny ip host 72.250.209.156 host 64.1.10.130 deny ip host 72.250.212.203 host 64.1.10.130 deny ip host 72.250.213.133 host 64.1.10.130 deny ip host 72.250.219.56 host 64.1.10.130 deny ip host 72.250.220.99 host 64.1.10.130 deny ip host 72.250.221.41 host 64.1.10.130 deny ip host 72.250.221.150 host 64.1.10.130 deny ip host 72.250.221.194 host 64.1.10.130 deny ip host 72.250.222.164 host 64.1.10.130 deny ip host 73.8.213.51 host 64.1.10.130 deny ip host 73.63.8.101 host 64.1.10.130 deny ip host 74.84.74.74 host 64.1.10.130 deny ip host 74.118.151.190 host 64.1.10.130 deny ip host 74.205.144.106 host 64.1.10.130 deny ip host 74.205.144.211 host 64.1.10.130 deny ip host 74.205.144.218 host 64.1.10.130 deny ip host 74.205.145.122 host 64.1.10.130 deny ip host 74.205.146.219 host 64.1.10.130 deny ip host 74.205.147.42 host 64.1.10.130 deny ip host 74.205.148.78 host 64.1.10.130 deny ip host 74.205.148.126 host 64.1.10.130 deny ip host 76.77.241.166 host 64.1.10.130 deny ip host 96.66.68.142 host 64.1.10.130 deny ip host 97.64.160.82 host 64.1.10.130 deny ip host 98.158.33.10 host 64.1.10.130 deny ip host 98.158.33.26 host 64.1.10.130 deny ip host 104.201.67.26 host 64.1.10.130 deny ip host 147.92.49.189 host 64.1.10.130 deny ip host 162.17.54.34 host 64.1.10.130 deny ip host 173.198.165.166 host 64.1.10.130 deny ip host 173.198.166.66 host 64.1.10.130 deny ip host 173.198.166.70 host 64.1.10.130 deny ip host 173.225.234.10 host 64.1.10.130 deny ip host 173.240.87.182 host 64.1.10.130 deny ip host 173.244.141.22 host 64.1.10.130 deny ip host 199.19.115.248 host 64.1.10.130 deny ip host 199.168.68.171 host 64.1.10.130 deny ip host 199.168.71.118 host 64.1.10.130 deny ip host 204.28.241.10 host 64.1.10.130 deny ip host 204.28.241.58 host 64.1.10.130 deny ip host 204.28.241.90 host 64.1.10.130 deny ip host 204.28.241.150 host 64.1.10.130 deny ip host 204.28.242.2 host 64.1.10.130 deny ip host 204.28.242.38 host 64.1.10.130 deny ip host 204.28.242.42 host 64.1.10.130 deny ip host 204.28.242.62 host 64.1.10.130 deny ip host 204.28.253.32 host 64.1.10.130 deny ip host 204.235.44.3 host 64.1.10.130 deny ip host 205.170.23.26 host 64.1.10.130 deny ip host 205.185.94.42 host 64.1.10.130 deny ip host 205.185.94.238 host 64.1.10.130 deny ip host 206.248.58.243 host 64.1.10.130 deny ip host 208.73.252.226 host 64.1.10.130 deny ip host 208.81.199.22 host 64.1.10.130 deny ip host 208.123.252.72 host 64.1.10.130 deny ip host 209.206.65.254 host 64.1.10.130 deny ip host 216.73.236.30 host 64.1.10.130 deny ip host 216.114.45.2 host 64.1.10.130 deny ip host 216.114.62.146 host 64.1.10.130 deny ip host 216.228.69.74 host 64.1.10.130 permit ip any any ! interface vlan80 ip access-group SBC-Security-2019-05-26-1838 out ! ! ! ! ! Stop Pasting at this point ! ! ! ! ! ! Paste the following, a few rows at a time after the platform has stabilized. ! ip access-list extended SBC-Security-2019-05-26-1838 no 10 no 20 no 30 no 40 no 50 no 60 no 70 no 80 no 90 no 100 no 110 no 120 no 130 no 140 no 150 no 160 no 170 no 180 no 190 no 200 no 210 no 220 no 230 no 240 no 250 no 260 no 270 no 280 no 290 no 300 no 310 no 320 no 330 no 340 no 350 no 360 no 370 no 380 no 390 no 400 no 410 no 420 no 430 no 440 no 450 no 460 no 470 no 480 no 490 no 500 no 510 no 520 no 530 no 540 no 550 no 560 no 570 no 580 no 590 no 600 no 610 no 620 no 630 no 640 no 650 no 660 no 670 no 680 no 690 no 700 no 710 no 720 no 730 no 740 no 750 no 760 no 770 no 780 no 790 no 800 no 810 no 820 no 830 no 840 no 850 no 860 no 870 no 880 no 890 no 900 no 910 no 920 no 930 no 940 no 950 no 960 no 970 no 980 no 990 no 1000 no 1010 no 1020 no 1030 no 1040 no 1050 no 1060 no 1070 no 1080 no 1090 no 1100 ! interface vlan80 no ip access-group SBC-Security-2019-05-26-1838 out ! no ip access-list extended SBC-Security-2019-05-26-1838 end
Apply ACL on sw4-inverness-co
Log into sw4-inverness-co.suburbanbroadband.net and run the following command:
w4-inverness-co# conf t Enter configuration commands, one per line. End with CNTL/Z. sw4-inverness-co(config)#
Paste in the ACL generated from the cn10-inverness-co.
After it is pasted in, run the following command:
sw4-inverness-co(config-if)# end sw4-inverness-co#
Verify om AS1/2 that the RegistrationsPerMinute has dropped
While still logged into AS1 or AS2, verify that the RegistrationsPerMinute count has dropped.
AS_CLI> qcurrent May 24, 2019 ===================================== NbOfActiveCalls 5.0 SIPSetupSignalDelay 16.0 SIPAnswerSignalDelay 2.0 RegistrationsPerMinute 984.0 SIPMsgRetryToNE::10.1.75.182 0.0 SIPMsgRetryToNE::172.16.18.4 0.0 SIPMsgRetryToNE::64.1.8.229 0.0 SIPMsgRetryToNE::172.16.18.5 0.0 SIPMsgRetryToNE::172.16.18.148 83.0 SIPMsgRetryToNE::10.1.75.183 0.0 SIPMsgRetryToNE::172.16.18.3 0.0 SIPMsgRetryToNE::10.1.75.181 83.0 SIPMsgRetryToNE::172.16.18.130 0.0 SIPMsgRetryPercentToOther 100.0 MGCPDialtoneDelay 0.0 MGCPSetupSignalDelay 0.0 MGCPAnswerSignalDelay 0.0 MGCPMessageRetryPercent 0.0 CallsPerSecond 0.0
This should drop down to a very low number within a couple minutes of applying the ACL. If it has not, check your work.
AS_CLI> qcurrent May 24, 2019 ===================================== NbOfActiveCalls 5.0 SIPSetupSignalDelay 16.0 SIPAnswerSignalDelay 2.0 RegistrationsPerMinute 39.0 SIPMsgRetryToNE::10.1.75.182 0.0 SIPMsgRetryToNE::172.16.18.4 0.0 SIPMsgRetryToNE::64.1.8.229 0.0 SIPMsgRetryToNE::172.16.18.5 0.0 SIPMsgRetryToNE::172.16.18.148 66.0 SIPMsgRetryToNE::10.1.75.183 0.0 SIPMsgRetryToNE::172.16.18.3 0.0 SIPMsgRetryToNE::10.1.75.181 83.0 SIPMsgRetryToNE::172.16.18.130 0.0 SIPMsgRetryPercentToOther 73.0 MGCPDialtoneDelay 0.0 MGCPSetupSignalDelay 0.0 MGCPAnswerSignalDelay 0.0 MGCPMessageRetryPercent 0.0 CallsPerSecond 0.0
If Enterprise Support team is in Loveland, and 72.19.129.42 is not in the ACL, they can check to see if their phones are working. If it is in the ACL, you can remove that line, and have Enterprise Support retest.
When the RegistrationsPerMinute has dropped to a substantially low number, like 39 shown above, you can start to remove lines from the ACL.
SBC CPU with ACL applied
With the ACL applied, there should be lower little SIP traffic, and the CPU will be low.
Denver3820# show processes cpu Task Name Task Id Pri Status Total CPU Avg Now -------------- -------- --- ---------- -------------- ----- ----- tSipd 1c8c8144 80 READY 47:17:26.151 8.0 2.5 tFlowGdTmr 0d43b030 62 PEND+T 4:06:24.624 0.6 0.6 tNpDmaRx 0d399220 61 READY 2:41:14.421 0.4 0.3 tNpDmaTx 0d3e59f0 60 PEND 3:13:38.455 0.5 0.1 tMbcd 0347a64c 78 PEND+T 1:00:44.814 0.1 0.0 tIpFrag 17218adc 60 PEND 42:58.989 0.1 0.0 tNpwbNpmRx 0d3999c4 60 READY 29:48.228 0.0 0.0 tAlarm 1cbfc878 60 DELAY 26:58.348 0.0 0.0 BusM A 033460b4 100 READY 25:18.960 0.0 0.0 tNetTask 03350b10 50 PEND 24:48.192 0.0 0.0 tArpMgr 0d45388c 61 PEND 9:44.400 0.0 0.0 tAtcpd 03468c84 75 PEND+T 1:07.465 0.0 0.0 nPCSL_timer 03436cb4 100 DELAY 1:03.450 0.0 0.0 tSysmand 0d19dc50 75 PEND+T 1:02.962 0.0 0.0 tTaskCheck 0342a664 100 DELAY 44.914 0.0 0.0 tAlgd 1c788d2c 80 PEND+T 40.815 0.0 0.0 -------------- -------- --- ---------- -------------- ----- ----- Applications 60:38:36.079 10.3 System 587:41:47 4.1
Controlled Registration / Removal of ACL
Log into sw4-inverness-co.suburbanbroadband.com and issue the following commands:
w4-inverness-co# conf t Enter configuration commands, one per line. End with CNTL/Z. sw4-inverness-co(config)# ip access-list extended SBC-Security-2019-05-26-1728
Enter the following commands, 5-10 at a time, while monitoring the per-minute registration rate on AS1/2. If the rate raises, ensure that it is dropping back to less than 200 before continuing with the next set of commands. Total time should be less than 10 minutes.
no 10 no 20 no 30 no 40 no 50 no 60 no 70 no 80 no 90 no 100 no 110 no 120 no 130 no 140 no 150 no 160 no 170 no 180 no 190 no 200 no 210 no 220 no 230 no 240 no 250 no 260 no 270 no 280 no 290 no 300 no 310 no 320 no 330 no 340 no 350 no 360 no 370 no 380 no 390 no 400 no 410 no 420 no 430 no 440 no 450 no 460 no 470 no 480 no 490 no 500 no 510 no 520 no 530 no 540 no 550 no 560 no 570 no 580 no 590 no 600 no 610 no 620 no 630 no 640 no 650 no 660 no 670 no 680 no 690 no 700 no 700 no 710 no 720 no 730 no 740 no 750 no 760 no 770 no 780 no 790 no 800 no 810 no 820 no 830 no 840 no 850 no 860 no 870 no 880 no 890 no 900 no 910 no 920 no 930 no 940 no 950 no 960 no 970 no 980 no 990 no 1000 no 1010 no 1020 no 1030 no 1040 no 1050 no 1060 no 1070 no 1080 no 1090 no 1100
After these commands are entered, the ACL should be empty. You can remove the ACL from the config and the interface with the following commands:
interface vlan80 ip access-group SBC-Security-2019-05-26-1728 out ! no ip access-list extended SBC-Security-2019-05-26-1728
Finish up by running these commands:
sw4-inverness-co(config)# end sw4-inverness-co#
