User:Sfiggins/Broadworks Controlled Registration

From Labrats.us
< User:Sfiggins
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigationJump to search


Verify RegistrationsPerMinute on AS1/2

If the RegistrationsPerMinute count on the active Application server is greater than a couple hundred, the platform will need help recovering.

Log into castrum.risebroadband.com, and then ssh to 172.16.18.3 or 10.1.75.181, depending on which AS is active. 

$ ssh castrum.risebroadband.com
[sfiggins@castrum ~]$ ssh 172.16.18.3 -l bwadmin
bwadmin@172.16.18.3's password: 

bwadmin@as1.jabvoice.net$ bwcli


======================================================================
BroadWorks Command Line Interface
  Type HELP for more information
======================================================================
Reading initial CLI command file...

AS_CLI>

Then run the qcurrent command to see the current state of the queue. 

AS_CLI> qcurrent
May 24, 2019 =====================================
 NbOfActiveCalls                               2.0
 SIPSetupSignalDelay                          16.0
 SIPAnswerSignalDelay                          2.0
 RegistrationsPerMinute                    12821.0
 SIPMsgRetryToNE::10.1.75.182                  0.0
 SIPMsgRetryToNE::172.16.18.4                  0.0
 SIPMsgRetryToNE::64.1.8.229                   0.0
 SIPMsgRetryToNE::172.16.18.5                  0.0
 SIPMsgRetryToNE::172.16.18.148               75.0
 SIPMsgRetryToNE::10.1.75.183                  0.0
 SIPMsgRetryToNE::172.16.18.3                  0.0
 SIPMsgRetryToNE::10.1.75.181                 83.0
 SIPMsgRetryToNE::172.16.18.130                0.0
 SIPMsgRetryPercentToOther                    91.0
 MGCPDialtoneDelay                             0.0
 MGCPSetupSignalDelay                          0.0
 MGCPAnswerSignalDelay                         0.0
 MGCPMessageRetryPercent                       0.0
 CallsPerSecond                                0.0

If this number is jumping around from from 6,000-18,000, the platform will not be able to recover itself, and you need to proceed with the rest of the document. If this number is steadily decreasing, you can monitor it to see if it will recover on its own.

Log into the SBC and check the CPU

Log into castrum again, then telnet into the SBC. 

[sfiggins@castrum ~]$ telnet 64.1.10.132
Trying 64.1.10.132...
Connected to 64.1.10.132.
Escape character is '^]'.

Password: 
Denver3820> en
Password: 
Denver3820#

Look a the CPU. 

Denver3820# show processes cpu
Task Name        Task Id Pri Status          Total CPU   Avg   Now
--------------  -------- --- ---------- -------------- ----- -----
tSipd           1c8c8144  80 READY        47:15:13.431   8.0  10.3
tNpDmaTx        0d3e59f0  60 PEND          3:13:26.747   0.5   1.0
tFlowGdTmr      0d43b030  62 PEND+T        4:06:15.324   0.6   0.6
tNpDmaRx        0d399220  61 READY         2:41:07.554   0.4   0.5
tMbcd           0347a64c  78 PEND+T        1:00:41.640   0.1   0.1
tIpFrag         17218adc  60 PEND            42:58.169   0.1   0.0
tNpwbNpmRx      0d3999c4  60 READY           29:47.101   0.0   0.0
tAlarm          1cbfc878  60 DELAY           26:57.329   0.0   0.0
BusM A          033460b4 100 READY           25:18.067   0.0   0.0
tNetTask        03350b10  50 PEND            24:47.240   0.0   0.0
tArpMgr         0d45388c  61 PEND             9:44.128   0.0   0.0
tAtcpd          03468c84  75 PEND+T           1:07.423   0.0   0.0
nPCSL_timer     03436cb4 100 DELAY            1:03.407   0.0   0.0
tSysmand        0d19dc50  75 PEND+T           1:02.914   0.0   0.0
tTaskCheck      0342a664 100 DELAY              44.885   0.0   0.0
tAlgd           1c788d2c  80 PEND+T             40.789   0.0   0.0
--------------  -------- --- ---------- -------------- ----- -----
Applications                              60:35:47.922  10.3
System                                   587:19:12            13.3

The SBC should be between 5 and 15% CPU. If this is stable, the SBC will not need further remediation. It is it higher, check it again after the ACL is in place. Without traffic, the CPU on the SBC should be very low.

Additional commands that are of interest: 

Denver3820# show sipd realms 
11:57:51-57
                   ----- Inbound -----  ---- Outbound ----- -- Latency --  Max
Realm              Active  Rate  ConEx  Active  Rate  ConEx    Avg    Max Burst 
VGtoBSoft         I    11   0.4      0       0   0.0      0  0.000  0.000     9
VGtoBSoft_private I     0   0.0      0      11   0.4      0  2.042 16.442     9
bw-as-access      I     0   0.0      0       0   0.0      0  0.084  0.590    66
bw-as-core        I     0   0.0      0       0   0.0      0 11.710 29.464    66
ils-t7k-access    I     0   0.0      0       0   0.0      0  0.000  0.000     0
ils-t7k-core      I     0   0.0      0       0   0.0      0  0.000  0.000     0
soho-o365-private I     0   0.0      0       0   0.0      0  0.000  0.000     0
soho-o365-public  I     0   0.0      0       0   0.0      0  0.000  0.000     0
soho-um-private   I     0   0.0      0       0   0.0      0  0.000  0.000     0
soho-um-public    I     0   0.0      0       0   0.0      0  0.000  0.000     0

This will show which realms are in use. It does not seem to track registrations. The above no calls involving the phones. 

Denver3820# show sipd realms bw-as-access
11:58:24-30
Realm bw-as-access() [In Service]
                             -- Period -- -------- Lifetime --------
                   Active    High   Total      Total  PerMax    High
Inbound Sessions        0       0       0     259720      44      52
  Rate Exceeded         -       -       0          0       0       -
  Num Exceeded          -       -       0          0       0       -
  Burst Rate            0       0       0          0       0      12
  Reg Rate Exceeded     -       -       0          0       0       -
  Reg Burst Rate        0       0       0          0       0       0
Outbound Sessions       0       0       0     496348     102     113
  Rate Exceeded         -       -       0          0       0       -
  Num Exceeded          -       -       0          0       0       -
  Burst Rate            0       0       0          0       0      63
  Reg Rate Exceeded     -       -       0          0       0       -
Out of Service          -       -       0          0       0       -
Trans Timeout           0       0       0          0       0       0
Requests Sent           -       -       9   10193974    1223       -
Requests Complete       -       -       8    8888161    1197       -
Seizure                 -       -       0     508581     103       -
Answer                  -       -       0     137077      68       -
  ASR Exceeded          -       -       0          0       0       -
Requests Received       -       -    6818  274595943   17291       -
QoS Major Exceeded      -       -       0          0       0       -
QoS Critical Exceeded   -       -       0          0       0       -
Latency=0.609; max=2.482
QoS R-Factor Avg=0.00; max=0.00

You can see that there is not much going on in these counters.

Grab packet capture from cn10-inverness-co on eth1, and format into the top 110 hosts.

Log into cn10-inverness-co.suburbanbroadband.net and run the following commands: 

# /usr/local/bin/sbc-acl.sh


This will generate a configuration that looks like this: 

!
! - Starting TCPDUMP for 60 seconds.  Please wait.
!
!
! Paste the following into sw4-inverness-co configuration.
!
!
ip access-list extended SBC-Security-2019-05-26-1838
 deny ip host 12.44.24.2 host 64.1.10.130
 deny ip host 12.70.162.58 host 64.1.10.130
 deny ip host 12.119.159.66 host 64.1.10.130
 deny ip host 24.149.3.162 host 64.1.10.130
 deny ip host 24.196.104.14 host 64.1.10.130
 deny ip host 24.234.156.148 host 64.1.10.130
 deny ip host 35.130.74.43 host 64.1.10.130
 deny ip host 38.76.99.26 host 64.1.10.130
 deny ip host 40.131.51.194 host 64.1.10.130
 deny ip host 47.44.70.163 host 64.1.10.130
 deny ip host 50.234.168.98 host 64.1.10.130
 deny ip host 50.234.168.174 host 64.1.10.130
 deny ip host 63.225.119.2 host 64.1.10.130
 deny ip host 63.248.254.18 host 64.1.10.130
 deny ip host 63.249.33.34 host 64.1.10.130
 deny ip host 63.249.43.58 host 64.1.10.130
 deny ip host 64.6.11.61 host 64.1.10.130
 deny ip host 64.64.154.144 host 64.1.10.130
 deny ip host 64.64.154.205 host 64.1.10.130
 deny ip host 64.92.130.162 host 64.1.10.130
 deny ip host 65.114.218.19 host 64.1.10.130
 deny ip host 65.158.61.2 host 64.1.10.130
 deny ip host 66.160.212.190 host 64.1.10.130
 deny ip host 66.160.219.2 host 64.1.10.130
 deny ip host 66.160.223.26 host 64.1.10.130
 deny ip host 66.160.255.66 host 64.1.10.130
 deny ip host 66.185.8.30 host 64.1.10.130
 deny ip host 66.185.12.38 host 64.1.10.130
 deny ip host 66.211.11.196 host 64.1.10.130
 deny ip host 67.202.159.242 host 64.1.10.130
 deny ip host 67.217.11.105 host 64.1.10.130
 deny ip host 67.237.218.163 host 64.1.10.130
 deny ip host 69.20.190.174 host 64.1.10.130
 deny ip host 69.169.254.178 host 64.1.10.130
 deny ip host 69.170.67.222 host 64.1.10.130
 deny ip host 69.170.95.246 host 64.1.10.130
 deny ip host 69.197.98.22 host 64.1.10.130
 deny ip host 70.166.203.100 host 64.1.10.130
 deny ip host 71.237.0.218 host 64.1.10.130
 deny ip host 72.19.143.166 host 64.1.10.130
 deny ip host 72.19.143.190 host 64.1.10.130
 deny ip host 72.19.147.229 host 64.1.10.130
 deny ip host 72.19.179.138 host 64.1.10.130
 deny ip host 72.19.183.78 host 64.1.10.130
 deny ip host 72.19.183.86 host 64.1.10.130
 deny ip host 72.19.183.122 host 64.1.10.130
 deny ip host 72.19.184.211 host 64.1.10.130
 deny ip host 72.164.199.218 host 64.1.10.130
 deny ip host 72.250.209.146 host 64.1.10.130
 deny ip host 72.250.209.147 host 64.1.10.130
 deny ip host 72.250.209.156 host 64.1.10.130
 deny ip host 72.250.212.203 host 64.1.10.130
 deny ip host 72.250.213.133 host 64.1.10.130
 deny ip host 72.250.219.56 host 64.1.10.130
 deny ip host 72.250.220.99 host 64.1.10.130
 deny ip host 72.250.221.41 host 64.1.10.130
 deny ip host 72.250.221.150 host 64.1.10.130
 deny ip host 72.250.221.194 host 64.1.10.130
 deny ip host 72.250.222.164 host 64.1.10.130
 deny ip host 73.8.213.51 host 64.1.10.130
 deny ip host 73.63.8.101 host 64.1.10.130
 deny ip host 74.84.74.74 host 64.1.10.130
 deny ip host 74.118.151.190 host 64.1.10.130
 deny ip host 74.205.144.106 host 64.1.10.130
 deny ip host 74.205.144.211 host 64.1.10.130
 deny ip host 74.205.144.218 host 64.1.10.130
 deny ip host 74.205.145.122 host 64.1.10.130
 deny ip host 74.205.146.219 host 64.1.10.130
 deny ip host 74.205.147.42 host 64.1.10.130
 deny ip host 74.205.148.78 host 64.1.10.130
 deny ip host 74.205.148.126 host 64.1.10.130
 deny ip host 76.77.241.166 host 64.1.10.130
 deny ip host 96.66.68.142 host 64.1.10.130
 deny ip host 97.64.160.82 host 64.1.10.130
 deny ip host 98.158.33.10 host 64.1.10.130
 deny ip host 98.158.33.26 host 64.1.10.130
 deny ip host 104.201.67.26 host 64.1.10.130
 deny ip host 147.92.49.189 host 64.1.10.130
 deny ip host 162.17.54.34 host 64.1.10.130
 deny ip host 173.198.165.166 host 64.1.10.130
 deny ip host 173.198.166.66 host 64.1.10.130
 deny ip host 173.198.166.70 host 64.1.10.130
 deny ip host 173.225.234.10 host 64.1.10.130
 deny ip host 173.240.87.182 host 64.1.10.130
 deny ip host 173.244.141.22 host 64.1.10.130
 deny ip host 199.19.115.248 host 64.1.10.130
 deny ip host 199.168.68.171 host 64.1.10.130
 deny ip host 199.168.71.118 host 64.1.10.130
 deny ip host 204.28.241.10 host 64.1.10.130
 deny ip host 204.28.241.58 host 64.1.10.130
 deny ip host 204.28.241.90 host 64.1.10.130
 deny ip host 204.28.241.150 host 64.1.10.130
 deny ip host 204.28.242.2 host 64.1.10.130
 deny ip host 204.28.242.38 host 64.1.10.130
 deny ip host 204.28.242.42 host 64.1.10.130
 deny ip host 204.28.242.62 host 64.1.10.130
 deny ip host 204.28.253.32 host 64.1.10.130
 deny ip host 204.235.44.3 host 64.1.10.130
 deny ip host 205.170.23.26 host 64.1.10.130
 deny ip host 205.185.94.42 host 64.1.10.130
 deny ip host 205.185.94.238 host 64.1.10.130
 deny ip host 206.248.58.243 host 64.1.10.130
 deny ip host 208.73.252.226 host 64.1.10.130
 deny ip host 208.81.199.22 host 64.1.10.130
 deny ip host 208.123.252.72 host 64.1.10.130
 deny ip host 209.206.65.254 host 64.1.10.130
 deny ip host 216.73.236.30 host 64.1.10.130
 deny ip host 216.114.45.2 host 64.1.10.130
 deny ip host 216.114.62.146 host 64.1.10.130
 deny ip host 216.228.69.74 host 64.1.10.130
 permit ip any any
!
interface vlan80
 ip access-group SBC-Security-2019-05-26-1838 out
!
!
!
!
! Stop Pasting at this point
!
!
!
!
!
! Paste the following, a few rows at a time after the platform has stabilized.
!
ip access-list extended SBC-Security-2019-05-26-1838
 no 10
 no 20
 no 30
 no 40
 no 50
 no 60
 no 70
 no 80
 no 90
 no 100
 no 110
 no 120
 no 130
 no 140
 no 150
 no 160
 no 170
 no 180
 no 190
 no 200
 no 210
 no 220
 no 230
 no 240
 no 250
 no 260
 no 270
 no 280
 no 290
 no 300
 no 310
 no 320
 no 330
 no 340
 no 350
 no 360
 no 370
 no 380
 no 390
 no 400
 no 410
 no 420
 no 430
 no 440
 no 450
 no 460
 no 470
 no 480
 no 490
 no 500
 no 510
 no 520
 no 530
 no 540
 no 550
 no 560
 no 570
 no 580
 no 590
 no 600
 no 610
 no 620
 no 630
 no 640
 no 650
 no 660
 no 670
 no 680
 no 690
 no 700
 no 710
 no 720
 no 730
 no 740
 no 750
 no 760
 no 770
 no 780
 no 790
 no 800
 no 810
 no 820
 no 830
 no 840
 no 850
 no 860
 no 870
 no 880
 no 890
 no 900
 no 910
 no 920
 no 930
 no 940
 no 950
 no 960
 no 970
 no 980
 no 990
 no 1000
 no 1010
 no 1020
 no 1030
 no 1040
 no 1050
 no 1060
 no 1070
 no 1080
 no 1090
 no 1100
!
interface vlan80
 no ip access-group SBC-Security-2019-05-26-1838 out
!
no ip access-list extended SBC-Security-2019-05-26-1838
end

Apply ACL on sw4-inverness-co

Log into sw4-inverness-co.suburbanbroadband.net and run the following command: 

w4-inverness-co# conf t
Enter configuration commands, one per line.  End with CNTL/Z.
sw4-inverness-co(config)#

Paste in the ACL generated from the cn10-inverness-co.

After it is pasted in, run the following command: 

sw4-inverness-co(config-if)# end
sw4-inverness-co#

Verify om AS1/2 that the RegistrationsPerMinute has dropped

While still logged into AS1 or AS2, verify that the RegistrationsPerMinute count has dropped. 

AS_CLI> qcurrent
May 24, 2019 =====================================
 NbOfActiveCalls                               5.0
 SIPSetupSignalDelay                          16.0
 SIPAnswerSignalDelay                          2.0
 RegistrationsPerMinute                      984.0
 SIPMsgRetryToNE::10.1.75.182                  0.0
 SIPMsgRetryToNE::172.16.18.4                  0.0
 SIPMsgRetryToNE::64.1.8.229                   0.0
 SIPMsgRetryToNE::172.16.18.5                  0.0
 SIPMsgRetryToNE::172.16.18.148               83.0
 SIPMsgRetryToNE::10.1.75.183                  0.0
 SIPMsgRetryToNE::172.16.18.3                  0.0
 SIPMsgRetryToNE::10.1.75.181                 83.0
 SIPMsgRetryToNE::172.16.18.130                0.0
 SIPMsgRetryPercentToOther                   100.0
 MGCPDialtoneDelay                             0.0
 MGCPSetupSignalDelay                          0.0
 MGCPAnswerSignalDelay                         0.0
 MGCPMessageRetryPercent                       0.0
 CallsPerSecond                                0.0

This should drop down to a very low number within a couple minutes of applying the ACL. If it has not, check your work. 

AS_CLI> qcurrent
May 24, 2019 =====================================
 NbOfActiveCalls                               5.0
 SIPSetupSignalDelay                          16.0
 SIPAnswerSignalDelay                          2.0
 RegistrationsPerMinute                       39.0
 SIPMsgRetryToNE::10.1.75.182                  0.0
 SIPMsgRetryToNE::172.16.18.4                  0.0
 SIPMsgRetryToNE::64.1.8.229                   0.0
 SIPMsgRetryToNE::172.16.18.5                  0.0
 SIPMsgRetryToNE::172.16.18.148               66.0
 SIPMsgRetryToNE::10.1.75.183                  0.0
 SIPMsgRetryToNE::172.16.18.3                  0.0
 SIPMsgRetryToNE::10.1.75.181                 83.0
 SIPMsgRetryToNE::172.16.18.130                0.0
 SIPMsgRetryPercentToOther                    73.0
 MGCPDialtoneDelay                             0.0
 MGCPSetupSignalDelay                          0.0
 MGCPAnswerSignalDelay                         0.0
 MGCPMessageRetryPercent                       0.0
 CallsPerSecond                                0.0

If Enterprise Support team is in Loveland, and 72.19.129.42 is not in the ACL, they can check to see if their phones are working. If it is in the ACL, you can remove that line, and have Enterprise Support retest.

When the RegistrationsPerMinute has dropped to a substantially low number, like 39 shown above, you can start to remove lines from the ACL.

SBC CPU with ACL applied

With the ACL applied, there should be lower little SIP traffic, and the CPU will be low. 

Denver3820# show processes cpu
Task Name        Task Id Pri Status          Total CPU   Avg   Now
--------------  -------- --- ---------- -------------- ----- -----
tSipd           1c8c8144  80 READY        47:17:26.151   8.0   2.5
tFlowGdTmr      0d43b030  62 PEND+T        4:06:24.624   0.6   0.6
tNpDmaRx        0d399220  61 READY         2:41:14.421   0.4   0.3
tNpDmaTx        0d3e59f0  60 PEND          3:13:38.455   0.5   0.1
tMbcd           0347a64c  78 PEND+T        1:00:44.814   0.1   0.0
tIpFrag         17218adc  60 PEND            42:58.989   0.1   0.0
tNpwbNpmRx      0d3999c4  60 READY           29:48.228   0.0   0.0
tAlarm          1cbfc878  60 DELAY           26:58.348   0.0   0.0
BusM A          033460b4 100 READY           25:18.960   0.0   0.0
tNetTask        03350b10  50 PEND            24:48.192   0.0   0.0
tArpMgr         0d45388c  61 PEND             9:44.400   0.0   0.0
tAtcpd          03468c84  75 PEND+T           1:07.465   0.0   0.0
nPCSL_timer     03436cb4 100 DELAY            1:03.450   0.0   0.0
tSysmand        0d19dc50  75 PEND+T           1:02.962   0.0   0.0
tTaskCheck      0342a664 100 DELAY              44.914   0.0   0.0
tAlgd           1c788d2c  80 PEND+T             40.815   0.0   0.0
--------------  -------- --- ---------- -------------- ----- -----
Applications                              60:38:36.079  10.3
System                                   587:41:47             4.1

Controlled Registration / Removal of ACL

Log into sw4-inverness-co.suburbanbroadband.com and issue the following commands: 

w4-inverness-co# conf t
Enter configuration commands, one per line.  End with CNTL/Z.
sw4-inverness-co(config)# ip access-list extended SBC-Security-2019-05-26-1728

Enter the following commands, 5-10 at a time, while monitoring the per-minute registration rate on AS1/2. If the rate raises, ensure that it is dropping back to less than 200 before continuing with the next set of commands. Total time should be less than 10 minutes. 

no 10
no 20
no 30
no 40
no 50
no 60
no 70
no 80
no 90 
no 100
no 110
no 120
no 130
no 140
no 150
no 160
no 170
no 180
no 190
no 200
no 210
no 220
no 230
no 240
no 250
no 260
no 270
no 280
no 290
no 300
no 310
no 320
no 330
no 340
no 350
no 360
no 370
no 380
no 390
no 400
no 410
no 420
no 430
no 440
no 450
no 460
no 470
no 480
no 490
no 500
no 510
no 520
no 530
no 540
no 550
no 560 
no 570
no 580
no 590
no 600
no 610
no 620
no 630
no 640
no 650
no 660
no 670
no 680
no 690
no 700
no 700
no 710
no 720
no 730
no 740
no 750
no 760
no 770
no 780
no 790
no 800
no 810
no 820
no 830
no 840
no 850
no 860
no 870
no 880
no 890
no 900
no 910
no 920
no 930
no 940
no 950
no 960
no 970
no 980
no 990
no 1000
no 1010
no 1020
no 1030
no 1040
no 1050
no 1060
no 1070
no 1080
no 1090
no 1100

After these commands are entered, the ACL should be empty. You can remove the ACL from the config and the interface with the following commands: 

interface vlan80
 ip access-group SBC-Security-2019-05-26-1728 out
!
no ip access-list extended SBC-Security-2019-05-26-1728

Finish up by running these commands: 

sw4-inverness-co(config)# end
sw4-inverness-co#
Retrieved from "https://www.labrats.us//wiki/index.php?title=User:Sfiggins/Broadworks_Controlled_Registration&oldid=169"