Firewall Troubleshooting 2: Policy Setting issue
Background
My company is an outsourced call center company, we use our client designed software and client call web-based platform.
ALL the client designed software and call platform need to connect to client site server IP 10.10.10.10.
Most traffics send to Client Site F server is Voice Traffic.
Diagram
Network
We have 4 branches in different locations A, B, C, and D.
Branch A IP: 192.168.50.0/24
Branch B IP: 192.168.52.0/24
Branch C IP: 192.168.54.0/24
Branch D IP: 192.168.56.0/24
Client Site server F IP: 10.10.10.10
HSRP Router IP: 192.168.59.1
FWE gateway IP: 192.168.59.253
All the traffics will go through MPLS Network to firewall FWE.
We use the FEW as a getaway router.
Issues
We monitored that there were intermittent connectivity issues from all sites to Client Sites F. All applications were not able to connect to servers in the client site. The outage last only for a short period.
Troubleshoot
During the network interruption, we did some basic network connection test before we invested this issue. The result as below:
All Branches were not able to ping reach the Client site server.
All Branches were able to reach the internet (8.8.8.8).
Tracert the traffic from All Branches to the Client site server, the traffic was not able to reach the FEW, the pack dropped on HSRP Router IP: 192.168.59.1.
Tracert the traffic from All Branches to the internet, can reach it.
Form this information, we can identify the firewall drop the traffic. The issue should be on the firewall.
Generated the firewall log, There was some denial of traffic incidents on the WatchGuard FW, the log shows as below:
FWDeny, Application identified, pri=4, disp=Deny, policy=Outgoing-00, protocol=50414/tcp, src_ip=192.168.52.21, src_port=50414, dst_ip=10.10.10.10, dst_port=8899, src_intf=1-FWE, dst_intf=2-ClientSite_F, rc=101, pckt_len=1392, ttl=56, pr_info=offset 5 A 2854546271 win 29440, app_id=80, app_name=Web Streaming, app_cat_id=4, app_cat_name=Media streaming services, app_beh_id=4; app_beh_name=Media, 3000-0149
From this log, we can tell the traffic from src_ip=192.168.52.21 to dst_ip=10.10.10.10 got denied, cause it matched the policy rule named: policy=Outgoing-00.
The Deny reason is FW Application Identified Deny, The exact app name is app_name=Web Streaming, app_cat_name= Media Streaming services, app_beh_name=Media.
The Application Identified rule is one of the features under the Application Control function in FW.
So We went to the Firewall policy setting, checked the Policy rule: Policy=Outgoing-00. Under this Policy, we went to the policy application control section, searched Web Streaming under application control, found the action for this app control is drop.
So we can see the Firewall blocked the traffic cause the voice traffic has been identified as web streaming.
Solution
1. Changed the web streaming action to allowed.
2. Put Client site F server IP into Whitelist.
Lesson Learned
1. A firewall log is a powerful tool for troubleshooting
2. The Firewall traffic log is based on each policy, has to manually enable each policy rules log.
Comments
Post a Comment