Owner: User #14587667

Bumble Test Timeline

6/19/2015

• Installed HP MSR 4080 chassis in Rack 2/Pod 4.
• Installed MPU-100 (x2), and SPU-100 cards into MSR 4080 chassis.

6/23/2015

• Racked and Setup console access for Cisco 2911 and 3845 in Rack 2 to use for Bumble testing.
  • Configure E1 cards in 2911: 

    HWIC0 (2E1): card type e1 0 0
    HWIC1 (1E1): card type e1 0 1

• Created RANCID project for Bumble network devices.  Added 2911, 3845.
  • Added static route to 10.9.8.0/24
• Setup MSR
  • Configure static IP, static route to 10.9.8.0/24, NTPNetwork Time Protocol client

6/24/2015

• Requested POS card and E1 cables (MSR (DB15) <-> Cisco (RJ48C)) from User #75979.

7/10/2015

• Installed 4-port E1 card in MSR
• Racked Cisco 3825 (from old cabinet) and 2811 (from NDBNetwork Devices Branch lab).  Configured console server and connected ethernet cables to TOR2 gi1/0/5-6.
• Backed up original 2811 config and connected to network (IP = 172.20.100.226)

7/16/2015

• Configured MSR to Cisco E1 links

7/17/2015

• Configured MSR to Cisco Mutilink ppp lines (3E1).
• Installed 1E1 HWIC into 2811.  Upon reboot, the 2811 gives a memory error and will not boot.

7/18/2015

• Installed 2E1 HWIC in 3845. Removed old 1E1 HWIC.
• Configured 4E1 on 3845 (IP = 8.10.1.30) and MSR.
• Determine OSPFOpen Shortest Path First configuration in target config.

osfp 1

  import-route direct                     # Redistribute networks of local active interfaces

  import-route static                     #Redistribute static routes

area 0.0.0.0

  network 192.168.168.4 0.0.0.3         #Enable OSPFOpen Shortest Path First on the interface attached to this network

7/30/2015

• Configured SNMPSimple Network Management Protocol on MSR and SolarWinds.
• Configured 3825 (NE40 surrogate).

7/31/2015

• Configured netstream on MSR.  Confirmed int gi2/0/0 is being monitored by solardwinds.
• Completed SNMPSimple Network Management Protocol configuration on MSR.
• Configured lo0, 1, and 2 on 3825.
• Configured OSPFOpen Shortest Path First on 3825 and MSR.

8/3/2015

• Configured loopback interfaces on 3845, 2911, 3825.
• Configured ACLs on MSR.

8/4/2015

• Add NM ESW card to 3825 and configure host switchports. Advertised XXX.XXX.XXX.X (DNIC-RNET-192-100-100[US])/24 and XXX.X.XXX.X (JUMPSTART-5[US])/24 on OSPF.
• Created host VMs for Attack side of network.
• Spoke with GYSON regarding current test network setup.

8/5/2015

• Installed PoS card in ASR1002.
• Configured Mgt int on ASRAzure Site Recovery and added to RANCID. Configured NTPNetwork Time Protocol client.
• Configured loopbacks on ASR.
• Backup and replace 2811.  Replaced CF card.  Install and configure E1 card on new 2811.

8/10/2015

• Configured DelaySim using tc

  • ### Since the following parameters are applied on both interfaces, the effective properties are as follows:
    # Delay: 5s +/- 1s with a normal distribution (default)
    # Packet Loss: 50% of packets will be randomly dropped and each successive probability depends by 25% on the last one
    # Packet Duplication: 3%
    # Packet Corruption: 1% of all traffic
    # Packet Reordering: first 25% of packets (with a correlation of 50%) will get sent immediately
    sudo tc qdisc add dev eth0 root netem delay 2.5s 0.5s loss 25% 25% duplicate 1.5% corrupt 0.5% reorder 12.5% 50%
    sudo tc qdisc add dev eth1 root netem delay 2.5s 0.5s loss 25% 25% duplicate 1.5% corrupt 0.5% reorder 12.5% 50%

• Created Seeds VMs for 2911,2811,3845, and ASR.
• Configured and tested POS interfaces on ASRAzure Site Recovery (had to create 2 VRFs)

8/11/2015

• Configured TOR2 switch and vCenter with VLANs for IXIA connection
• Added FastEthernet HWIC to 2851 to used for Seeds/IXIA network.
• Upgraded 2851 IOSApple operating system for small devices to 15.0 in order to support Fa HWIC

8/12/2015

• Created/configured VMs for Bumble testing
• Updated MSR with target changes (specifically ACLs)

8/13/2015

• Configure QoS policy on MSR.
• Upgraded Cisco 3825 IOSApple operating system for small devices to 15.1.  Installed Fa HWIC in 3825 (.111 and .112).
• Configured 3825
• Added 2 web servers behind 3825.

8/14/2015

• Unpack/rack H3C
• Ran patch cables from Core Rack (R3) to Rack 2 for Ixia testing.

8/17/2015

• Configure Windows Server 2003 VMVirtual Machine IP settings.  Setup FTPFile Transfer Protocol server.

8/21/2015

• Installed Bumble on MSR.
• CT 2.4 which was delieved is not compatible with current ICON image (Debian 7.8)

8/24/2015

• Downloaded and installed Ubuntu Server 10.04. This is the library version that was delievered with Bumble.
• Performed smoke test of Bumble.
• Added survey commands:

  • tp survey create 192.0.0.0 255.0.0.0 0 0.0.0.0 0.0.0.0 0 icmp -en

  • tp survey create 192.0.0.0 255.0.0.0 0 0.0.0.0 0.0.0.0 0 tcp -en

• Added redir rules:
  • redir create 10.1.243.2 255.255.255.255 0 0 XXX.X.XXX.XXX (JUMPSTART-5[US]) 255.255.255.255 0 0 tcp 10.1.243.2 0 XXX.X.XXX.XXX (JUMPSTART-5[US]) 0 0 -en
• Potential bug:
  • Add survey and redir rule.
    • redir create 10.1.243.2 255.255.255.255 0 0 XXX.X.XXX.XXX (JUMPSTART-5[US]) 255.255.255.255 0 0 tcp 10.1.243.2 0 XXX.X.XXX.XXX (JUMPSTART-5[US]) 0 0 -en
    • tp survey create XXX.X.XXX.X (JUMPSTART-5[US]) 255.255.255.0 0 0.0.0.0 0.0.0.0 0 tcp -en
  • Test redir.
  • Run Ixia test.
  • Test redir.  If redir fails, then re-try this sequence, but this time without adding the survey rules.  I wonder if the survey transaction log is filling causing the redir rule to stop working.  The translog stopped at index 4000 (is this coincidence or is this the size of the buffer?).

8/25/2015

• Setup IXIA for testing through multiple interfaces.

8/26/2015

• Tweaked IXIA to test through MSR with endpoints on neighboring devices.
• Setup and began 72 hour test through all neighbors.  Score was active on DUT, but no redir/survey rules were created.

8/31/2015

9/1/2015

• Wrote expect scripts to install patch.bin file and add a redir rule.

9/3/2015

• Phone Call with User #75980:
  • score requires a minimum MTUMaximum Transmission Unit of 480Bytes for return traffic (from implant to CTCounter Terrorism).  
  • There is an undefined parameter in the ilm.txt file to configure the the Commx Rx size (CommsMaxIpRxSize)
• Performed individual commands while running tcpdump to capture return packets from implant to route.  These captures depict each packet size for the UDPUser Datagram Protocol packets coming back in response to C2.

9/4/2015

• Modified expect scripts to automate the following: reboot MSR, install implant, and run score survey test.

9/25/2015

• Started 72-hour IXIA enterprise profile for MSR
• Added latency to delay simulator
  • sudo tc qdisc add dev eth0 root netem delay 2.5s 0.5s loss 25% 25% duplicate 2% corrupt 3% reorder 25% 50%   (also added same command to eth1)
• Started fresh Atk putty window and ran command: date && sudo time ./reboot_target.exp && sudo ./install_and_run.sh
• Script ran all weekend and all tests were successful, except an ilm connect error was encountered when attempting to cleanup the redir rule after the last test.
• The results can be found in the folder: \\10.9.8.21\Share\Testing\JQJARGON\Test_Results\2015-09-25

9/28/2015

• Inspected wireshark data for test run over the weekend (see previous entry on this page) and noticed several TCPTransport Control Protocol retransmissions, DUPs, and a few Telnet RST. It seems that the RSTReal-time Transport Protocol packets are sent by the MSR to terminate the telnet session.

9/30/2015

• Temporarily add routes to ASRAzure Site Recovery and TR-CoreSwx to allow VMVirtual Machine #3 (XX.XX.XXX.X (EARTHLINKBUSINESS-BLK[US])) access to DAB proxy so that nmap can be installed.
  • Add route to TR CoreSwx:
    • ip route XX.XX.XXX.X (EARTHLINKBUSINESS-BLK[US]) 255.255.255.0 172.20.100.234 name Temp_For_Bumble_VM
  • Add route to Cisco ASR:
    • ip route 0.0.0.0 0.0.0.0 XXX.XXX.XX.X (EU-MAP-5[US])
  • Setup apt-get proxy settings on VMVirtual Machine #4.
  • Remove routes from routers when complete.

10/1/2015

• Started Tests 1-4 with latency.  Ran in 6.5 hrs.  All tests passed.  Results: Bumble-wLatency_ICON5_2015-10-01_134059_window1.log
• Interesting output: 

  [192.168.168.5]> module show

  [Success]
  Name Version Boot Persistence State Address Path
  paw 7.0.0-0.9.1-0.3 unk active
  PLI 1.0.0-0.9.1-0.1 unk STOPPED
  score 2.5.99-0.9.1-0.2 unk active
  ************ Success ************
  [module show]

  [192.168.168.5]> redir show
  [Success]
  *** error - no (or incomplete) response from remote 0
  *** error - no (or incomplete) response from remote 0
  no filters are defined
  ************ Success ************
  [redir show]

• pcap file: test_output/151001_134345-tcpdump.pcap
• This test resulted in BUM-8.

10/6/2015

• Installed Xplico server and Analyzed pcap (from BUM-8).
  • Xplico decodes C2 and categorizes it as "Undecoded TCP-UDP" traffic.  
  • SNMP is also categorized as "Undecoded TCP-UDP" traffic.
  • The telnet sessions used during PBD installation is also clearly visible.  This shows all of the telnet commands that are issued (turning on/off Info-Center, ftp of patch file, etc).

10/9/2015

• Started all tests at ~9:55AM (run_x_times.sh) to run 15 times over the weekend. Latency enabled.
• Started IXIA ~10:15AM.

10/23/2015

• Friday afternoon User #75978 started MSR performance baseline.  This is getting captured by solarwinds.  Take baseline measurements and snapshots on Monday before proceeding with Bumble 3.1 testing.
• Bumble 3.1 testing will begin Monday.

10/27/2015

• Installed Bumble 3.1
• All expect scripts used to automate testing stopped working.  New version of CT/ilm seems to display output differently than previous CTCounter Terrorism version.

10/28/2015

• Troubleshooted and modified expect scripts to resume automated testing.
  • All expect scripts working.  Ran run_x_times.sh 3 times without failing.
• Ran expect scripts w/o latency 60 times overnight (install once).
  • Failed after looping 16 times.  Received "ilm is an unsupported command or group!" BUM-13

10/29/2015

• Modified expect scripts to account for potential ilm connection error.  Will not attempt to reconnect.

10/30/2015

• Sucessfully ran Bumble 3.1 eighty (80) times overnight.
  • Results: /home/user1/testscripts/batch_results/20151030_1020-all_results
• Started test to run over the weekend (15 times)
  • Added latency.  Includes new "ct_pt5-savetrans_ftp.exp" test. Not running ixia.

11/3/2015

• Ran tests overnight.  Took 4.5 hours to run tests 1-4/
  • All tests ran successfully
  • Tets Results:  /home/user1/testscripts/batch_results/20151104_1120-all_results

11/6/2015

• Ran IXIA test.  Saved graphs for CPU/Memory usage and interface throughput.

11/9/2015

• Investigated BUM-15.  Sent User #75981 log files.

11/10/2015

• Re-installed PBD after exhausting all investigation into BUM-15.
• Ran automated tests 10 times with latency.
  • ./ct_pt2-redir_v2.exp failed 2 times
  • ./confirm_transtable_file_exists.exp failed 3 times
  • ./confirm_transtable_ftp_exists.exp failed 5 times

11/12/2015

• Configured the IXIA to perform Stack Scrambling (fuzzing).
• Turned off latency. Ran tests 80 times overnight.  Did not re-install since swiffer was stopped.
• BATCH_DIR=/home/user1/testscripts/batch_results/20151113_1238-all_results
• ICON_LOG=Bumble_ICON6_2015-11-12_142601_window1.log

11/13/2015

• Re-installed implant (since swiffer had been removed).  Turned on latency, Ran tests 50 times over the weekend.  IXIA still running Stack Scrambler (fuzzing).
• Tests ran successfully 15 times.  The only errors were with verifying the transtable was saved to the ftp server.  However, I am certain there is an issue with the script because when I run the command manually, the transtable is successfully saved to the ftp server.  Addtionally there was an error with redirection (2 out of 15).  Again, I believe this would be an issue with the scripts not being very resilient when encountering latency.

11/18/2015

• Started IXIA Stack Scrambler and ran test with latency 3 times overnight.

Attachments:

• image2015-10-6 10:35:44.png

• image2015-9-28 8:44:11.png

• image2015-9-1 13:51:18.png

Previous versions:

| 1 empty | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | 100 |