My SE device joined an ECC-enabled SE network the first time, so why won't it join again after I leave the network?

Once customers begin using the full SE security suite, complete with ECC/CBKE support, they often observe that after joining a network and doing key establishment, they encounter problems if they leave that network and then later attempt to join it again.

This is normal SE network behavior that has to do with the negotiation of a unique link key upon joining…

You initially come into the PAN with a preconfigured trust center link key between your joining device (we’ll call it a PCT in this explanation, but it could be any SE device [PCT, IPD, Smart Appliance, Meter, etc]) and the ESP, and this key is used to convey the active NWK layer key from the Trust Center (ESP) down to the joining device. In a typical (deployed) SE network, that initially preconfigured link key is also unique per device (not fixed to a global value as may be the case during some testing scenarios) and is derived through an AES MMO hash function employing a combination of the new device’s EUI64 and a variable-length installation code (such as the one found in the EmberZNet releases under tool\certificates\test-install-code.txt). Those pieces of data are then communicated out of band to the utility provider (or whomever administrates the ESP) so that the resulting key can be computed and installed into the ESP’s key table (via emberSetKeyTableEntry API in our stack).

Once the PCT enters the network and authenticates successfully with the ESP, the CBKE process commences, and the initial key is soon replaced with a unique, random, CBKE-based trust center link key to the ESP, which you then use for SE communications thereafter. That results in the ESP’s key table entry for your device now containing the new, CBKE-authorized link key.

However, if you then leave the network, because the Leave mechanism in ZigBee is neither reliable nor terribly secure (see related FAQs at the bottom of this article), the ESP (even if it does hear the leave announcement) doesn’t purge the APS security data (in the link key table) for the PCT when it gets the “Update Device: Device Left” notification in the Trust Center Join Handler callback (although it does purge NWK layer security data such as neighbor entries and frame counters when a neighbor/child node leaves) .

So when your SE PCT goes to join freshly next time, it calls emberSetInitialSecurityState(), which wipes any prior security data to allow for a new join and goes back to using the initial, preconfigured link key (either the global one or some install-code-derived one, typically). However, there’s still an existing key table entry at the ESP that points to the most recent link key (which will be the CBKE-authorized one by now). Thus, the NWK key is now delivered to the joining node encrypted with the CBKE-authorized link key, but the joining node is back to using the non-CBKE-authorized initial key (and has no more knowledge of the CBKE-authorized link key negotiated in the previous session), so the mismatch in link keys used caused an APS layer decryption failure as the NWK key is received, which results in the post-join authentication process failing and the stack failing to come up, citing a link key problem (stack status 0xAF: EMBER_PRECONFIGURED_KEY_REQUIRED).

The solution to this problem is to go back through the usual pre-joining handshake with the ESP, where the PCT’s key table entry is updated to use the initial link key employed by the joining device. (Or, if you are using a global link key as your initial trust center link key, you can simply delete the key table entry for the joining node and let the trust center fall back to using the global link key to communicate with that device.) Note that you could also alleviate the problem by adding custom code to the emberTrustCenterJoinHandler() to have it delete a node’s key table entry when your Trust Center receives an EMBER_DEVICE_LEFT notification for that device, but (1) you can’t count on all SE ESPs behaving in this way, and (2) this notification is based on the unreliable and insecure Leave mechanism (as noted in the related FAQ about notification of joining/leaving devices), so the notification may be missed or falsified and is therefore risky as a way to clear out the prior APS security state for nodes that want to leave and come back.

Once the ESP falls back to using the initial link key expected by the joining device, the NWK key can be passed successfully to the incoming node to allow it to come online (stackStatus 0×90: EMBER_NETWORK_UP) again.

As far as how this replacement/deletion of a key table entry occurs, this will vary according to the implementation of the ESP, but if the ESP supports the command line interface [CLI] supplied by default in Ember’s AppBuilder-based ZCL AppFramework, this supports removal of a key table entry via the serial command “keys delete i”, where “i” is replaced with the index of the key table entry you wish to clear out. (The “keys” command by itself will dump the table so you can check the indexes.) It also supports changing of link key table entries via the “option link …” CLI command. See app\sampleApps.html for a full list of supported CLI commands in our HA/SE application framework.