Key Management for NetBackup

From World History Wiki
Jump to: navigation, search
World History Wiki is Brought to you by:
S.J.'s Adventures

These documents are for entertainment purposes only. Use at your own risk. This site is in no way affiliated with Veritas or NetBackup, and any trademarks are those of their respective companies.

If you need assistance with anything NetBackup and/or Disaster Recovery related, You can find me on linkedIn.

The NetBackup encryption Key Management Service (KMS) is simple to deploy if you have tape drives that support standard T10 SCSI security protocols. The difficult part is defining requirements, process, and procedures to insure keys remain secure, without compromising restoration capabilities.

Requirements may include things such as:

  1. All physical tapes going off-site, or that are handled in any way by a third party, are to have the data on them protected with encryption using AES_256 Cipher types.
  2. On-site "Archive" should NOT be encrypted to insure they are fully recoverable for an indefinite period of time, but must be stored in a secured and climate controlled on-site location and not handled by any 3rd party vendors.
  3. Keys are to be replaced or rotated on a regular schedule, such as Monthly, Quarterly, or Yearly.
  4. All encrypted tapes must be able to be restored from for the life of the data retention.
    Encryption Key rotation policies must take this into consideration.
  5. NetBackup's built in Key Management Service (KMS) is to be use for key management of LTO tape drive hardware encryption
    This insures proper tracking of what encryption key was used for which data set on which tapes.
    Details can be found in the Veritas NetBackup™ Security and Encryption Guide Chapter ? (page ???).
  6. Pass-phrases are to be used to insure the ability to rebuild the Key management database.
  7. Information Security is to create and maintain the pass phrase(s), and their associated "Key Names" and "Key Tags" to insure the KMS database can be re-built if necessary.
  8. The KMS database files, or pass-phrases should never be given to out-side parties, unless the media they are on is encrypted.
    See the DR readiness section for more details.
  9. The Description of an individual key is to indicate when the key was or is meant to be in use.

NOTE: NetBackup environments prior to version 7.1 had limitations and bugs with the KMS capabilities.

Key Management Considerations

To help improve security of the data, it is an industrial standard to regularly rotate encryption keys. This also insures expired data on outdated tapes can no longer be read when they are disposed of.

Things to Consider

Since only 10 keys are allowed per Key Group (and subsequently per Volume Pool), it's important to determine how many keys will be needed, and what rotation schedule can be used, to meet the rotation criteria for that group and it's associated Volume Pool.

  • Adding a new key too often may cause old keys to be retired too quickly, such that old data on older tapes will not be able to be recovered, despite not having reached their expiration date.
  • Adding new keys too infrequently poses a potential security risk, and may create inconsistencies with the expiration of data on older tapes that are to be retired.
  • The "ENCR_" prefix on Volume Pools is essential to tell the "BPTM" process that the tapes is to be encrypted. The Volume Pool name is then provided to "KMS", and "KMS" identifies it as an exact match to a Key Group name to pick the active key to use for backups.
    This requires that Key Group names also follow standard volume pool naming conventions.
Key Rotation and Usage Policy


  • If your maximum retention is 10 years, and each group/pool can have up to 10 keys, the key rotations should occur yearly, with any keys older then 10 years being deleted from the database to make room for newer keys.
    • In environments were the retention is shorter, more frequent rotations can occur.
    • A key must be put into a "Deprecated" or "Terminated" state no later then 12 months after the maximum retention for data/tapes in the environment; based on when the key was last in an "active" state.
    • A key should only be deleted to make room for a new key to be created within the group, or to insure retired media can no longer be read.
  • "Offsite Archive" tapes that are kept "indefinitely" should be created with a key or set of keys that are to NEVER be deleted, nor put into a "Deprecated" or "Terminated" state.
    • Such keys should still be rotated on a yearly basis, but after a maximum of 10 years, the keys will have to be re-used, as they cannot be deleted.
      In this case the Key's description should contain date ranges that depict the time frames during which the key was used.
    • No other keys should be re-instated as "Active" after having been removed from an "Active" state.
  • Tapes retained on-site, in a secure location, do not need to be encrypted.
    • "Archive" tapes that are stored on-site should NEVER be encrypted to insure recoverability.
    • On-site tapes should be stored in a secured and climate controlled location.
  • Tapes sent off-site, or that are handled by a third party at any time, MUST be encrypted.
  • Catalog Backup tapes are an exception and should not be encrypted to insure the catalog can be easily and quickly restored.
    • Since the Catalog is merely a listing of what was backed up it does not contain any highly sensitive data within it.
    • NOTE: The KMS data files are NOT included in the NetBackup Catalog backups, and should be backed up separately with special consideration.

Basic Commands

  • Create New Key Group
    nbkmsutil -creatkg -kgname ENCR_VolPool
  • Create New Key and make it the active key for it's Key Group
    nbkmsutil -createkey -kgname ENCR_VolPool -keyname keyMM-DD-YYYYtoMM-DD-YYYY -activate -desc "Offsite tapes from MM-DD-YYYY to MM-DD-YYYY"
  • List Current Keys within a Key Group with details on each key
    nbkmsutil -listkeys -kgname ENCR_VolPool
  • Change the State of a specific Key
    nbkmsutil -modifykey -keyname keyMM-DD-YYYYtoMM-DD-YYYY -kgname ENCR_VolPool -state STATE
    Where "STATE" is either: active, inactive, deprecated, or terminated.
  • Delete a Key
    nbkmsutil -deletekey -keyname Offsite_07-05-2011 -kgname ENCR_VolPool
    Only keys in a "terminated" or "prelive" state can be deleted.
  • Recover a Key that was previously deleted or to re-create a lost database
    nbkmsutil -recoverkey -kgname ENCR_VolPool -keyname keyMM-DD-YYYYtoMM-DD-YYYY -tag <Key Tag>
    This also requires using the same pass-phrase as the original key.
    Key Tags can be obtained by using the command to "Listing Current Keys".

For additional details on Recovering keys and the key management Database see the DR Procedures and Policies of this document.

Any time a Key is added, removed, updated, or other actions taken that modifies the KMS database, you must make sure the KMS database is properly backed up, and that Information Security is also updating their records.

NetBackup Key Management Service

About Keys and Key Groups

  • Key Group definitions consists of the following:
    • Name - Given to a key group. Should be unique within the keystore. Renaming of the key group is supported if the new name is unique within the keystore.
      Must match the associated Volume Pool name.
    • Tag - Unique key group identifier (not mutable).
    • Cipher - Supported cipher. All keys belonging to this key group are created with this cipher in mind (not mutable).
    • Description - Any description (mutable).
    • Creation Time - Time of creation of this key group (not mutable).
    • Last Modification Time - Time of last modification to any of the mutable attributes (not mutable).

  • Key Records are defined as and with:
    • Encryption Key - This key is given to the tape drive.
    • Encryption Key Tag - This tag is the identifier for the encryption key, and is used by NetBackup to identify which Key a Backup Image was created with (especially important for restore functionality).
    • Record State - Each of the key records has a state. The states are prelive, active, inactive, deprecated, and terminated.
    • Metadata - Metadata includes logical name, creation date, modification date, and description.
    • Groups - All key records must belong to a group.
      A key group can only have one active state key record at any time.
      NetBackup 7.1 supports 100 key groups, and 10 encryption keys are allowed per key group.
    • Description - A modifiable free form field to input additional information. Especially helpful for describing when the key was or is meant to be in use.

  • The following key record states are available:
    • Prelive - indicates that the record has been created, but has not been used (can be deleted).
    • Active - indicates that the record and key are used for encryption and decryption.
    • Inactive - indicates that the record and key cannot be used for encryption but can be used for decryption.
    • Deprecated - indicates that the record cannot be used for encryption or decryption.
    • Terminated - indicates that the record cannot be used for encryption or decryption (can be deleted).


For details see the Veritas NetBackup™ Security and Encryption Guide page ???.

Below is an example procedure to implement and maintain NetBackup's KMS.

Installing the KMS database

Some steps in this section should be completed by a representative from Information Security

  1. A NetBackup Administrator logs into the target environment and runs, as root, the command:
    nbkms -createemptydb

  2. An Information Security Administrator enters a passphrase for the Host Master Key (HMK), as well as an ID for the HMK.
    The Information Security Administrator also enters a passphrase and ID for the key protection key (KPK).
    NOTE: Failing to enter a passphrase will result in a randomly generated key that cannot be re-produced
    1. The Information Security Administrator Documents the passpharse and ID in a secure location, and provides the ID to the NetBackup Administrator to use for future reference.
    2. The Information Security Administrator also provides the NetBackup Team with instructions on how to help the Information Security team locate and recognize the correct key for the correct environment.
    3. The NetBackup Administrator documents the ID and instructions in a secure location.
  3. The NetBackup Administrator uses the grep command to ensure that the service has started:
    ps -ef | grep nbkms
    1. If it is not started, The NetBackup Administrator starts the service by running the following command as root:

Create the "ENCR_VolPool" Key Group and Key(s)

Additional Key Groups may be created in the future if the need arises.
Some steps in this section are to be completed by a representative from Information Security.

NOTE: Key Group names must be an identical match to the Volume Pool name, so following standard volume pool naming conventions is required; with the additional prefix of "ENCR_" that is required by KMS.

  1. For tapes going "Offsite" the Key Group and Volume Pool name could be "ENCR_Offsite"
    If Archive tapes are also sent off-site, then an additional Group and Pool could be created using the name of "ENCR_Offsite_Archive"
    The command to run is:
    nbkmsutil -createkg -kgname ENCR_Offsite
    If This is for China, you must specify a different cypher from the default:
    nbkmsutil -createkg -kgname ENCR_Offsite -cipher AES_128
  2. Create a key record by using the "-createkey" option:
    nbkmsutil -createkey -kgname ENCR_Offsite -keyname keyMM-DD-YYYYtoMM-DD-YYYY -activate -desc "Offsite tapes from MM-DD-YYYY to MM-DD-YYYY"
    The "Keyname" of "keyMM-DD-YYYYtoMM-DD-YYYY" and the "MM-DD-YYYY" in the description is used to designate when the key was in an "activate" state to assist with key rotations.
    1. A Information Security Administrator must provide the passphrase when prompted. It will ask for it twice as a confirmation.
      Failure to provide a passphrase may result in a randomly generated key that cannot be reproduced.
    2. You can create additional key records using the same command, but do not use the "-activate" switch except on the key you want to become the active key upon creation.
      Only one Key can be active, per group, at a given time.

  3. The Information Security Administrator Documents the Passpharse, Key Tag, and Key Name in a secure location.
    1. Run the following command to obtain the necessary information (change the Key Group name as needed).
      nbkmsutil -listkeys -kgname ENCR_Offsite
      It would be wise to save the output of this command.
    2. The Information Security Administrator also provides the NetBackup Team with instructions on how to help the Information Security team locate and recognize the correct key(s) for the correct environment.
    3. The NetBackup Administrator documents the Key Name, Key Tag, and instructions in a secure location.

All of the following steps can be completed by NetBackup Administrator, but some may require informing the Information Security team so they can update their records.

Create and use the "ENCR_Offsite" Volume Pool

  1. Under "Media" management, create a new Volume Pool named "ENCR_Offsite"
    If you also have Archive tapes going off-site, also create an "ENCR_Offsite_Archive" pool.
    1. The New pool should mimic current settings of the existing "Offsite" pool.
      And the "Offsite_Archive" pool if applicable.
    2. See Volume Naming and Groups for NetBackup as a reference.

  2. Under "Policies" management, modify any policies previously going to the "Offsite" pool to now go to the "ENCR_Offsite" pool.
    1. These updates should only be done to Policies and other processes that write data to physical tape.
    2. If there are configurations to use the "Offiste_Archive" pool, they should also be updated appropriately.
    3. Be sure to also check if any schedules are configured to overwrite the Policy specified Volume Pool, and update them as needed.
    4. Do NOT configure the Catalog Backup (even if going off-site) or on-site "Archive" tapes to go to an Encrypted Volume Pool

  3. Under "Vault Management" check for any Vault Profiles that are configured for Duplication to "Offsite" tapes.
    See Vault Configuration for NetBackup as a reference.

  4. Under "Storage Lifecycle Policies" update and policies that create "Offsite" tapes to use the "ENCR_Offsite" Volume Pool instead.
    See Understanding NetBackup Storage Lifecycle Policies as a reference.

Modifying a Key

Change the state of a key [ and optionally it's description ]

nbkmsutil -modifykey -keyname <key_name> -kgname <key_group_name> [ -state <new_state> ] [ -desc <new_description> ]

Delete a key (key must first be in a "terminated" or "prelive" state)

nbkmsutil -deletekey -keyname <key_name> -kgname <key_group_name>

Any time a Key is added, removed, updated, or other actions taken that modifies the KMS database, you must make sure the KMS database is properly backed up, and that Information Security is informed so they can also update their records.

DR readiness

In order to insure you can restore encrypted tapes, the KMS database must also be protected from disastrous events. While there are plenty of ways and methods to do this, the real challenge with protecting the KMS DB is to insure the database isn't easily compromised when storing it off-site, at a DR location, or while in transit.

To insure the KMS database is recoverable in the event of a loss, copies of the KMS database should: be kept at a designated DR site, and Information Security needs to store a copy of the pass-phrases in a secondary secured location.

In the event of a disaster, the KMS database must be able to be re-created or restored independently of the encrypted tapes, or any other NetBackup involved restores. In other words, do not rely on backups of the KMS database done to encrypted tapes, or you will not be able to restore them without having them in the first place.

NOTE: The KMS database is NOT backed up as part of the catalog backup, and must be backed up separately.

Any time the KMS database is updated, it should be backed up again, and the list of passphrases and related information updated to reflect the new phrase, and to insure the removal of any reference to any keys that have been deleted.

Saving Passphrases

Keeping a list of all the passphrases alone isn't enough to insure the KMS database can be re-build properly from passphrases.

It is also important to keep a list all of the keys that belong to a Key Group name, and their associated Key Tags. A list can be generated by the following command:

nbkmsutil -listkeys -kgname <key_group_name>

Note: Veritas recommends that you keep a record of the output of the nbkmsutil -listkeys command. The key tag that is listed in the output is necessary if you need to recover keys.

When Information Security stores the passphrases, they will also need to keep a record of each associated Key Tag and Key Name related to the respective passphrase.

Backing up the KMS DB

Backing up the KMS database is a simple matter of quiescing the database and then coping the files to a DR site or other secure location.

To quiesce the KMS DB, run:

nbkmsutil -quiescedb

This command returns with a quiesce successful statement and an indication of the number of outstanding calls.

After you have copied the files, you can unquiesce the KMS database files by using this command:

nbkmsutil -unquiescedb

A quiesce sets the KMS DB to read-only administrator mode. Quiescing is required to make a consistent backup copy of the KMS DB files.

File locations

  • Key file or key database
    Contains the data encryption keys. The key file is located at:
  • Host master key
    Contains the encryption key that encrypts and protects the KMS_DATA.dat key file using AES 256. The host master key is located at:
  • Key protection key
    Encryption key that encrypts and protects individual records in the KMS_DATA.dat key file using AES 256. The key protection key is located at:
    Currently the same key protection key is used to encrypt all of the records.

Recovery Procedures

NOTE: The KMS data files are not included in the NetBackup catalog backups. Simply restoring the catalog will NOT restore your keys

  • To rebuild the database from passphrases, simply follow the instructions listed above, for "Installing the KMS database" and "Create the "ENCR_Offsite" Key Group". Once these are created, you can use the "-recoverkey" option providing the appropriate Key Name and Key Tag (can also be used to migrate a key to a different environment):
nbkmsutil -recoverkey -kgname <key_group_name> -keyname <key_name> -tag <Key Tag>
This also requires entering the same exact pass-phrase as the original key when prompted.

Back to NetBackup