ArcGIS Data Store allows you to create highly available or scalable data stores by including more than one machine in each type of data store that you deploy.
Options for adding machines vary by data store type and are explained below:
Relational data store
The relational data store is always deployed in primary-standby mode. The first machine on which you install ArcGIS Data Store and create a relational data store is the primary relational data store machine. You can install and configure one other relational data store with the same GIS Server site, making this second relational data store machine the standby machine. Hosted feature layer data is replicated from the primary to the standby machine.
If the primary machine becomes unavailable (for example, if it crashes, loses power, or loses network connectivity), the standby machine is promoted to primary. You will only be without your hosted feature layer data for the period of time it takes for the standby machine to become the primary—this can take up to a few minutes. Once the standby becomes the primary relational data store machine, it sends its backup files to the same location as the former primary machine.
If you can correct the problem with the former primary machine—for example, if the machine was unplugged and you plugged it back in, or the machine temporarily lost network connectivity but network connectivity was later restored—the former primary becomes the standby machine for the relational data store.
If you cannot correct the problem—for example, the machine crashes and cannot be recovered—use the removemachine utility to unregister the failed machine. Configure a new relational data store machine to act as the standby machine.
For information about when a relational data store fails over, see ArcGIS Data Store deployment modes.
Tile cache data store
Note:
ArcGIS Enterprise 11.5 is the last release that supports the tile cache data store. If you have hosted scene layer caches stored in the tile cache data store, you must configure an object store and migrate hosted scene layer caches at this release. See the tile cache data store deprecation notice for more information.
The number of machines you can add to a tile cache data store depends on the deployment mode you use.
- Primary-standby mode—In primary-standby mode, a tile cache data store can contain a single machine or, to provide fail-over functionality, it can contain two machines. The default deployment mode for a tile cache data store is primary-standby mode.
The first machine on which you install ArcGIS Data Store and create a tile cache data store is the primary tile cache data store machine. You can install and configure one other tile cache data store with the same GIS Server site, making this second tile cache data store machine the standby machine. Hosted scene layer cache data is replicated from the primary to the standby machine. Similar to relational data stores, the standby becomes primary if the primary machine fails.
When both primary and standby machines are running, ArcGIS can use the primary or the standby tile cache data store machine when processing read-only queries. This improves performance when you're viewing a hosted scene layer.
For information about when a tile cache data store running in primary-standby mode fails over, see ArcGIS Data Store deployment modes.
- Cluster mode—In cluster mode, a tile cache data store can contain multiple data store machines to provide for scalability.
The hosted scene layer cache data is duplicated on two of the tile cache data store machines in the cluster.
When you add machines to a cluster, existing scene caches remain on the original machines. To rebalance the existing caches to use the newly added machines, you must create a backup of the tile cache data store and restore it. For instructions, see Rebalance scene layer caches. Alternatively, configure an object store cluster that meets the resource requirements for your data, and migrate hosted scene layer caches to it.
Recovery after loss of a tile cache machine in cluster mode
If you have multiple machines in the tile cache data store for the purpose of high availability and one of the machines is lost or goes offline, you may need to redistribute scene cache data.
Attempt to bring the failed machine back online. For example, if a network or power cable is unplugged or the machine was shut off, you can rectify the issue by plugging in the network or power cable or turning on the computer. Once back online, the machine rejoins the tile cache data store and the data store continues to function.
If a machine crashes and cannot be resurrected, you need to take steps to ensure that scene layer caches are still highly available. See Recover a data store for instructions.
Spatiotemporal big data store
The spatiotemporal big data store is always deployed in cluster mode. You can install ArcGIS Data Store on several machines, create spatiotemporal big data stores on each, and register each one with the same hosting GIS Server site.
Note:
Always use an odd number of machines in the spatiotemporal big data store.
Data in the spatiotemporal big data store is duplicated on at least two spatiotemporal big data store machines registered to the same hosting server.
If any one machine fails, data still exists on at least one other machine. The data store replicates the data to another machine that is still functional.
If the machine comes back online, it rejoins the cluster and data is rebalanced.
If the failed machine cannot be recovered, you can install ArcGIS Data Store on more machines and create spatiotemporal big data stores that are registered with the same hosting GIS Server site. In this example, three machines are added: one to replace the one that failed, and two more to accommodate increased numbers of hosted spatiotemporal feature layers. Data is rebalanced across all seven machines. Data for newly published spatiotemporal feature layers is also distributed.
Note:
If you script the creation of multiple spatiotemporal big data store machines, one spatiotemporal big data store machine must be manually configured with the GIS Server site before you can script the creation of additional spatiotemporal big data store machines. Include wait times in your script to be sure the additional spatiotemporal big data store machines are not added at the same time.
Machine roles
In a spatiotemporal big data store cluster, there are two roles: coordinator and data.
- Coordinator—The coordinator keeps track of what machines are in or have been removed from the cluster and the health of each machine. The coordinator also controls the distribution of data across the cluster and rebalancing data when a machine is lost or removed.
There must be at least one coordinator machine in the cluster. A spatiotemporal big data store can contain a maximum of five machines that have the coordinator role assigned to them. The first coordinator machine added to the cluster will be the current coordinator of the cluster; one of the other machines assigned the coordinator role can take over if the current coordinator machine is lost or removed from the cluster.
Machines that fill the coordinator role experience higher network traffic as a result of all the communication required to manage all the machines in the cluster.
- Data—Data machines store and search the spatial and nonspatial tables that populate hosted spatiotemporal layers. Machines that fill the data role must have a large amount of memory and processing power.
A machine can fill either role or both roles.
When you create a spatiotemporal big data store machine using the Data Store configuration wizard, the first three machines are created in both roles. In other words, all three machines will store data, the first machine will also serve as the current coordinator, and either of the other two machines can be promoted to coordinator if the current coordinator machine fails. Starting with the fourth machine, all machines added to the cluster using the Data Store configuration wizard are deployed in the data role only.
To control the role filled by a particular machine in the cluster when you create it, use the configuredatastore utility to create the spatiotemporal big data store. For example, you can run the configuredatastore utility to create the first three machines in the coordinator role only and create two more machines in the data role only, for a total of a five-machine cluster. This configuration isolates the network communications that could slow performance when the coordinator and data roles are filled by the same machine. It also allows you to have extra resources on the data only machines where they are most needed.
You can also use the configuredatastore utility to change the role of an existing, healthy spatiotemporal big data machine.
Supported role changes are as follows:
- Specify --roles data to change a coordinator to a data machine. This is possible if there are other coordinator-eligible machines in the cluster. You can change a coordinator machine to a data role if there are at least three other coordinator-eligible machines in the cluster.
- Specify --roles data to remove the coordinator role from a machine that was deployed with both roles. This leaves the machine in the data role. You can remove the coordinator role from the machine if there are at least three other coordinator-eligible machines in the cluster.
- Specify --roles coord,data to add the coordinator role to a machine that was initially deployed to fill only the data role. A maximum of five machines can include the coordinator role. If there are already five coordinator machines in the cluster, you cannot add the coordinator role to a data machine in the cluster.
Run the describedatastore utility on any machine in the cluster to identify which machines are assigned the coordinator role, which are assigned the data role, and which machine is the current coordinator for the cluster.
You can only remove the coordinator role from a machine if there are three other coordinators in the cluster. In other words, if there are three machines that fill both the coordinator and data roles in the cluster, you cannot remove the coordinator role from any of those machines.
The number of machines in the cluster affects the status of the cluster when a machine fails and affects what role you can assign to the replacement machine. Keep in mind that a spatiotemporal big data store cluster should contain an odd number of machines.
In a cluster with only one coordinator machine, failure of that machine renders the spatiotemporal big data store unusable. You must add the coordinator role to at least two of the remaining machines or add two machines in the coordinator role to the cluster to bring the spatiotemporal big data store back online. Therefore, it is recommended that you have three or five machines in the coordinator role in the cluster. These machines can be in both the coordinator and data role or, if you have other machines in the cluster that fill the data role, they can be in just the coordinator role.
You must have at least two data machines in the cluster to prevent data loss if a data machine fails. In a cluster with only two data machines, when one data machine fails, add at least one new data machine to the cluster. If you do not add at least one data machine, you risk losing data if the remaining data machine fails. When there is only one data machine in the cluster, the only way to recover the data if the data machine fails is to restore the spatiotemporal big data store from a backup.
Graph store
Starting with ArcGIS Data Store 11.5, the graph store can contain one machine (single instance) or three machines (cluster). If you require a highly available graph store, configure a graph store that contains three machines.
By default, when you install ArcGIS Data Store and create a graph store, it is created in single instance mode, but you can deploy in cluster mode instead. When you deploy in cluster mode, you must install and configure two other graph store machines with the same hosting GIS Server site. The graph store is not functional until it contains three machines.
When you deploy in cluster mode, hosted knowledge graphs will be duplicated on two of the three machines for high availability. The hosting server also queries all three machines, which improves query performance.
If you deploy in single instance mode, you can use the configuredatastore utility to change the graph store to cluster mode.
Note:
You must have a default backup location configured for the graph store before you can change the graph store from single instance to cluster mode. You cannot change the graph store from cluster mode to single instance mode.
Legacy:
Graph stores created with ArcGIS Data Store 11.1 or earlier can contain only one machine. Graph stores created with ArcGIS Data Store 11.2, 11.3, or 11.4 can contain a maximum of two machines.
If a machine in the cluster fails, data still exists on at least one of the other machines in the cluster. At this point, the graph store is in an unhealthy, read-only state, but existing graph layers continue to function. If the failed machine recovers—for example, it was temporarily unavailable due to a network issue that was corrected—the graph store cluster returns to a healthy state.
If the failed machine cannot be recovered, you must do the following to maintain the required three machines in the graph store cluster:
- First, run the removemachine utility to remove the failed machine from the cluster. This marks the machine for deletion.
- Configure a graph store on a new machine and add it to the cluster. The machine that was marked for removal is now unregistered from the graph store cluster.
If two machines in the cluster fail and cannot be recovered, the graph store is no longer functional. You cannot remove and replace the two machines. You must restore the graph store from a backup.
Object store
You can create an object store that contains one machine or you can deploy an object store in cluster mode that contains three or more machines.
A single-machine object store with a large amount of disk space and RAM can provide adequate performance, depending on how many web layer types your organization is storing in the object store and how many users are accessing those web layers. If you require scalability to allow the object store to grow as needed, deploy the object store in cluster mode when you create it.
Note:
You cannot change modes after deploying the object store.
When you deploy the object store in cluster mode, it must contain at least three machines. You can add machines to the object store cluster if the existing machines get overloaded. Similarly, if one of the machines in the object store cluster fails and you can't bring it back online, you can install ArcGIS Data Store on a new machine, and configure it as an object store with the same hosting server.
If the cluster contains only three machines, you must add a new machine to the cluster before you can run the removemachine utility to remove the failed machine.
After you add a machine to the cluster, newly enabled query caches and new web layers will use the new machine in the cluster.
Add a machine to a data store
Use the Data Store configuration wizard or the configuredatastore utility to add a machine to an ArcGIS Data Store type.
You can have one standby relational data store or tile cache data store that is running in primary-standby mode. You can have multiple machines in a spatiotemporal big data store and in a tile cache data store or object store that is running in cluster mode. You must have three machines in a graph store running in cluster mode. All machines that participate in the same data store must use the same operating system and be the same ArcGIS Data Store release.
Use the Data Store configuration wizard to add a machine to a data store
Follow the steps below to use the Data Store configuration wizard to add a machine to a data store.
Note:
To add a machine to a tile cache data store, you must use the configuredatastore utility instead, which is described in the next section.- Install ArcGIS Data Store on another server.
Each standby machine, primary machine, or node in a cluster must be installed on different physical servers. Use the same network account for the ArcGIS Data Store account on all machines in the data store.
- Open the ArcGIS Data Store configuration wizard.
- Specify the fully qualified name of one of the machines in the GIS Server site that is acting as the ArcGIS Enterprise hosting server.
The name is in the format gisserver.example.com. Use the same GIS Server site as you did when configuring the other machine or machines in the same data store for this ArcGIS Enterprise deployment.
- Type the username and password of the ArcGIS Server primary site administrator and click Next.
- Choose the type of data store to create.
- Specify a location for the new data store's directory and click Next.
- Review the information in the Configuration Summary. If it is correct, click Finish. If changes are needed, click Back to make corrections.
The High Availability Role indicates that this is a standby machine for relational data stores.
When the setup completes successfully, you have an additional data store machine to which ArcGIS Data Store can replicate data or caches.
If there were errors with the setup, click Next on the Configuration Summary dialog box to reopen the Data Store configuration wizard so you can provide different information. See Troubleshoot ArcGIS Data Store for causes and resolutions to setup errors.
Use the configuredatastore utility to add a machine to a data store
You can install ArcGIS Data Store on another server or servers and use the configuredatastore utility to add it to an existing data store. The utility is installed in <ArcGIS Data Store installation directory>/tools.
Each standby machine, primary machine, or node in a cluster must be installed on different physical servers.
Note:
You must use the same network account for the ArcGIS Data Store account on all machines in the data store.
Follow these steps to use the configuredatastore utility to add a machine to a data store:
- Open a Command Prompt using the Run As Administrator option on the additional server where you installed ArcGIS Data Store.
- Run the configuredatastore utility, and provide the fully qualified domain name of a machine in the GIS Server site that is acting as the hosting server, and provide the ArcGIS Server primary site administrator username and password.
Use the --stores operation to specify the type of data store to be created.
You must specify the same GIS Server machine name and data store type that you used for the data store to which you want to add this machine.
For example, the following adds a standby relational data store to a machine in the GIS Server site with fully qualified name agsserver.ntwk.com. The data store directory on this machine is c:\data\.
configuredatastore agsserver.ntwk.com siteadmin P$@pass c:\data\ --stores relational
Another data store machine is added to the data store.
See the configuredatastore section of the ArcGIS Data Store utility reference for syntax and more examples.