How to replicate the Nuxeo repository

Goals

The system replication feature aims to clone entire collection of data existing in a Nuxeo system.

Consequently the clone is importable in another system leading to a complete replication of the system. Such feature is obviously an important gain because it allows:

• complete backup of system
• complete data migration
• replication of complex systems

• a common module,
• export module and
• import module.

In this current stage, only the documentary base is replicated. The relations, vocabularies, users replications are left for later implementation. That because:

• relations are stored usually in Jena storage: this can be moved as it is. As long as replication preserves the document ID the relations can be moved independently.
• vocabularies are also stored in directory structures which can be moved independently.
• users and groups are usally stored outside the Nuxeo repository (LDAP) which supposedly remain the same. If it is not the case, the Customer needs to previously ensure the same user / groups structure is in palce.

More can be found at http://doc.nuxeo.org/5.2/books/nuxeo-book/html-single/#admin-replication

User guide

The services (export on 5.1 and 5.2, import on 5.2) are made available either though JSF UI or through a MBean visible in JMX console. The UI is available only if the web component is also deployed.
The feature code is currently located in addon project nuxeo-platform-replication (as also nuxeo-platform-importer feature, which is used inside replication). In order to have it deployed the following jars needs to be deployed:

• Export (5.1.6+)
  • nuxeo-platform-replication-common-api-$version.jar  
  • nuxeo-platform-replication-exporter-core-$version.jar
  • nuxeo-platform-replication-exporter-api-$version.jar
  • nuxeo-platform-replication-exporter-web-$version.jar
  • nuxeo-platform-replication-exporter-mbean-5.1.6.sar (only for 5.1.6)

• Import (5.2+)
  • nuxeo-platform-replication-importer-api-$version.jar
  • nuxeo-platform-importer-core-$version.jar
  • nuxeo-platform-replication-importer-core-$version.jar
  • nuxeo-platform-replication-common-api-$version.jar
  • nuxeo-platform-replication-importer-web-$version.jar

Export instructions – JSF UI

The link “Export” is present in the top list of actions. The page allows selecting the destination of the archive. The exported artefacts are stored here, so enough space must be ensured.
Once the export launched (pressing once on the link “Export”) the page displays the progress, updating the number of documents exported in top of the page. At the end of export, status “Done” is displayed.
Please be patient and don't press twice the link.
Also, the status and information about the export can be seen in the server log (see bellow).

Export instructions – JMX console MBean

Both releases 5.1.6 and 5.2 benefits the existence of MBeans. The beans interface can be found in JMX console, under the “nx” section, as service “Exporter” on 5.1.6, respectively “ExporterService” on 5.2.  The method “export” requires 2 parameters: the repository name (usually “default”) and the path to save the archive (the third boolean one is not effective).
For the results, watch the server logs (see bellow).

Import instructions – JSF UI

The link “Import” is present in the top list of actions. The page allows selecting the source of the archive.
Once the import launched (pressing once on the link “Import”) the page displays the progress, updating the number of documents imported in top of the page. At the end of import, status “Done” is displayed.
Please logout and login in order to refresh the view.
Please be patient and don't press twice the link.
Also, the status and information about the import can be seen in the server log.

Import instructions – JMX console MBean

Release 5.2 benefits the existence of MBeans. The beans interface can be found in JMX console, under the “nx” section, as service “ImporterService”.  The method “importDocuments” requires 1 parameter: the path to save the archive.
For the results, watch the server logs.

Results: logging

The export or import results are best viewed in the logging system, as configured in JBoss. Attention: the level of logging for replication tool need to be at least INFO to see the summary. The summary contains

• the number of documents attempted to be exported
• the type of errors encountered, each one with
• a short description
• and the list of documents in fault.

15:41:08,200 INFO [ExporterReporter] Summary of export action
15:41:08,200 INFO [ExporterReporter] 108005 documents attempted to export
15:41:08,200 INFO [ExporterReporter] time elapsed: 45 minutes, 48 seconds and 38 milliseconds, velocity: 39.30 
15:41:08,200 INFO [ExporterReporter]   2 documents are missing a version.
15:41:08,200 INFO [ExporterReporter]   They are still available for import with no versions attached.
15:41:08,200 INFO [ExporterReporter]     version <unknown> for document /home/user/export/Documentary Base/Usual documents/default-domain/sections/sectiunea/notade_1257254138886
15:41:08,200 INFO [ExporterReporter]     version <unknown> for document /home/user/export/Documentary Base/Usual documents/default-domain/sections/sectiunea/notades_1257254164645
15:41:08,200 INFO [ExporterReporter]   1 documents are missing a blob file.
15:41:08,200 INFO [ExporterReporter]   Still they are available for import with a fake blob file instead.
15:41:08,200 INFO [ExporterReporter]     95d1c8b2.blob for document /home/user/export/Documentary Base/Usual documents/default-domain/workspaces/deee/filefile

Operation completed with no errors recorded.

Exporting

The following errors are possible:

• unknown system error, with the message

<# of documents> documents yields unexpected error.
Their status is undefined from the exporter perspective.

• document structure is corrupted, with the message

<# of documents> documents are compromised.
They couldn't be exported. Check log for more details.

• missing children / versions, with the message

for <# of documents> documents children are not available.
The children couldn't be read: they are not listed nor exported.

for <# of documents> documents versions are not available.
The versions couldn't be read: they are not listed nor exported.

• a particular version is missing, with the message

<# of documents> documents are missing a version.
They are still available for import with no versions attached.

• head of a version is missing, with the message

<# of documents> versions are orphans.
They are still available for import with no live document attached.

• blobs are missing, with the message

<# of documents> documents are missing a blob file.
Still they are available for import with a fake blob file instead.

Importing

The following errors are possible:

• unknown system error, with the message

<# of documents> documents yields unexpected error.
Their status is undefined from the importer perspective.

• document structure is corrupted, with the message

<# of documents> documents are compromised.
They are imported but as empty documents - including the title.

• repository import of a document failed, with the message

<# of documents> documents failed to be cloned in repository.
They couldn't be imported. Check log for more details.

• applying the custom schema change failed, with the message

<# of documents> documents custom schema update failed.
The documents are imported as they are, without any custom change.

• document type denied to be imported, with the message

<# of documents> documents were rejected based on the type selection.
The documents are not imported.

• the ACL couldn't be applied on a document, with the message

<# of documents> documents failure to update the ACL system.
The documents are imported and preserved with default security rights.

Known bugs

1.The export facility was tested and proved working fine on 5.1.6 and 5.2 servers with various backend storages (JCR, H2, PostgreSQL). The import was tested on 5.2 only with PostgreSQL backend. It is known that H2 doesn't support the import. The error thrown is timeout trying to lock a table. Nor JCR supports the import, as the node ID can't be created voluntarily in JCR system.
2.When exporting proxies, the targeted documents are also exported, increasing redundantly the size of archive: http://jira.nuxeo.org/browse/NXP-3825
3.The multithreaded import doesn't work correctly: http://jira.nuxeo.org/browse/NXP-3826, the feature is disabled in UI
4.Sometimes the UI could crash after or during the import, because the documents are changed massively in repository. Just login again.
5.The visual reports are sometime broken. The data recorded in log are the one to be considered as real.