Data migration strategies for user data stored in KERIA

[iFergal]

Recovery sync

In the KERIA/Signify architecture, each agent provisioned in the cloud has it's own set of databases and Signify itself is stateless except for the bran (sometimes referred to as the "passcode"). The bran is our recovery phrase, which we BIP-39 encode to a word list.

We do however have a local SQLite database which is primarily there to:

• Cache lists of identifiers, credentials, connections
• Store Cardano Connect/dApp connections
• Store user preferences

For the time being, we have been avoiding any local backups of this SQLite database in favour of allowing the user to recover using their recovery phrase and KERIA connect URL, and sync as much data back to the edge as we can. Some things are lost but are not vitally important, and we can improve on that in the future by having additional KERIA APIs for storing more user data. This in general could be a completely different topic of discussion which will no doubt become even more relevant when we consider backup and recovery of cloud databases.

Data

We have SQLite database at the edge, and the LMDB database in the cloud. Both are migrated as you would expect when upgrading either the app or KERIA. The LMDB migrations are of standard KERI data, like how KELs are stored, or OOBIs etc.

The harder thing to consider is due to our current recovery strategy, we have user data stored in the cloud that is non-standard. For example, for each connection, we can store connection notes, and we track big events like credential issuances and store it with the KERIA Contact using the contacts() API in Signify.

Standard LMDB migrations from keripy or KERIA will not touch this data - it is data that is entirely driven by the edge wallet, and KERIA has no inclination of what a Signify client intends to store there.

Edge initiated migrations

So it becomes tricky to manage when that data should be migrated - to me, there are only two immediate options:

1. The edge Signify client migrates it when it connects.
2. KERIA has a mechanism for scheduling custom migration scripts, that are bespoke to an edge client.

So far, for our 1.2.0 release we have been working on the first option, and it works. The second option feels like a lot of work, but it is worth considering before we release 1.2.0 and align on a strategy.

Happy to hear other options too. For example, we could become less dependent on the cloud and have better backup mechanisms for SQLite. We could even use KERIA to serve as a backup layer for it, and recognise that the entire onboarding/recovery flow could be done with a separate SQLite database than the one that's used during normal app flow.

On-connect

This means as part of migrations, we will connect to the cloud and fire a bunch of Signify calls to migrate data in the cloud. It is very important that each "area" of migrations in the cloud, like a KERIA Contact, has it's own version number so that migrations are idempotent in case we need to re-run them for completion, as we cannot perform a LMDB transaction from the edge.

However, it is also possible that we are recovering a wallet, so the connection to the cloud does not happen on app launch, but at the edge of the recovery flow. In this case, we must make sure that we create a separation between local SQLite migrations and cloud migrations.

For example, we shouldn't iterate over all local connections in SQLite, and update them in KERIA, since the connection list is not there before recovering. We should iterate over all KERIA Contacts, and update them with only data from Signify.

Additionally, if we a 1.2.0 edge and cloud migration to apply, and a 1.3.0 edge migration - both of the edge migrations will happen before we display the UI for the user to enter their recovery phrase and connect URL. So 1.3.0 edge should be allowed to happen out of order to 1.2.0 cloud.

Scheduled updates

Scheduling a migration from the edge on the surface raises a lot of concerns. The coordination of it feels like it would take a lot of effort, given there could be various wallets connected to the same KERIA, with varying Signify versions. But it also introduces the complexity of mapping the intentions of the edge client to the cloud DB - or the edge being dependent on knowing LMDB database structures which breaks a lot of rules.

There is no use case for this data being accessed by anything other than the wallet, so it feels like it brings a lot of complexity to just reduce needing to make (possibly) many API calls to migrate.

[rcmorano]

Thanks for sharing all this, it helps a lot to understand the current picture!

First of all, I fully agree with:

There is no use case for this data being accessed by anything other than the wallet, so it feels like it brings a lot of complexity to just reduce needing to make (possibly) many API calls to migrate.

Orchestrating this server-side feels like overengineering the problem with no actual benefit.

I would advocate for versioning the schema of the data stored in the contacts API data itself, so wallets could initiate edge migration processes when there is any version mismatch.
Unfortunately, I haven't found any good json/json-schema migration tool that would provide with json payload transformations, so maybe, given we already use sqlite in the wallet, we could develop some simple system that leverages sqlite migrations. Something like:

• Wallet gathers data from the could at version X.Y.Z

• Wallet applies corresponding sqlite schema version X.Y.Z into a temporary db

• Wallet run migrations on this db so it's updated to latest version X.Y.Z+N

  • If everything ran correctly:
    • Temporary db becomes the actually used db
    • New migrated data is pushed to the cloud overwritting whatever is there already
  • If there was some issue, it doesn't overwrite anything and reports the issue via support

  WDYT?

[iFergal]

The caveat is that we can't push all of the migration data to the cloud in one go. In the case of contacts, we need to update contact by contact which is why I suggest each contact has a version.

So the update becomes idempotent. If all contacts are updated, you can save the overall version number so we don't try to migrate it again, but if it fails halfway through we can still iterate over all contacts and not have any corrupted data.

This should give an idea:

veridian-wallet/src/core/storage/sqliteStorage/migrations/v1.2.0.1-connections-per-account.ts

Lines 104 to 219 in 66159a6

	cloudMigrationStatements: async (signifyClient: SignifyClient) => {
	console.log('Starting cloud KERIA migration: Converting connections to account-based model');
	let identifiers = (await signifyClient.identifiers().list()).aids;
	identifiers = identifiers.filter((identifier: any) => !identifier.name.startsWith('XX'));
	const contacts = await signifyClient.contacts().list();
	if(identifiers.length === 0) {
	for (const contact of contacts) {
	await signifyClient.contacts().delete(contact.id);
	}
	return;
	}
	for (const contact of contacts) {
	if(contact['version'] === '1.2.0') {
	console.log(`Contact ${contact.id} is already migrated from v1.2.0, skipping migration`);
	continue;
	}
	const contactUpdates: Record<string, unknown> = {};
	contactUpdates['version'] = '1.2.0';
	const keysToDelete: string[] = [];
	const historyItems: Array<{ key: string; identifier: string; data: any}> = [];
	const noteItems: Array<{ key: string; data: any}> = [];
	for(const key of Object.keys(contact)) {
	if (key.startsWith('history:ipex') ||
	key.startsWith('history:revoke')) {
	const historyData = JSON.parse(contact[key] as string);
	const historyID = historyData.id;
	const exchange = await signifyClient.exchanges().get(historyID);
	historyData.historyType = connectionHistoryTypeNumericToStringValueMap[historyData.historyType];
	if (historyData.historyType === 'CREDENTIAL_PRESENTED') {
	historyItems.push({ key, identifier: exchange.exn.i, data: JSON.stringify(historyData) });
	} else {
	historyItems.push({ key, identifier: exchange.exn.rp, data: JSON.stringify(historyData) });
	}
	} else if (key.startsWith('note:')) {
	noteItems.push({ key, data: contact[key] });
	}
	}
	const sharedIdentifierPrefix = contact.sharedIdentifier;
	if(sharedIdentifierPrefix) {
	const sharedIdentifier = identifiers.find((id: any) => id.prefix === sharedIdentifierPrefix);
	if(sharedIdentifier) {
	for(const historyItem of historyItems) {
	if(sharedIdentifier.prefix === historyItem.identifier) {
	const newPrefixedHistoryItem = `${sharedIdentifierPrefix}:${historyItem.key}`;
	contactUpdates[newPrefixedHistoryItem] = historyItem.data;
	}
	keysToDelete.push(historyItem.key);
	}
	for(const noteItem of noteItems) {
	const newPrefixedNote = `${sharedIdentifierPrefix}:${noteItem.key}`;
	contactUpdates[newPrefixedNote] = noteItem.data;
	keysToDelete.push(noteItem.key);
	}
	contactUpdates[`${sharedIdentifierPrefix}:createdAt`] = contact['createdAt'];
	keysToDelete.push('createdAt');
	} else {
	// delete contact if sharedIdentifier soft deleted
	await signifyClient.contacts().delete(contact.id);
	continue;
	}
	} else {
	// associate history items to the correct identifier
	for(const historyItem of historyItems) {
	const identifier = identifiers.find((id: any) => id.prefix === historyItem.identifier);
	if(identifier) {
	contactUpdates[`${identifier.prefix}:${historyItem.key}`] = historyItem.data;
	}
	keysToDelete.push(historyItem.key);
	}
	// associate createdAt and all notes for every non-deleted identifier
	for(const prefix of identifiers) {
	contactUpdates[`${prefix}:createdAt`] = contact['createdAt'];
	for(const noteItem of noteItems) {
	const newPrefixedNote = `${prefix}:${noteItem.key}`;
	contactUpdates[newPrefixedNote] = noteItem.data;
	}
	}
	// remove createdAt and all notes
	keysToDelete.push('createdAt');
	for(const noteItem of noteItems) {
	keysToDelete.push(noteItem.key);
	}
	}
	for(const key of keysToDelete) {
	contactUpdates[key] = null;
	}
	await signifyClient.contacts().update(contact.id, contactUpdates);
	}
	console.log(`Cloud migration completed: ${contacts.length} connections migrated to account-based model`);
	},
	};

So I don't think involving SQLite helps us there. We are basically hand rolling the schema update, which is necessary as it's a key value store and not relational. Note that we need to fetch more info from Signify while crafting the update, e.g. using exchanges().get(<said>); because we're not in a situation like an SQLite migration where it can depend on other tables in the SQL statements.

[rcmorano]

Got it, and agree, it doesn't make any sense to apply sqlite migrations if in API contacts are independent objects (which I guess correspond with wallet's sqlite rows in some table(s)) 😞 I thought it may would be just a big object that we stored for our whole table 😓
So yeah, each contact's version should be tracked independently and the migrations tailored like that I guess, at least I couldn't find a good way to do so with any tool/lib :/

[iFergal]

Yeah, we have some "representation" of some of the info stored in the contact, but the detailed info just stays in the cloud and is fetched on demand (e.g. when specific connection is opened in the UI).

The motivation for moving things to the cloud was to recover everything on a new device with just your seed and connect URL, and we do a pretty good job of that. This avoids needing a local backup, which was a simplification we wanted for v1.

But if you lose your cloud DB, you're in trouble, so I think some form of a local backup will make sense eventually. If you moved all of this user info to the edge, this problem goes away.

A primary issue we have right now is the seed phrase is only used for securing communications to KERIA and encrypting salts. The salts act as seeds for created identifiers, and are stored encrypted on KERIA (generated at the edge). We also store them locally.

So right now, if you lose BOTH your local device and cloud at the same time, you lose your identifiers, even with the seed. We have a ticket somewhere to back these up elsewhere, which I think takes priority over backing up the entire DB but maybe these initiatives could be combined.

[iFergal]

In saying that, I like the idea of the data being in the cloud, and the "local backup" copying this data down, so you could do a Signify Recovery Boot or something. But if you lost the device and cloud remained, you could just just connect and get everything back.

It's the trade-off of whose backup is better: the users or the providers.

[rcmorano]

Well, we can assume users will probably not make much backups, specially, if they get used to -somehow- having that functionality already.
It makes sense to me to rely on cloud as it's a piece of this infrastructure that needs to be in place anyways and will usually provide higher reliability/redundancy (we could save daily snapshot for the mdb for some period of time) than users' backups. But because these are not mutually exclusive, in any case it would make sense to have local backups that you can use, for example, to migrate away from one provider to another/your own.