{{
`${userDetail.user.displayName}${userDetail.user.cachedBungieGlobalDisplayNameCode ? `#${standardizeBungieGlobalCode(userDetail.user.cachedBungieGlobalDisplayNameCode)}` : ""}`}}
{{getUserClans()}}
originally posted in:BungieNetPlatform
View Entire Topic
4/24/2015 6:27:34 PM
3
Manifest -> Schema -> SQL
I was playing around with the data in the manifest files (specifically the world one) and I wanted to import it all into a relational database, so I started creating tables equal to those in the SQLite database... but then I realised just how much data was there. So I've gone about generating a schema based on the data that's in there which can then be used for SQL stuff. It's not done yet, but this is the stage I'm at (yes, I'm blogging).
[url=http://pastebin.com/raw.php?i=DQLJwKNv]Here's an example[/url] based on an outdated manifest file. Install JSONView if it doesn't come out all pretty.
The idea is that when a new manifest file is released, you have some routine which drops all your existing tables, then based on the schema you generate new tables and insert the new data accordingly without needing to do it by hand. It does its best to condense anonymous objects with varying numbers of properties into a single object (ie. DestinyDirectorBookDefinition.transitionNodes) which would then be appropriate for just one database table, but it only really works so far if all those objects are owned under the same root object. That's why it fails for DestinyDirectorBookDefinition.notificationNodes and all the x, y, height, width properties in the Grimoire sections (for now...).
The SQL type determination is still a work in progress, but the idea is that it runs through the data and chooses/upgrades to the most appropriate SQL type (in this case MySQL). Object types translate to tables and array types become foreign keys.
Thoughts?
-
3 RepliesI was actually exploring a similar effort as yours but instead of using SQL I opted to use the CouchDB database instead. There are a couple of reason why I went with couchdb instead: - CouchDB is already JSON based so there is no need to generate schema's. - CouchDB supports an HTTP/JSON API which makes integration with mobile & browser based applications a lot easier. - With the build-in document versioning you can see what changes are made between different manifest updates. So when you load in a new manifest update you can easily sync this between various of consumers. See next point. - The replication protocol is super powerful and allows you to completely sync the db to a mobile device for offline use for example. - Not relational, as we have no control over the data structure provided in the manifest this will be easier to maintain as there is no need massive schema changes. - Easy to administer and backup. So far, I haven't encountered any issues with this attempt. But would love to read your blog post about your experiences once you're done :)
