How to Read the Manifest (Reference)

Know this thread is super old and with Destiny 2 coming out, very few will be interested in this. But for practice I got the manifest working in Node.js. Any comments/critiques are more than welcome! Since formatting isn't preserved here, I just posted a link to the code. [url]https://github.com/lrwong/destiny_testing/blob/master/api_testing/public/javascripts/server/manifest.js[/url]

Anyone done this in Python?

Note - You don't actually need to extract the JSON to get the hash for a given row - you CAN use the id column. OP is correct that they don't always match - but this is for a very simple reason. itemHash appears to be an unsigned 32bit integer, but SQLite is interpreting the value as a signed int. See this query: SELECT * FROM DestinyInventoryItemDefinition WHERE CAST(SUBSTR(json, 13,10) AS INT) != id AND CAST(SUBSTR(json, 13,10) AS INT) != 4294967296 + id No rows are returned, because either the itemHash matched the id (first case of WHERE clause) or converting the unsigned itemHash to a signed int causes it to match the id (second case of the WHERE clause). So, if you're using a C-like language with casts, just make your itemHash an unsigned 32 bit int, cast it to a signed int, and then use that value as the id. If you're using a weakly-typed language like PHP, or if your language doesn't allow you to specify 32-bit ints, you can just convert it manually: SELECT * FROM DestinyInventoryItemDefinition WHERE id + 4294967296 = <ItemHash> OR id = <itemHash> Just replace <ItemHash> above with the item hash you're looking for. And obviously, replace "DestinyInventoryItemDefinition" with the table you're interested in. Should be significantly more efficient than parsing the entire database, and should also be much faster for lookup.

Thanks for this - was very useful! As you requested, here's a class I put together to replicate what you're doing, but in C#. It's not a class I'm using, and it's not a perfect replication; My class doesn't produce seperated json files, but creates a single large json object that contains the entire manifest. I did construct the code to be easy for someone to produce those files however. It's also not "perfect" code, as it's just a throwaway class I've created purely for this thread, in the hope it'll be useful - so don't be judging ;) It requires JSON.net, but you could quite happily get around that if you desire. [spoiler] using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Threading.Tasks; using System.Data.SQLite; using Newtonsoft.Json.Linq; using System.Net; using System.IO; using System.Reflection; using System.IO.Compression; namespace ThrowAwayCode { public class Manifest { /// <summary> /// Queriable Json respresentative of the bungie Manifest /// </summary> public dynamic ManifestData { get; set; } /// <summary> /// static method to fetch the latest manifest file from bungie, unzip it, and pass back a populated Manifest class /// </summary> /// <returns>Manifest class that represents the current "mobileWorldContentPaths" manifest</returns> public static Manifest LoadManifest() { //get the manifest details string requestURL = "https://www.bungie.net/Platform/Destiny/Manifest"; HttpWebRequest request = (HttpWebRequest)WebRequest.Create(requestURL); //request.Headers.Add("X-API-Key", apiKey); //not currently required request.KeepAlive = true; request.Host = "www.bungie.net"; HttpWebResponse response = (HttpWebResponse)request.GetResponse(); string responseText; using (TextReader tr = new StreamReader(response.GetResponseStream())) { responseText = tr.ReadToEnd(); } dynamic json = JObject.Parse(responseText); //find and delete the existing manifest if it already exists - you probably don't want to do this, but this is throwaway ;) string manifestPath = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location); string jsonVersion = json.Response.version; string manifestFile = Path.Combine(manifestPath, jsonVersion + ".manifest"); if (File.Exists(manifestFile)) File.Delete(manifestFile); try { //get the current manifest file stream string manifestURL = json.Response.mobileWorldContentPaths.en; request = (HttpWebRequest)WebRequest.Create("https://www.bungie.net" + manifestURL); using (Stream stream = request.GetResponse().GetResponseStream()) { //need to use a memory stream as the Microsoft ZipArchive doesn't seem to like non-seekable streams using (MemoryStream ms = new MemoryStream()) { stream.CopyTo(ms); ms.Seek(0, SeekOrigin.Begin); //extract the manifest file from it's zip container ZipArchive zippedManifest = new ZipArchive(ms); zippedManifest.Entries[0].ExtractToFile(manifestFile); } } //time to turn the manifest into a class! return new Manifest(manifestFile); } catch (Exception ex) { throw; } } //used to map the database to json public struct ManifestBranch { public string TableName; public List<dynamic> Json; public List<string> JsonString; } /// <summary> /// where the magic happens... /// Takes a SQLite manifest file, and returns a Manifest class with all the data organised into queriable json. /// </summary> /// <param name="manifestFile">the bungie manifest SQLite database file</param> public Manifest(string manifestFile) { using (var sqConn = new SQLiteConnection("Data Source=" + manifestFile + ";Version=3;")) { sqConn.Open(); //build a list of all the tables List<string> tableNames = new List<string>(); using (var sqCmd = new SQLiteCommand("SELECT name FROM sqlite_master WHERE type='table'",sqConn)) { using (var sqReader = sqCmd.ExecuteReader()) { while(sqReader.Read()) tableNames.Add((string)sqReader["name"]); } } //get the json for each row, in each table, and store it in a lovely array List<ManifestBranch> manifestData = new List<ManifestBranch>(); foreach(var tableName in tableNames) { var manifestBranch = new ManifestBranch { TableName = tableName, Json = new List<dynamic>(), JsonString=new List<string>() }; using (var sqCmd = new SQLiteCommand("SELECT * FROM " + tableName,sqConn)) { using (var sqReader = sqCmd.ExecuteReader()) { while (sqReader.Read()) { byte[] jsonData = (byte[])sqReader["json"]; string jsonString = Encoding.ASCII.GetString(jsonData); manifestBranch.Json.Add(JObject.Parse(jsonString)); //you don't need to do this unless you want queriable json at this level ;) manifestBranch.JsonString.Add(jsonString); } } } manifestData.Add(manifestBranch); } //this next bit takes all of the json, for all of the tables, and wraps it up nicely //into a single json string, which is then made dynamic and thus queriable ^_^ string fullJson = "{\"manifest\":["; foreach(var manifestBranch in manifestData) { fullJson += "{\"" + manifestBranch.TableName + "\":[" + string.Join(",", manifestBranch.JsonString) + "]},"; } fullJson = fullJson.TrimEnd(',') + "]}"; this.ManifestData = JObject.Parse(fullJson); //instead of the above, you can just loop through each branch and create individual files to directly replicate lowlines code example //however, it's probably best to do this further up, where we're looping through each table, rather than down here at the end ;) sqConn.Close(); } } } } [/spoiler] thanks for the original post!

For the /Manifest/{type}/{hash} endpoint, what are the options for the {type}? I haven't been able to find an explanation for this anywhere, and I did search through the list of globals that has been posted on this forum. Thanks!