Deserialising like a pro — Getting to grips with Moshi’s JsonReader
If you’ve ever done JSON deserialising (that is, taking a string of JSON and transforming it into an object in your Android app), you’ve almost certainly used the Gson library at some point. And there’s a good reason — it was easy to use and it largely served us well. It’s probably why it lasted for so long (since 2008, which is pretty good going for an Android library).
It seems though that it’s time to say goodbye to Gson and hello to Moshi. Why? Well, you have all the usual arguments, but the strong case is that Gson uses reflection for deserialisation whereas Moshi gives you the option of code generation, and of course less reflection is better —you get fewer run-time surprises and faster execution.
Anyway, you’re probably not here to be convinced by the benefits of Moshi. You’re probably here because you have to use it and have spent a good few hours baffled at how little documentation and/or StackOverflow content there is. Yup. I was too.
The Easy Bits — Simple Deserialisation
I’m not going to spend too much time on this because these are the things that *are* well documented. If you’re doing simple deserialisation, you probably won’t even need to change too much of your codebase. The main differences are that you replace…
Gson()with…
Moshi.Builder().build()And in your data classes, the @SerializedName annotation becomes @JsonName.
To take advantage of the code generator instead of using reflection, annotate your data classes with @JsonClass like below:
@JsonClass(generateAdapter = true)
data class SimpleJson(
val mySimpleField: String
)// Basic JSON string
val smpJsonStr = "{ \"mySimpleField\":\"json\" }"// And this line manually deserialises
val simpleJsonObj = moshi.adapter(SimpleJson::class.java)
.fromJson(smpJsonStr)
Simple JSON deserialisation is as easy as it was before, with the added benefit of code generation.
But what about an endpoint that returns very different JSON depending on, for example, state. Or what about JSON that isn’t quite in the correct format for your data classes? There are many many different ways to go about this, but when it comes to tricky transformations that you want to do at the deserialisation stage, you’re probably going to consider writing a custom adapter (although you might want to check a standard one doesn’t exist for your use case). For those used to Gson, this will be roughly similar to a custom Deserialiser.
What is an adapter and what does it do?
Put simply, an adapter implements two methods: fromJson() and toJson() and, for a specific class type, it will take either a string or a JsonReader() and return that type. When you write a custom adapter you are essentially taking over the responsibility of turning that chunk of JSON into the class the rest of your app will consume. You are literally doing the serialisation.
Obviously this is a bit of work, so you might want to consider other options — a perfectly reasonable one is to have one API object that deals with the deserialisation, and then have some sort of converter to change that into the class the rest of your app uses. You obviously do not need to do all transformations at the deserialisation stage.
However, I came across a use case fairly recently that did require this sort of intervention. A sealed class with multiple subclasses (each having very different fields, and some none) that express different levels of state. Depending on JSON content that may or may not be present, we wanted to return the appropriate state and the content that goes with that state. (Some Moshi experts will at this point suggest that the PolymorphicJsonAdapterFactory, which eloquently allows you to define a target class based on the value of a field in the JSON, would be adequate. But I had multiple fields and conditions to deal with — a custom adapter was the optimal solution.)
You’re writing too much. Just show me an adapter.
OK fine…
class MyClassAdapter: JsonAdapter<MyClass>() {
@FromJson
override fun fromJson(reader: JsonReader): MyClass? {
// Convert from JSON to MyClass here
}@ToJson
override fun toJson(writer: JsonWriter, value: MyClass?) {
// Convert from MyClass to JSON here
}
}
Pretty simple no? Two methods that convert in either direction. Then you add it into the Moshi build declaration…
Moshi
.Builder()
.add(MyClassAdapter())
.build()It’s important to note that this adapter is precisely the same as the generated adapters Moshi creates for your simpler classes.
Previously, I showed you a very simple data class called SimpleJson that was annotated with @JsonClass(generateAdapter = true). Moshi generates an adapter for that class. If you want to see it, create a data class, annotate it with @JsonClass with generateAdapter set to true, and rebuild. Then search for MyClassNameJsonAdapter in Android studio (replacing MyClassName with your class name, obviously…). I recommend this as an exercise because at some point you’ll need to step through this code, but also because the approach taken in the generated code will be very similar for when you write your own adapters.
That’s all well and good, but what goes *into* the fromJson() method
Glad you asked.
The fromJson() method takes a JsonReader. This is the class I’d like to focus on, because the number of tutorials available were pretty scant and didn’t actually adequately explained how the reader worked. So forget about adapters for now, let’s just look at one task: taking a chunk of JSON and turning it into an instance of an object.
Think of a JsonReader instance as like Pacman… but instead of white pills, it eats chunks of JSON. As you call various methods on the JsonReader instance, it will progress through the document, consuming as it goes along.
The Pacman analogy is important because depending on the methods you call, different chunks of the document will be consumed or glossed over. Let’s take an example.
We want to read this JSON…
{
"book": {
"title": "Frankenstein",
"yearWritten": "1818",
"authors": [{
"name": "Mary Shelley",
"shortBiography": "Some small bio here"
}]
}
}We don’t need to bother with adapters or anything like that to explore the JsonReader on its own. In fact, setting up a JsonReader with a manual string of JSON is the best way of understanding how it traverses it.
You can manually create a JsonReader with the above JSON as below:
val bookJson = "{\n" +
" \"book\": {\n" +
" \"title\": \"Frankenstein\",\n" +
" \"yearWritten\": \"1818\",\n" +
" \"authors\": [{\n" +
" \"name\": \"Mary Shelley\",\n" +
" \"shortBiography\": \"Some small bio here\"\n" +
" }]\n" +
" }\n" +
"}"
val source = Buffer()
source.writeString(bookJson, UTF_8)
val reader = JsonReader.of(source)Now our reader is ready to consume the JSON. Think of our Pacman. He’s currently at the beginning of the JSON, ready to “waka waka” his way through the chunks of JSON for us to process and do whatever it is we need to do with them.
When debugging, your best friend is JsonReader’s peek() method. This tells you what is next to be consumed by the reader. It was a lifesaver for me when I couldn’t realise why the reader was falling over at one point in my code. In fact, following this exercise with the JSON manually helped me pinpoint where I was going wrong.
So let’s do that…
val bookJson = "{\n" +
" \"book\": {\n" +
" \"title\": \"Frankenstein\",\n" +
" \"yearWritten\": \"1818\",\n" +
" \"authors\": [{\n" +
" \"name\": \"Mary Shelley\",\n" +
" \"shortBiography\": \"Some small bio here\"\n" +
" }]\n" +
" }\n" +
"}"
val source = Buffer()
source.writeString(bookJson, UTF_8)
val reader = JsonReader.of(source)
val nextBit = reader.peek()If you run this code and analyse nextBit, you’ll see that it returns a Token with the name BEGIN_OBJECT. This is precisely what you’d expect, because our JSON (like any JSON) begins with opening curly braces { denoting the beginning of an object.
We therefore have two options with the reader. We can decide we are not interested in this JSON and call reader.skipValue() . If we do this and then call reader.peek() again, we will get a Token with the value END_DOCUMENT. This is expected because we’ve told the reader to skip the entire object (i.e., move to the corresponding closing brace to that first opening brace, which means the end of the JSON).
As we are actually interested in the JSON, we will instead call the appropriate method for consuming an object: reader.beginObject() .
Our Pacman has now eaten the open brace. You can probably mentally work out what might be next, but let’s look anyway…
// ...val reader = JsonReader.of(source)
reader.beginObject() // {
val nextBit = reader.peek()
The Token returned has the value NAME. This indicates to us that we need to call reader.nextName() to consume it. So let’s do that and peek again.
// ...val reader = JsonReader.of(source)
reader.beginObject() // {
reader.nextName() // "book"
val nextBit = reader.peek() // "BEGIN_OBJECT"
(Note: reader.nextName() returns the value of the name. This may not seem useful right now, but it will when you write your adapter. For now, we don’t do anything with the resulting value)
As you’d expect, because the next item in the JSON to be consumed after the name “book” is an opening curly brace, the peek() tells us that we are about to begin an object. So, once again, we call reader.beginObject() .
This could get tedious if I continue this line by line, but what you can hopefully see is that this process gives us an idea of what to expect from the reader based on the JSON in front of us. Here is what we end up with if we follow this process through right to the end of the JSON…
// ...val reader = JsonReader.of(source)
reader.beginObject() // {
reader.nextName() // "book"
reader.beginObject() // {
reader.nextName() // "title"
val title = reader.nextString() // "Frankenstein"
reader.nextName() // "yearWritten"
val yearWritten = reader.nextString() // "1818"
reader.nextName() // "authors"
reader.beginArray() // [
reader.beginObject() // {
reader.nextName() // "name"
val authorName = reader.nextString() // "Mary Shelley"
reader.nextName() // "shortBiography"
val authorBio = reader.nextString() // "Some small bio here"
reader.endObject() // }
reader.endArray() // ]
reader.endObject() // }
reader.endObject() // }
reader.peek() // "END_DOCUMENT"
This may seem like a tiresome and pointless process, and you’d never deserialise like this in an adapter, but it’s here to show you precisely how the JsonReader traverses the JSON and what precisely you need to do at every possible point. I’ve highlighted the more interesting new bits.
reader.nextString() returns the value if the next item to be consumed is a STRING . There are also nextX() methods for Boolean, Int, Null, etc as you’d expect. Those methods are to retrieve the value after the reader has consumed the name.
In the same way we opened an object with reader.beginObject(), we open an array with reader.beginArray(). Once you encounter the end of an object or an array, you consume it and move to the next part of the JSON with reader.endObject() and reader.endArray() respectively.
OK, so if you wouldn’t deserialise JSON like this, how would you?
One thing you may have realised at this point is that it would be absolutely tedious to deserialise line-by-line like we have done above in this manual process. For a start, what happens if our back-end developer adds “dateOfBirth” as a field within the author object between “name” and “shortBiography”? The code above would break and the application would crash when we call endObject() — because the object no longer ends at that point. What about books that have more than one author? Our code above assumes there’ll only ever be one, despite the fact authors is plural and the value is an array.
It’s also important to note at this point that because we don’t do anything special with the deserialisation, everything we did above could be done by a generated adapter. Those 18 lines of code could have been replaced with one annotation. But, of course, this was for demonstration purposes only!
So let’s get really specific with our use-case. The object we want the JSON to go in only has three fields — bookName, primaryAuthorName and isPre20thCentury. From the perspective of the JSON, those values are in bold here:
{
"book": {
"title": "Frankenstein",
"yearWritten": "1818",
"authors": [{
"name": "Mary Shelley",
"shortBiography": "Some small bio here"
}]
}
}One of the values is not in the state we want it — it has the year as a string, but we want a Boolean that says whether the book is pre-20th-century or not. This is not a simple deserialisation. As mentioned previously, you might opt for the approach of serialising into data classes that reflect the JSON and then convert into the classes you want afterwards, but there may be times when this isn’t appropriate.
So let’s construct our data class:
data class BookInfo(
val bookName: String,
val primaryAuthorName: String,
val isPre20thCentury: Boolean
)Now let’s get to the code. We start again with the same approach — manually creating our JsonReader the same as before. We know that our JSON is going to be wrapped with this “book” envelope each time, so we can skip the name and go right into the object…
val bookJson = "{\n" +
" \"book\": {\n" +
" \"title\": \"Frankenstein\",\n" +
" \"yearWritten\": \"1818\",\n" +
" \"authors\": [{\n" +
" \"name\": \"Mary Shelley\",\n" +
" \"shortBiography\": \"Some small bio here\"\n" +
" }]\n" +
" }\n" +
"}"
val source = Buffer()
source.writeString(bookJson, UTF_8)
val reader = JsonReader.of(source)
reader.beginObject()
reader.skipName()
reader.beginObject()Now, we’re in our book object, which we know has at least one field we are interested in, but since we last wrote this code, dozens more may have been added in front of it. We don’t want our code to rely on the field we are searching to be at a particular point. So we need a way of looping through the fields and only picking out the one we’re interested in. Luckily, JsonReader has the means for you to do that:
// ...val reader = JsonReader.of(source)
reader.beginObject()
reader.skipName()
reader.beginObject()var title = ""
var authorName = ""
var isPre20thCentury: Boolean? = null
while (reader.hasNext()) {
when (reader.nextName()) {
"title" -> title = reader.nextString()
"yearWritten" -> isPre20thCentury = reader.nextString().toInt() < 1900
"authors" -> {
reader.beginArray()
reader.beginObject()
while(reader.peek() != JsonReader.Token.END_OBJECT) {
when (reader.nextName()) {
"name" -> authorName = reader.nextString()
else -> reader.skipValue()
}
}
reader.endObject()
while (reader.peek() != JsonReader.Token.END_ARRAY) {
reader.skipValue()
}
reader.endArray()
}
else -> reader.skipValue()
}
}
reader.endObject()
reader.endObject()
Now, there’s a lot in there (and it looks pretty messy), so let’s take it bit by bit.
First of all, we create temporary variables to hold our values:
var title = ""
var authorName = ""
var isPre20thCentury: Boolean? = nullSecondly, we know there may be any number of fields in our JSON object, so we want to go through all of them to find the one we are interested in. We wrap the reader.hasNext() call in a while loop. Then on each iteration we grab the name and apply a when on it to see if it’s the one we want:
while (reader.hasNext()) {
when (reader.nextName()) {
// content
}
}The first two parts of the when statement are pretty simple. The book title is there as we want it, and we just need to do a small check on the year value to see if it is pre-20th-century.
"title" -> title = reader.nextString()
"yearWritten" -> isPre20thCentury = reader.nextString().toInt() < 1900The authors when clause is a little more complex (well, as you can see, it isn’t just a line of code like the others). We know that it is an array of objects, and we are only interested in the first object, so we first of all call beginArray() and beginObject(). Then we loop through all the names of the object until we reach END_OBJECT:
while(reader.peek() != JsonReader.Token.END_OBJECT) {
when (reader.nextName()) {
"name" -> authorName = reader.nextString()
else -> reader.skipValue()
}
}
reader.endObject()If we find the name “name”, we get the next string and assign it to our variable. But you’ll notice that we have an else clause too. If the field is not the one we want, we want the JsonReader to skip it and then move onto the next field. Even if we find the field we want, we still need to continue skipping all the values, because if we call reader.endObject() when we are not at the end of the object, the JsonReader will reject it and throw an exception.
For this precise reason, we then skip every other object in the array afterwards (as we are only interested in the first primary author and any subsequent authors are not relevant to us):
while (reader.peek() != JsonReader.Token.END_ARRAY) {
reader.skipValue()
}
reader.endArray()After reader.endArray() we are now back in the root book object. We add a final reader.skipValue() to the outer when statement and then we end both objects.
OK that’s great, so does this just slot into a fromJson() method in an adapter?
Pretty much, yes! Your fromJson() method will typically receive a JsonReader at the beginning of the object your specific adapter is interested in dealing with. This is also why it’s important to perform good housekeeping and call endObject/endArray() and ensure the reader has consumed the entire thing, otherwise when it’s passed along to the next adapter, it won’t be in the right place and things will go boom.
The only thing missing from our code sample above was the instantiation of the data class, so for completeness, here it all is:
reader.beginObject()
reader.skipName()
reader.beginObject()var title = ""
var authorName = ""
var isPre20thCentury: Boolean? = null
while (reader.hasNext()) {
when (reader.nextName()) {
"title" -> title = reader.nextString()
"yearWritten" -> isPre20thCentury = reader.nextString().toInt() < 1900
"authors" -> {
reader.beginArray()
reader.beginObject()
while(reader.peek() != JsonReader.Token.END_OBJECT) {
when (reader.nextName()) {
"name" -> authorName = reader.nextString()
else -> reader.skipValue()
}
}
reader.endObject()
while (reader.peek() != JsonReader.Token.END_ARRAY) {
reader.skipValue()
}
reader.endArray()
}
else -> reader.skipValue()
}
}
reader.endObject()
reader.endObject()val bookInfo = BookInfo(
title,
authorName,
requireNotNull(isPre20thCentury)
)
And of course, this would pretty much be the body of your fromJson(), if you return the bookInfo instance at the end.
(NB: for better efficiency, you’ll want to take a look at JsonReader.selectName() in place of JsonReader.nextName(), but I won’t cover that here)
Thanks for reading
This is the first time I’ve done one of these in a while, but after fighting with JsonReader I felt the need to share the knowledge. Please give your comments below, and if I’ve got anything wrong or missed out something please mention it. You can also get me on Twitter.
