Go, SQS and Unmarshal
Making sense of what our client sends us, we use json.Unmarshal
Let's research a use case where we don't know about data type beforehand. Most people use interface{} but is it optimal? Read on to know.
// Envelope represents basic sqs event format
type Envelope struct {
EventID string `json:"eventId"`
EventTime time.Time `json:"eventTime"`
Source string `json:"source"` // service name that publishes the message
Type string `json:"type"` // for example "UPSERT"
Data interface{} `json:"data"` // populate with your message
}
Example:
package main
import (
"fmt"
"encoding/json"
"time"
"github.com/satori/go.uuid"
)
type Envelope struct {
EventID string `json:"eventId"`
EventTime time.Time `json:"eventTime"`
Source string `json:"source"`
Type string `json:"type"`
Data interface{} `json:"data"` // populate with your message
}
type Payload struct {
Email string
ID uint64
}
func main() {
e := Envelope{
EventID: uuid.NewV4().String(),
EventTime: time.Now(),
Source: "GO_PLAYGROUND",
Type: "TEST",
Data: Payload{
Email: "lorem.ipsum@lipsum.com",
ID: 123456,
},
}
data, _ := json.Marshal(e)
var incoming Envelope
_ = json.Unmarshal(data, &incoming)
fmt.Println("Hello, playground", incoming.Data)
}
Output:
Hello, playground map[Email:lorem.ipsum@lipsum.com ID:123456]
With Data as interface{}, on unmarshall it will lead to a map data structure(map[string]interface{}). It's really hard to extract payload data from the above construct.
It's better to parse it this way
package main
import (
"fmt"
"reflect"
"encoding/json"
"time"
"github.com/satori/go.uuid"
)
type Envelope struct {
EventID string `json:"eventId"`
EventTime time.Time `json:"eventTime"`
Source string `json:"source"`
Type string `json:"type"`
Data interface{} `json:"data"` // populate with your message
}
type Payload struct {
Email string
ID uint64
}
func main() {
e := Envelope{
EventID: uuid.NewV4().String(),
EventTime: time.Now(),
Source: "GO_PLAYGROUND",
Type: "TEST",
Data: Payload{
Email: "lorem.ipsum@lipsum.com",
ID: 123456,
},
}
data, _ := json.Marshal(e)
incoming := Envelope{
Data:&Payload{},
}
_ = json.Unmarshal(data, &incoming)
fmt.Println("Hello, playground", incoming.Data)
fmt.Println("Payload type", reflect.TypeOf(incoming.Data))
}
Output:
Hello, playground &{lorem.ipsum@lipsum.com 123456}
Payload type *main.Payload
Looks like we got rid of map and got Payload concrete type instead, which is cool but did you notice that we are required to know the concrete type beforehand !
Enters json.RawMessage...
// Envelope represents basic sqs event format
type Envelope struct {
EventID string `json:"eventId"`
EventTime time.Time `json:"eventTime"`
Source string `json:"source"` // service name that publishes the message
Type string `json:"type"` // for example "UPSERT"
Data json.RawMessage `json:"data"` // populate with your message
}
With Data as json.RawMessage, on umarshall it will lead to byte slice(json.RawMessage).
Example:
package main
import (
"encoding/json"
"fmt"
"time"
"github.com/satori/go.uuid"
)
type Envelope struct {
EventID string `json:"eventId"`
EventTime time.Time `json:"eventTime"`
Source string `json:"source"` // service name that publishes the message
Type string `json:"type"` // for example "UPSERT"
Data json.RawMessage `json:"data"` // populate with your message
}
type Payload struct {
Email string
ID uint64
}
func main() {
bytepayload, _ := json.Marshal(
&Payload{
Email: "lorem.ipsum@lipsum.com",
ID: 123456,
},
)
rawbytepayload := json.RawMessage(bytepayload)
e := Envelope{
EventID: uuid.NewV4().String(),
EventTime: time.Now(),
Source: "GO_PLAYGROUND",
Type: "TEST",
Data: rawbytepayload,
}
data, _ := json.Marshal(e)
var incoming Envelope
_ = json.Unmarshal(data, &incoming)
fmt.Println("Hello, playground", incoming.Data)
}
Output:
Hello, playground [123 34 69 109 97 105 108 34 ........]
As json.RawMessagecreates a byte slice at the end so it basically can consume anything.
package main
import (
"fmt"
"reflect"
"encoding/json"
"time"
"github.com/satori/go.uuid"
)
// Envelope is envelope at our sqs publisher
type Envelope struct {
EventID string `json:"eventId"`
EventTime time.Time `json:"eventTime"`
Source string `json:"source"`
Type string `json:"type"`
Data interface{} `json:"data"` // populate with your message
}
// IncomingEnvelope is envelope at our generic sqs consumer.
type IncomingEnvelope struct {
EventID string `json:"eventId"`
EventTime time.Time `json:"eventTime"`
Source string `json:"source"`
Type string `json:"type"`
Data json.RawMessage `json:"data"` // populate with your message
}
type Payload struct {
Email string
ID uint64
}
func main() {
e := Envelope{
EventID: uuid.NewV4().String(),
EventTime: time.Now(),
Source: "GO_PLAYGROUND",
Type: "TEST",
Data: Payload{
Email: "lorem.ipsum@lipsum.com",
ID: 123456,
},
}
data, _ := json.Marshal(e)
var incoming IncomingEnvelope // this is diff
_ = json.Unmarshal(data, &incoming)
fmt.Println("Hello, playground", incoming.Data)
fmt.Println("Payload type", reflect.TypeOf(incoming.Data))
}
Output:
Hello, playground [123 34 ......]
Payload type json.RawMessage
Nice! We can then use the data slice and parse it into any valid concrete type.
So using interface{} at consumer side doesn't work? It does!
package main
import (
"encoding/json"
"fmt"
"time"
"github.com/satori/go.uuid"
)
type Envelope struct {
EventID string `json:"eventId"`
EventTime time.Time `json:"eventTime"`
Source string `json:"source"` // service name that publishes the message
Type string `json:"type"` // for example "UPSERT"
Data json.RawMessage `json:"data"` // populate with your message
}
type IncomingEnvelope struct {
EventID string `json:"eventId"`
EventTime time.Time `json:"eventTime"`
Source string `json:"source"` // service name that publishes the message
Type string `json:"type"` // for example "UPSERT"
Data interface{} `json:"data"` // populate with your message
}
type Payload struct {
Email string
ID uint64
}
func main() {
bytepayload, _ := json.Marshal(
&Payload{
Email: "lorem.ipsum@lipsum.com",
ID: 123456,
},
)
rawbytepayload := json.RawMessage(bytepayload)
e := Envelope{
EventID: uuid.NewV4().String(),
EventTime: time.Now(),
Source: "GO_PLAYGROUND",
Type: "TEST",
Data: rawbytepayload,
}
data, _ := json.Marshal(e)
var incoming = IncomingEnvelope{
Data:&Payload{}, // we need to know this beforehand
}
_ = json.Unmarshal(data, &incoming)
fmt.Println("Hello, playground", incoming.Data)
}
Output:
Hello, playground &{lorem.ipsum@lipsum.com 123456}
So using above shared solution on such a type can work for my consumer? No.
Why ? Being a generic consumer I don't know what type my consumer has sent when it published the event!
TLDR; Use json.RawMessage where you are not sure about exact concrete types. In my use-case it was a generic SQS subscriber to be used by multiple backend services/workers.
Bonus: It outputs byte slice which more efficient to move around(data is still kept low level) rather than a map. You can read about this more on Googling.
All examples presented above are go-playground runnable, do try them. I don't suggest to ignore errors in production code.
Thanks!