Robust Serialization with JSON and Custom Types

In this article, you will learn how to build a robust JSON serialization pipeline in Python that handles custom data types, graceful error recovery, and schema migrations.

Topics we will cover include:

• How to define a custom JSON encoder to handle Python types like datetime, set, Path, and Enum.
• How to write a custom decoder function that reconstructs those original Python objects from a serialized JSON string, including graceful handling of malformed data.
• How to manage schema changes over time using a factory function that applies defaults and migrations during deserialization.

Introduction and Motivation

Anyone fairly experienced in constructing machine learning pipelines might have noticed at some stage that only a small percentage of their time is dedicated to writing the model architecture. What do they spend the rest of that time on? Data plumbing: the process of cleaning, preprocessing, and transforming data to make it look perfectly digestible for downstream models.

This article tackles a hidden payoff often discovered along the machine learning engineering journey: the constant need to move structured datasets between files, processes, microservices, and APIs. Throughout these processes, you need a solution to serialize and deserialize data in a reliable and robust fashion.

We will explore some practical strategies for serializing custom types in a way that enables parsing them back securely, as well as managing data schema changes and dealing with malformed data — all without necessarily crashing the whole pipeline!

Before going deep into the three-step solution, let’s look at the root problem: JSON limitations when encountering certain data types like datetime, set, Path, or Enum. Suppose this is the configuration for a machine learning experiment:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19 from datetime import datetime

from enum import Enum

from pathlib import Path

import json

class ModelType(Enum):

RANDOM_FOREST="rf"

NEURAL_NET="nn"

experiment_config={

"experiment_id":1042,

"model_type":ModelType.NEURAL_NET,

"features":{"age","income","location"},# A set!

"data_path":Path("/data/v2/train.csv"),

"timestamp":datetime.now(),

"nested_meta":{

"retries":3

}

}

The moment you try to serialize this experiment configuration using json.dumps(experiment_config), you will immediately come across an error like this:

1 TypeError:Object of type ModelType is not JSON serializable

These kinds of errors are what we are about to fix by defining a robust pipeline for encoding and decoding JSON objects with custom types.

1. Defining a Custom Encoder

First, we will define a custom encoder class that inherits from json.JSONEncoder. This is crucial to be able to “customize” the existing json.dumps() with the extra logic we will implement in our class to handle custom data types that JSON cannot handle by default.

Besides subclassing json.JSONEncoder, we also need to override the default() method. This is where, one by one, we will define the custom approach to handle each specific data type we might need to work with, using conditionals in the form if isinstance(obj, DATA_TYPE):, replacing DATA_TYPE with a specific data type or class name, as shown below:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24 class RobustEncoder(json.JSONEncoder):

def default(self,obj):

Handling Datetimes

if isinstance(obj,datetime):

return{"__type__":"datetime","value":obj.isoformat()}

Handling Sets (convert to list for JSON)

if isinstance(obj,set):

return{"__type__":"set","value":list(obj)}

Handling Paths (convert to string)

if isinstance(obj,Path):

return{"__type__":"path","value":str(obj)}

Handling Enums (extract the underlying value)

if isinstance(obj,Enum):

return obj.value

Letting the base class handle everything else (or raise TypeError)

return super().default(obj)

Serializing our config object

json_string=json.dumps(experiment_config,cls=RobustEncoder,indent=2)

print(json_string)

One more important detail: the extra trick for robust serialization in the above code is attaching the __type__ marker to objects that might lose their original typing when they are turned into JSON primitives. This occurs, for instance, when converting Python sets into lists.

Notice how we use cls=RobustEncoder in this new call to json.dumps() to use our custom encoder.

Result:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23{

"experiment_id":1042,

"model_type":"nn",

"features":{

"__type__":"set",

"value":[

"income",

"location",

"age"

]

},

"data_path":{

"__type__":"path",

"value":"/data/v2/train.csv"

},

"timestamp":{

"__type__":"datetime",

"value":"2026-09-28T14:26:10.328310"

},

"nested_meta":{

"retries":3

}

}

No TypeError anymore!

2. Defining a Custom Decoder

For the inverse process — obtaining Python objects back from a serialized JSON string — the key is using a different method and argument: json.loads() alongside an object_hook parameter.

We don’t need to define a class through inheritance this time, but we do create a custom function named robust_decoder which, when passed to json.loads(), will run on all nested dictionaries in the JSON string during parsing. The code includes the logic to reconstruct original Python objects, such as Path, as needed:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27 import logging

def robust_decoder(dct):

Checking if this dictionary represents one of our custom types

if"__type__"in dct and"value"in dct:

type_name=dct["__type__"]

val=dct["value"]

try:

if type_name=="datetime":

return datetime.fromisoformat(val)

elif type_name=="set":

return set(val)

elif type_name=="path":

return Path(val)

except(ValueError,TypeError)as e:

Handling Malformed Data: Logging the error without causing a crash.

Return a safe fallback or a specialized error dictionary.

logging.warning(f"Malformed {type_name} data detected: {e}. Falling back to raw value.")

return val

return dct

Deserialize back to Python objects

parsed_config=json.loads(json_string,object_hook=robust_decoder)

print(type(parsed_config["timestamp"]))

print(type(parsed_config["features"]))

The function also includes exception handling in case malformed data is encountered.

Result (output of the two print calls):

1

2<class'datetime.datetime'>

<class'set'>

3. Dealing with Schema Changes

One extra common challenge in machine learning projects is that model metadata schemas (model specifications or configuration, in simpler terms) change constantly. It is not rare to add new hyperparameters, deprecate old ones, or rename a metric. Loading old JSON config objects into newer versions of your code therefore often requires wrapping the deserialized dictionary in an additional validation layer.

The following factory function can help achieve this, aided by the robust decoder function we implemented earlier:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18 def load_experiment_config(json_data:str)->dict:

1. Deserializing with our robust hook

raw_dict=json.loads(json_data,object_hook=robust_decoder)

2. Applying Schema Migrations / Defaults

Our older configuration didn't have 'learning_rate', so we provide a safe default.

validated_config={

"experiment_id":raw_dict.get("experiment_id","unknown"),

"model_type":raw_dict.get("model_type","rf"),

"features":raw_dict.get("features",set()),

"data_path":raw_dict.get("data_path",Path("./default_data.csv")),

"timestamp":raw_dict.get("timestamp",datetime.now()),

"learning_rate":raw_dict.get("learning_rate",0.001),# New schema hyperparameter

"nested_meta":raw_dict.get("nested_meta",{})

}

3. Discarding deprecated fields (e.g., if 'old_metric' was in raw_dict, optionally drop it here)

return validated_config

We explicitly added the defaulting mechanism for a "learning_rate" hyperparameter that didn’t exist in the original schema, but you can follow a similar approach for any other new hyperparameter that should be present in your updated model specifications.

Let’s look at how this enhanced strategy works by passing the original JSON string through the full pipeline:

1

2

3

4

5

6

7

8

9

10

11 print("Raw JSON being loaded:")

print(json_string)

print("\n"+"="*40+"\n")

Processing the raw, original JSON object through our robust loading and validation pipeline

final_config=load_experiment_config(json_string)

print("--- Validated Python Configuration ---")

for key,value in final_config.items():

Printing the key, the reconstructed value, and its underlying Python type

print(f"{key}: {value} (Type: {type(value).__name__})")

Results:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35 Raw JSON being loaded:

{

"experiment_id":1042,

"model_type":"nn",

"features":{

"__type__":"set",

"value":[

"income",

"location",

"age"

]

},

"data_path":{

"__type__":"path",

"value":"/data/v2/train.csv"

},

"timestamp":{

"__type__":"datetime",

"value":"2026-09-28T14:26:10.328310"

},

"nested_meta":{

"retries":3

}

}

========================================

---Validated Python Configuration---

experiment_id:1042(Type:int)

model_type:nn(Type:str)

features:{'income','location','age'}(Type:set)

data_path:/data/v2/train.csv(Type:PosixPath)

timestamp:2026-09-28 14:26:10.328310(Type:datetime)

learning_rate:0.001(Type:float)

nested_meta:{'retries':3}(Type:dict)

Wrapping Up

Through three practical steps, we have learned how to take full control of Python’s tools for serializing and deserializing JSON objects. These steps can be combined to form a robust serialization pipeline that handles custom data types which would otherwise raise errors. Not only can we handle nested JSON objects through the classes and functions we defined, but they are also designed to fail gracefully when malformed data is encountered. These techniques can meaningfully sharpen your data plumbing skills for machine learning engineering.

##### No comments yet.