Integrating a model into the V2 API (CURRENT)

If you want to see an example of a module integrated with deepaas, where those methods are actually implemented, please head over to the deephdc demo app.

Defining what to load

The DEEPaaS API uses Python’s Setuptools entry points that are dynamically loaded to offer the model functionality through the API. This allows you to offer several models using a single DEEPaaS instance, by defining different entry points for the different models.

Warning

Serving multiple models is marked as deprecated, and will be removed in a future major version of the API. Please ensure that you start using the model-name configuration option in your configuration file or the –model-name command line option as soon as possible.

When the DEEPaaS API is spawned it will look for the deepaas.v2.model entrypoint namespace, loading and adding the names found into the API namespace. In order to define your entry points, your module should leverage setuptools and be ready to be installed in the system. Then, in order to define your entry points, you should add the following to your setup.cfg configuration file:

[entry_points]

deepaas.v2.model =
    my_model = package_name.module

This will define an entry point in the deepaas.v2.model namespace, called my_model. All the required functionality will be fetched from the package_name.module module. This means that module should provide the Entry point (model) API as described below.

If you provide a class with the required functionality, the entry point will be defined as follows:

[entry_points]

deepaas.v2.model =
    my_model = package_name.module:Class

Again, this will define an entry point in the deepaas.v2.model namespace, called my_model. All the required functionality will be fetched from the package_name.module.Class class, meaning that an object of Class will be created and used as entry point. This also means that Class objects should provide the Entry point (model) API as described below.

Entry point (model) API

Regardless on the way you implement your entry point (i.e. as a module or as an object), you should expose the following functions or methods:

Defining model metadata

Your model entry point must implement a get_medatata function that will return some basic metadata information about your model, as follows:

get_metadata(self)

Return metadata from the exposed model.

The metadata that is expected should follow the schema that is shown below. This basically means that you should return a dictionary with the following aspect:

{
    "author": "Author name",
    "description": "Model description",
    "license": "Model's license",
    "url": "URL for the model (e.g. GitHub repository)",
    "version": "Model version",
}

If you want to integrate with the deephdc platform you should provide at least an [name, author, author-email, license]. You can nevertheless set them to None if you don’t feel like providing the information.

The schema that we are following is the following:

{
    "id": =  fields.Str(required=True,
                        description='Model identifier'),
    "name": fields.Str(required=True,
                       description='Model name'),
    "description": fields.Str(required=True,
                              description='Model description'),
    "license": fields.Str(required=False,
                          description='Model license'),
    "author": fields.Str(required=False,
                         description='Model author'),
    "version": fields.Str(required=False,
                          description='Model version'),
    "url": fields.Str(required=False,
                      description='Model url'),
    "links": fields.List(
        fields.Nested(
            {
                "rel": fields.Str(required=True),
                "href": fields.Url(required=True),
            }
        )
    )
}
Returns:

dictionary containing the model’s metadata.

Warming a model

You can initialize your model before any prediction or train is done by defining a warm function. This function receives no arguments and returns no result, but it will be call before the API is spawned.

You can use it to implement any loading or initialization that your model may use. This way, your model will be ready whenever a first prediction is done, reducint the waiting time.

warm(self)

Warm (initialize, load) the model.

This is called when the model is loaded, before the API is spawned.

If implemented, it should prepare the model for execution. This is useful for loading it into memory, perform any kind of preliminary checks, etc.

Training

Regarding training there are two functions to be defined. First of all, you can specify the training arguments to be defined (and published through the API) with the get_train_args function, as follows:

get_train_args(self)

Return the arguments that are needed to train the application.

This function should return a dictionary of webargs fields (check here for more information). For example:

from webargs import fields

(...)

def get_train_args():
    return {
        "arg1": fields.Str(
            required=False,  # force the user to define the value
            missing="foo",  # default value to use
            enum=["choice1", "choice2"],  # list of choices
            description="Argument one"  # help string
        ),
    }
Return dict:

A dictionary of webargs fields containing the application required arguments.

Then, you must implement the training function (named train) that will receive the defined arguments as keyword arguments:

train(self, **kwargs)

Perform a training.

Parameters:

kwargs – The keyword arguments that the predict method accepts must be defined by the get_train_args() method so the API is able to pass them down. Usually you would populate these with all the training hyper-parameters

Returns:

You can return any Python object that is JSON parseable (eg. dict, string, float).

Prediction and inference

For prediction, there are different functions to be implemented. First of all, as for the training, you can specify the prediction arguments to be defined, (and published through the API) with the get_predict_args as follows:

get_predict_args(self)

Return the arguments that are needed to perform a prediction.

This function should return a dictionary of webargs fields (check here for more information). For example:

from webargs import fields

(...)

def get_predict_args():
    return {
        "arg1": fields.Str(
            required=False,  # force the user to define the value
            missing="foo",  # default value to use
            enum=["choice1", "choice2"],  # list of choices
            description="Argument one"  # help string
        ),
    }
Return dict:

A dictionary of webargs fields containing the application required arguments.

Do not forget to add an input argument to hold your data. If you want to upload files for inference to the API, you should use a webargs.fields.Field field created as follows:

def get_predict_args():
    return {
        "data": fields.Field(
            description="Data file to perform inference on.",
            required=False,
            missing=None,
            type="file",
            location="form")
     }

You can also predict data stored in an URL by using:

def get_predict_args():
    return {
        "url": fields.Url(
            description="Url of data to perform inference on.",
            required=False,
            missing=None)
     }

Important

do not forget to add the location="form" and type="file" to the argument definition, otherwise it will not work as expected.

Once defined, you will receive an object of the class described below for each of the file arguments you declare. You can open and read the file stored in the filename attribute.

class UploadedFile(name, filename, content_type, original_filename)

Class to hold uploaded field metadata when passed to model’s methods

name

Name of the argument where this file is being sent.

filename

Complete file path to the temporary file in the filesystem,

content_type

Content-type of the uploaded file

original_filename

Filename of the original file being uploaded.

Then you should define the predict function as indicated below. You will receive all the arguments that have been parsed as keyword arguments:

predict(self, **kwargs)

Prediction from incoming keyword arguments.

Parameters:

kwargs – The keyword arguments that the predict method accepts must be defined by the get_predict_args() method so the API is able to pass them down.

Returns:

The response must be a str or a dict.

By default, the return values from these two functions will be casted into a string, and will be returned in the following JSON response:

{
   "status": "OK",
   "predictions": "<model response as string>"
}

However, it is recommended that you specify a custom response schema. This way the API exposed will be richer and it will be easier for developers to build applications against your API, as they will be able to discover the response schemas from your endpoints.

In order to define a custom response, the schema attribute is used:

schema = None

Returning different content types

Sometimes it is useful to return something different than a JSON file. For such cases, you can define an additional argument accept defining the content types that you are able to return as follows:

def get_predict_args():
    return {
        'accept': fields.Str(description="Media type(s) that is/are acceptable for the response.",
                             missing='application/zip',
                             validate=validate.OneOf(['application/zip', 'image/png', 'application/json']))
     }

Find in this link a comprehensive list of possible content types. Then the predict function will have to return the raw bytes of a file according to the user selection. For example:

def predict(**args):
    # Run your prediction

    # Return file according to user selection
    if args['accept'] == 'image/png':
        return open(img_path, 'rb')

    elif args['accept'] == 'application/json':
        return {'some': 'json'}

    elif args['accept'] == 'application/zip':
        return open(zip_path, 'rb')

If you want to return several content types at the same time (let’s say a JSON and an image), the easiest way it to return a zip file with all the files.

Using classes

Apart from using a module, you can base your entrypoints on classes. If you want to do so, you may find useful to inhering from the deepaas.model.v2.base.BaseModel abstract class:

class BaseModel

Base class for all models to be used with DEEPaaS.

Note that it is not needed for DEEPaaS to inherit from this abstract base class in order to expose the model functionality, but the entrypoint that is configured should expose the same API.

abstract get_metadata()

Return metadata from the exposed model.

The metadata that is expected should follow the schema that is shown below. This basically means that you should return a dictionary with the following aspect:

{
    "author": "Author name",
    "description": "Model description",
    "license": "Model's license",
    "url": "URL for the model (e.g. GitHub repository)",
    "version": "Model version",
}

If you want to integrate with the deephdc platform you should provide at least an [name, author, author-email, license]. You can nevertheless set them to None if you don’t feel like providing the information.

The schema that we are following is the following:

{
    "id": =  fields.Str(required=True,
                        description='Model identifier'),
    "name": fields.Str(required=True,
                       description='Model name'),
    "description": fields.Str(required=True,
                              description='Model description'),
    "license": fields.Str(required=False,
                          description='Model license'),
    "author": fields.Str(required=False,
                         description='Model author'),
    "version": fields.Str(required=False,
                          description='Model version'),
    "url": fields.Str(required=False,
                      description='Model url'),
    "links": fields.List(
        fields.Nested(
            {
                "rel": fields.Str(required=True),
                "href": fields.Url(required=True),
            }
        )
    )
}
Returns:

dictionary containing the model’s metadata.

abstract get_predict_args()

Return the arguments that are needed to perform a prediction.

This function should return a dictionary of webargs fields (check here for more information). For example:

from webargs import fields

(...)

def get_predict_args():
    return {
        "arg1": fields.Str(
            required=False,  # force the user to define the value
            missing="foo",  # default value to use
            enum=["choice1", "choice2"],  # list of choices
            description="Argument one"  # help string
        ),
    }
Return dict:

A dictionary of webargs fields containing the application required arguments.

abstract get_train_args()

Return the arguments that are needed to train the application.

This function should return a dictionary of webargs fields (check here for more information). For example:

from webargs import fields

(...)

def get_train_args():
    return {
        "arg1": fields.Str(
            required=False,  # force the user to define the value
            missing="foo",  # default value to use
            enum=["choice1", "choice2"],  # list of choices
            description="Argument one"  # help string
        ),
    }
Return dict:

A dictionary of webargs fields containing the application required arguments.

abstract predict(**kwargs)

Prediction from incoming keyword arguments.

Parameters:

kwargs – The keyword arguments that the predict method accepts must be defined by the get_predict_args() method so the API is able to pass them down.

Returns:

The response must be a str or a dict.

schema = None

Must contain a valid schema for the model’s predictions or None.

A valid schema is either a marshmallow.Schema subclass or a dictionary schema that can be converted into a schema.

In order to provide a consistent API specification we use this attribute to define the schema that all the prediction responses will follow, therefore: - If this attribute is set we will validate them against it. - If it is not set (i.e. schema = None), the model’s response will be converted into a string and the response will have the following form:

{
    "status": "OK",
    "predictions": "<model response as string>"
}

As previously stated, there are two ways of defining an schema here. If our response have the following form:

{
    "status": "OK",
    "predictions": [
        {
            "label": "foo",
            "probability": 1.0,
        },
        {
            "label": "bar",
            "probability": 0.5,
        },
    ]
}

We should define or schema as schema as follows:

  • Using a schema dictionary. This is the most straightforward way. In order to do so, you must use the marshmallow Python module, as follows:

    from marshmallow import fields
    
    schema = {
        "status": fields.Str(
                    description="Model predictions",
                    required=True
        ),
        "predictions": fields.List(
            fields.Nested(
                {
                    "label": fields.Str(required=True),
                    "probability": fields.Float(required=True),
                },
            )
        )
    }
    
  • Using a marshmallow.Schema subclass. Note that the schema must be the class that you have created, not an object:

    import marshmallow
    from marshmallow import fields
    
    class Prediction(marshmallow.Schema):
        label = fields.Str(required=True)
        probability = fields.Float(required=True)
    
    class Response(marshmallow.Schema):
        status = fields.Str(
            description="Model predictions",
            required=True
        )
        predictions = fields.List(fields.Nested(Prediction))
    
    schema = Response
    
abstract train(**kwargs)

Perform a training.

Parameters:

kwargs – The keyword arguments that the predict method accepts must be defined by the get_train_args() method so the API is able to pass them down. Usually you would populate these with all the training hyper-parameters

Returns:

You can return any Python object that is JSON parseable (eg. dict, string, float).

abstract warm()

Warm (initialize, load) the model.

This is called when the model is loaded, before the API is spawned.

If implemented, it should prepare the model for execution. This is useful for loading it into memory, perform any kind of preliminary checks, etc.

Warning

The API uses multiprocessing for handling tasks. Therefore if you use decorators around your methods, please follow best practices and use functools.wraps so that the methods are still pickable. Beware also of using global variables that might not be shared between processes.