limkopi ☕️

security | dev | finance | life

github instagram
A Better Logging with Python Logging, structlog and pythonjsonlogger
Dec 23, 2017
3 minutes read

If you haven’t been using logging in your python code, then you should stop using python.

Just kidding.

logging helps a lot in structuring logs. I mean you won’t want to manually write your logging patterns (e.g. YYYY-MM-DD [class_name.func_name] msg) in every function, right?

Getting Start with Python logging

The simplest way to start logging:

#!/usr/bin/python2
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

logger.info("Hello World")
logger.critical("Hello World")
logger.warning("Hello World")
logger.error("Hello World")

In this case, you are going to log all your messages at INFO level and above. We can adjust the logger level:

logging.basicConfig(level=logging.DEBUG)

Why is this important?

Remember the time when you see 10,000 lines of logs and you have no idea where you should start? If you are able to set different level of logging, you can direct the logs to different stream. For example:

import logging
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

handler = logging.FileHandler("debug.log")
handler.setLevel(logging.DEBUG)

logger.addHandler(handler)

handler = logging.FileHandler("error.log")
handler.setLevel(logging.ERROR)

logger.addHandler(handler)

This is very useful when you are dealing with tremendous amount of logs. There are a lot more you can do with logging. You can also format the log output such that it’s pretty printed:

import logging
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)

# create ouput to stdoutput at ERROR level
ch = logging.StreamHandler()
ch.setLevel(logging.ERROR)

# create a format for that
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
ch.setFormatter(formatter)

logger.addHandler(ch)

logger.error("Hello World")

This is what you will get on your screen: 2017-12-23 10:31:49,624 - __main__ - ERROR - Hello World

Much better, isn’t it?

Getting Started With structlog and pythonjsonlogger

You can structure your log better with structlog. Imagine you have 2 variables you will always need to log, you can add them using this library easily like a true variable.

This is what you can do:

from structlog import get_logger
log = get_logger()
log.info("log for fun", error=0, type="for fun")

This is the output: 2017-12-23 10:36.54 log for fun error=0 type=for fun

This is very convenient especially if you are logging for different apps.

Sometimes it is necessary to output your logs into json or dict format.

By default, logging does not support such format. One can argue that you can do it through the formatter but my stand is always use a standard library.

Imagine you have to create a dict for every type of logging that you want to do in your project, I think you will take more time formatting your log than to write your actual codes.

python-json-logger exists for such problem. It has a JSON formatter, which you can use freely with structlog.

from pythonjsonlogger import jsonlogger
import structlog
import logging

# configure structlog to render logging-based Formatter
structlog.configure(
    processors=[
        structlog.stdlib.filter_by_level,
        structlog.stdlib.add_logger_name,
        structlog.stdlib.add_log_level,
        structlog.stdlib.PositionalArgumentsFormatter(),
        structlog.processors.StackInfoRenderer(),
        structlog.processors.format_exc_info,
        structlog.processors.UnicodeDecoder(),
        structlog.stdlib.render_to_log_kwargs,
    ],
    context_class=dict,
    logger_factory=structlog.stdlib.LoggerFactory(),
    wrapper_class=structlog.stdlib.BoundLogger,
    cache_logger_on_first_use=True,
)

# configure logging formatter just like before with python-json-logger's formatter
handler = logging.StreamHandler()
handler.setFormatter(jsonlogger.JsonFormatter())

logger = logging.getLogger()
logger.addHandler(handler)

structlog.get_logger("log for fun").warning("hello", error=0, type="for funs")

This is what you will get in the output screen:

{"message": "hello", "logger": "log for fun", "type": "for funs", "level": "warning", "error": 0}

The beauty of logging lies in your effort in making the output readable and useful. Hopefully this can help anyone who’s struggling to keep up their logs.


category: dev

Back to posts