python-logging - w3toppers.com

Run subprocess and print output to logging

You could try to pass the pipe directly without buffering the whole subprocess output in memory: from subprocess import Popen, PIPE, STDOUT process = Popen(command_line_args, stdout=PIPE, stderr=STDOUT) with process.stdout: log_subprocess_output(process.stdout) exitcode = process.wait() # 0 means success where log_subprocess_output() could look like: def log_subprocess_output(pipe): for line in iter(pipe.readline, b”): # b’\n’-separated lines logging.info(‘got line from … Read more

How to use logging.getLogger(name) in multiple modules

As pointed out by @shmee in this answer, the logger hierarchy must be defined explicitly in the logger name, using dot-notation. That is, if the logger name in main_module.py is e.g. ‘a’, then the logger name in auxiliary_module.py must be ‘a.b’ (not just ‘b’), in order for it to inherit the configuration of logger ‘a’. … Read more

Django Setup Default Logging

Figured it out… You set the ‘catch all’ logger by referencing it with the empty string: ”. As an example, in the following setup I have the all log events getting saved to logs/mylog.log, with the exception of django.request log events which will be saved to logs/django_request.log. Because ‘propagate’ is set to False for my … Read more

Python logging: use milliseconds in time format

This should work too: logging.Formatter( fmt=”%(asctime)s.%(msecs)03d”, datefmt=”%Y-%m-%d,%H:%M:%S” )

How to obtain a Thread id in Python?

threading.get_ident() works, or threading.current_thread().ident (or threading.currentThread().ident for Python < 2.6).

Naming Python loggers

I typically don’t use or find a need for class-level loggers, but I keep my modules at a few classes at most. A simple: import logging LOG = logging.getLogger(__name__) At the top of the module and subsequent: LOG.info(‘Spam and eggs are tasty!’) from anywhere in the file typically gets me to where I want to … Read more

Suppress newline in Python logging module

If you wanted to do this you can change the logging handler terminator. I’m using Python 3.4. This was introduced in Python 3.2 as stated by Ninjakannon. handler = logging.StreamHandler() handler.terminator = “” When the StreamHandler writes it writes the terminator last.

python logging to database

I recently managed to write my own database logger in Python. Since I couldn’t find any example I thought I post mine here. Works with MS SQL. Database table could look like this: CREATE TABLE [db_name].[log]( [id] [bigint] IDENTITY(1,1) NOT NULL, [log_level] [int] NULL, [log_levelname] [char](32) NULL, [log] [char](2048) NOT NULL, [created_at] [datetime2](7) NOT NULL, … Read more

How to write custom python logging handler?

import logging class ProgressConsoleHandler(logging.StreamHandler): “”” A handler class which allows the cursor to stay on one line for selected messages “”” on_same_line = False def emit(self, record): try: msg = self.format(record) stream = self.stream same_line = hasattr(record, ‘same_line’) if self.on_same_line and not same_line: stream.write(self.terminator) stream.write(msg) if same_line: stream.write(‘… ‘) self.on_same_line = True else: stream.write(self.terminator) self.on_same_line … Read more

How do I log from my Python Spark script

You can get the logger from the SparkContext object: log4jLogger = sc._jvm.org.apache.log4j LOGGER = log4jLogger.LogManager.getLogger(__name__) LOGGER.info(“pyspark script logger initialized”)