Friday, 1 December 2023

GCP python script on VM cloud logging - No logs appearing

I am new to GCP / cloud services in general. If I have a simple python script below, how can I get it to show up in the google cloud logging explorer without modifying the python code? Is there something I need to enable on the gcp side?

import logging
import time
import sys

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')

logging.getLogger().setLevel(logging.INFO)

while True:
    print('****** print ******')
    time.sleep(3)
    logging.info('****** info ******')
    time.sleep(3)
    logging.critical('****** critical ******')
    time.sleep(3)
    sys.stdout.flush()

Here is what I've tried:

  1. Install the OPS agent when creating the VM
  2. All permissions in IAM are granted for logging
  3. Here are some of the commands I ran:
  • sudo journalctl -u google-fluentd -f
  • sudo journalctl -u google-fluentd -xe
  • sudo systemctl restart google-fluentd
  • sudo systemctl status google-fluentd

In cloud explorer I do see logs for when the fluentd service is restarted and other sys-logs like the one below, but I do not see any logs that are appearing in stdout (e.g. ****** print ******.

DEFAULT 2023-11-21T18:55:34Z [resource.labels.instanceId: test-service] Nov 21 18:55:34 debian systemd[1]: Starting GCE Workload Certificate refresh...
DEFAULT 2023-11-21T18:55:34Z [resource.labels.instanceId: test-service] Nov 21 18:55:34 debian gce_workload_cert_refresh[7493]: 2023/11/21 18:55:34: Done
DEFAULT 2023-11-21T18:55:34Z [resource.labels.instanceId: test-service] Nov 21 18:55:34 debian systemd[1]: gce-workload-cert-refresh.service: Succeeded.
DEFAULT 2023-11-21T18:55:34Z [resource.labels.instanceId: test-service] Nov 21 18:55:34 debian systemd[1]: Finished GCE Workload Certificate refresh.
DEFAULT 2023-11-21T18:55:38Z [resource.labels.instanceId: test-service] Nov 21 18:55:38 debian systemd[1]: Started Session 3 of user temp-user.
DEFAULT 2023-11-21T18:55:38Z [resource.labels.instanceId: test-service] Nov 21 18:55:38 debian dhclient[454]: XMT: Solicit on ens4, interval 122430ms.
DEFAULT 2023-11-21T18:57:41Z [resource.labels.instanceId: test-service] Nov 21 18:57:41 debian dhclient[454]: XMT: Solicit on ens4, interval 115540ms.
DEFAULT 2023-11-21T18:59:36Z [resource.labels.instanceId: test-service] Nov 21 18:59:36 debian dhclient[454]: XMT: Solicit on ens4, interval 117740ms.

Of course I am able to do something this:

import google.cloud.logging
import logging

client = google.cloud.logging.Client()
client.setup_logging()
logging.info('TEST')

The problem with the above is that I'm importing packages / SDKs where there are built-in logging / printing from the python standard library. I'm sure google doesn't think we're going to go in all of these huge packages and setup a google logging client. I basically just need everything that's sent from stdout to be also sent to google cloud logging.



from GCP python script on VM cloud logging - No logs appearing

No comments:

Post a Comment