Authenticating GMail API access programmatically

In order to round-trip test mail sending code in our GCP backend I am sending an email to a GMail inbox and attempting to verify its arrival. The current mechanism for authentication to the GMail API is fairly standard, pasted from the GMail API documentation and embedded in a function:

def authenticate():
    """Authenticates to the Gmail API using data in credentials.json,
    returning the service instance for use in queries etc."""
    store = file.Storage('token.json')
    creds = store.get()
    if not creds or creds.invalid:
        flow = client.flow_from_clientsecrets(CRED_FILE_PATH, SCOPES)
        creds = tools.run_flow(flow, store)
    service = build('gmail', 'v1', http=creds.authorize(Http()))
    return service

CRED_FILE_PATH points to a downloaded credentials file for the service. The absence of the token.json file triggers its re-creation after an authentication interaction via a browser window, as does the token's expiry.

This is an integration test that must run headless (i.e. with no interaction whatsoever). When re-authentication is required the test currently raises an exception when the authentication flow starts to access sys.argv, which means it sees the arguments to pytest!

I've been trying to find out how to authenticate reliably using a mechanism that does not require user interaction (such as an API key). Nothing in the documentation or on Stackoverflow seems to answer this question.

A more recent effort uses the keyfile from a service account with GMail delegation to avoid the interactive Oauth2 flows.

def authenticate():
    """Authenticates to the Gmail API using data in g_suite_access.json,
    returning the service instance for use in queries etc."""
    main_cred = service_account.Credentials.from_service_account_file(
        CRED_FILE_PATH, scopes=SCOPES)
    # Establish limited credential to minimise any damage.
    credentials = main_cred.with_subject(GMAIL_USER)
    service = build('gmail', 'v1', credentials=credentials)
    return service

On trying to use this service with

        response = service.users().messages().list(userId='me',
                                    q=f'subject:{subject}').execute()

I get:

google.auth.exceptions.RefreshError:
  ('unauthorized_client: Client is unauthorized to retrieve access tokens using this method.',
   '{\n "error": "unauthorized_client",\n "error_description": "Client is unauthorized to retrieve access tokens using this method."\n}')

I get the feeling there's something fundamental I'm not understanding.

from Authenticating GMail API access programmatically