Skip to content
This repository was archived by the owner on Aug 13, 2026. It is now read-only.
This repository was archived by the owner on Aug 13, 2026. It is now read-only.

[Logging] page_size appears to not work; no iterator(s) #203

Closed

Description

@DazWilkin

See #136

Code

requirements.txt:

google-auth==1.27.0
google-cloud-logging==2.2.0

And main.py (derived from the Google published sample)

Using client.list_entries:

import os

from google.cloud import logging
from google.cloud.logging import DESCENDING

client = logging.Client()

PROJECT = os.getenv("PROJECT")
resource = "projects/{project}".format(project=PROJECT)
log_name = "{resource}/logs/cloudaudit.googleapis.com%2Factivity".format(
    resource=resource)
filter = "logName=\"{log_name}\"".format(log_name=log_name)

entries = client.list_entries(filter_=filter,
                              order_by=DESCENDING,
                              page_size=3,
                              resource_names=[resource])

print("Listing entries for logger {}:".format(log_name))
for entry in entries:
    timestamp = entry.timestamp.isoformat()
    print("{}".format(timestamp))

Using logger.list_entries:

import os

from google.cloud import logging
from google.cloud.logging import DESCENDING

client = logging.Client()

log_name = "cloudaudit.googleapis.com%2Factivity"
logger = client.logger(log_name)

entries = logger.list_entries(
    order_by=DESCENDING,
    page_size=3,
)

print("Listing entries for logger {}:".format(log_name))
for entry in entries:
    timestamp = entry.timestamp.isoformat()
    print("{}".format(timestamp))

Steps to reproduce

Running the Python:

python3 cloud/main.py 
Listing entries for logger projects/dazwilkin-210302-66371557/logs/cloudaudit.googleapis.com%2Factivity:
2021-03-02T17:39:09.421835+00:00
2021-03-02T17:39:12.121825+00:00
2021-03-02T17:39:12.933201+00:00
2021-03-02T17:39:16.011850+00:00
2021-03-02T17:39:17.114823+00:00
2021-03-02T17:39:18.699000+00:00
2021-03-02T17:41:29.759709+00:00
2021-03-02T17:41:30.937346+00:00
2021-03-02T17:41:32.653000+00:00

NOTE Neither DESCENDING nor limited

Running the equivalent (!) gcloud yields the expected result:

gcloud logging read "logName=\"projects/${PROJECT}/logs/cloudaudit.googleapis.com%2Factivity\"" \
--project=${PROJECT} \
--order=desc \
--limit=3 \
--format="value(timestamp)" \
--log-http

...
== body start ==
{
  "filter": "timestamp>=\"2021-03-01T19:14:21.929877Z\" AND logName=\"projects/${PROJECT{/logs/cloudaudit.googleapis.com%2Factivity\"",
  "orderBy": "timestamp desc",
  "pageSize": 3,
  "resourceNames": ["projects/${PROJECT}"]
}
== body end ==
...
2021-03-02T17:41:32.653Z
2021-03-02T17:41:30.937346481Z
2021-03-02T17:41:29.759709770Z

Additionally:

  • The list_entries result should be an interator that can be paged; it is not
  • The library 'encourages' repetition of resource|project (logging_client.logger, filter and resource_names) and it's unclear how I can only specify this once

Activity

  1. daniel-sanche commented on Mar 3, 2021

    @daniel-sanche
    Contributor

    I believe this issue is related to #154: We currently have two different implementations (grpc and http), which are implemented in inconsistent ways. This results in the gRPC layer ignoring some of the arguments. Unfortuantely a fix will likely have to wait until our next major relase, because it'll have to be a breaking change.

    In the mean time, you can try to use HTTP mode instead of gRPC mode, which seems to work as you expect. You can turn it on with client = logging.Client(_use_grpc=0) or setting the DISABLE_GRPC environment variable to any value.

    Note though that if your code iterates through all pages. If you just want a single page, you have to iterate differently:

    for entry in next(entries.pages):
         timestamp = entry.timestamp.isoformat()
         print("{}".format(timestamp))
    

    https://googleapis.dev/python/google-api-core/latest/page_iterator.html

  2. added
    priority: p2Moderately-important priority. Fix may not be included in next release.
    type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.
    on Mar 3, 2021
  3. added this to the v3.0.0 Breaking milestone on Mar 3, 2021
  4. DazWilkin commented on Mar 3, 2021

    @DazWilkin
    Author

    It does not work (nor as documented by #154).

    env | grep GRPC
    DISABLE_GRPC=true
    python3 cloud/main.py
    Listing entries for logger cloudaudit.googleapis.com%2Factivity:
    2021-03-03T17:27:32.162000+00:00
    2021-03-03T17:27:30.517190+00:00
    2021-03-03T17:27:29.339759+00:00
    2021-03-03T17:25:47.088000+00:00
    2021-03-03T17:25:42.505007+00:00
    2021-03-03T17:25:42.115940+00:00
    2021-03-03T17:25:39.407426+00:00
    2021-03-03T17:25:38.717723+00:00
    2021-03-03T17:25:36.399174+00:00

    NOTE I'm using a new project today so the entries are different but the issue remains (all results instead of 3)

    Yields:

    • All results (expect 3)
    • method returns generator (suggesting DISABLE_GRPC is ineffective!?)

    And (unset DISABLE_GRPC):

    NOTE Elsewhere this variable is referenced as GOOGLE_CLOUD_DISBABLE_GRPC: True. I tried that and this then returns an google.api_core.page_iterator.HTTPIterator but all results

    client = logging.Client(_use_grpc=0)

    Yields:

    • All results (expect 3)
    • method returns an google.api_core.page_iterator.HTTPIterator

    I used a slightly different mechanism on the iterator:

    iterator = logger.list_entries(
        order_by=DESCENDING,
        page_size=3,
    )
    print(type(iterator))
    
    print("Listing entries for logger {}:".format(log_name))
    for page in iterator.pages:
        for entry in page:
            timestamp = entry.timestamp.isoformat()
            print("{}".format(timestamp))
  5. daniel-sanche commented on Mar 3, 2021

    @daniel-sanche
    Contributor

    NOTE Elsewhere this variable is referenced as GOOGLE_CLOUD_DISBABLE_GRPC: True. I tried that and this then returns an google.api_core.page_iterator.HTTPIterator but all results

    Sorry, you're right. The variable is called DISABLE_GRPC in the code, but it looks like the real name to use is GOOGLE_CLOUD_DISBABLE_GRPC

    I used a slightly different mechanism on the iterator:

    The problem with your code is the double loop. You are through iterating all pages, along with all entries in each page. This will give you all entries, just loaded in batches of 3. If you just want all 3 entries from a single page, you'll have to do something like the code I posted above:

    for entry in next(entries.pages):
         timestamp = entry.timestamp.isoformat()
         print("{}".format(timestamp))
    

    This should give you all 3 results on the first page of results, which seems to be what you're looking for. But let me know if you're still having issues.

  6. DazWilkin commented on Mar 3, 2021

    @DazWilkin
    Author

    Doh! Thanks. Yes, of course.

    I was thinking I would get one page of three results but I'm actually getting n pages of 3 results.

    Thank you!

  7. added
    🚨This issue needs some love.
    and removed
    🚨This issue needs some love.
    on Jun 2, 2021
  8. keenan-devrel commented on Sep 16, 2021

    @keenan-devrel

    This should be addressed in the upcoming 3.0 release planned for Q4 2021.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

api: loggingIssues related to the googleapis/python-logging API.priority: p2Moderately-important priority. Fix may not be included in next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions