Query Topics

query-topics is an operation for efficient searching of Volga topics at multiple sites. You can modify the search basically by:

• Choosing which sites to run the query on
• Choosing which topics to run the query towards
• Choosing what to search for
• Choosing how the output result should be presented

All parameters are individually documented, but here we present some example inputs.

Note that the query is always issued at the Control Tower.

By default, the Control Tower will make a best-effort attempt at sorting the incoming messages by timestamp, but it will prioritize throughput, so the order is not guaranteed to be correct. If timestamp ordering is important, use the sort option.

First example: Search a topic name called mytext on all sites and just print the data from that topic.

{
  "follow": true,
  "all-sites": true,
  "topics": [
    {
      "topic-names": ["mytext"],
      "output": {"payload-only": true}
    }
  ]
}

Next example is similar, searches the same topic, but only entries since one hour ago and forward, and it will grep for the string Login:

{
  "follow": false,
  "since": "1h",
  "all-sites": true,
  "topics": [
    {
      "topic-names": ["mytext"],
      "filter": {"re-match": "Login:"},
      "output": {"payload-only": true}
    }
  ]
}

When applications are deployed, several interesting system: topics are created by the system. For example scheduling information is available in a topic called system:scheduler-events. If an application, called cowboy-app is deployed on many sites, the following JSON queries all sites. The output data on the topic is JSON, and in that case, we can use the fields output modifier to only output certain parts of the JSON log structure

{
  "follow": false,
  "topics": [
    {
      "topic-names": ["system:scheduler-events"],
      "filter": {"re-match": "\"application\": \"cowboy-app\""},
      "output": {
        "payload-only": true,
        "fields": "data/[application,oper-status]"
      }
    }
  ]
}

When containers are deployed, the output on stdout,stderr from the container is collected in a container topic. If you are an application developer, searching and streaming the container output topics is obviously important. The following query searches for errors in all containers called cowboy. The query uses a few advanced features, first the query is not run at all sites, but rather the sites where our container is actually deployed. It also choses the topics, using labels instead of the topic names. Follow is true, thus this is basically tailing all our application logs, looking for errors.

{
  "follow": true,
  "sites-from-application-deployment": "cowboy-dep",
  "topics": [
    {
      "match-topic-labels": "container-name = cowboy",
      "filter": {
        "re-opts": ["caseless"],
        "re-match": "error|emerg|critical"
      },
      "output": {"payload-only": true}
    }
  ]
}