Logs

All logs generated by Enterprise Chef can be found in /var/log/opscode. Each service enabled on the system also has a sub-directory in which service-specific logs are located, typically found in /var/log/opscode/service_name.

View Logs

Enterprise Chef has built-in support for easily tailing the logs that are generated. To view all the logs being generated on a Enterprise Chef server, enter the following command:

$ private-chef-ctl tail

To view logs for a specific service:

$ private-chef-ctl tail SERVICENAME

where SERVICENAME should be replaced with name of the service for which log files will be viewed.

Another way to view log files is to use the system utility tail:

$ tail -50f /var/log/opscode/opscode-chef/current

tail Log Files

The tail subcommand is used to follow all Enterprise Chef logs for all services. This command can also be run for an individual service by specifying the name of the service in the command.

This subcommand has the following syntax:

$ private-chef-ctl tail name_of_service

where name_of_service represents the name of any service that is listed after running the service-list subcommand.

Another common approach to tailing the log files for a service is to use the system utility tail. For example:

$ tail -50f /var/log/opscode/opscode-chef/current

Supervisor

Supervisor logs are created and managed directly by the service supervisor, and are automatically rotated when a the current log file reaches 1,000,000 bytes. 10 log files are kept. The latest supervisor log is always located in /var/log/opscode/service_name/current and rotated logs have a filename starting with @ followed by a precise tai64n timestamp based on when the file was rotated.

Supervisor logs are available for the following services:

  • bifrost
  • bookshelf
  • couchdb
  • nginx
  • opscode-account
  • opscode-certificate
  • opscode-erchef
  • opscode-expander
  • opscode-expander-reindexer
  • opscode-solr
  • opscode-webui
  • postgresql
  • rabbitmq
  • redis

nginx, access

Nginx is an important entry point for data on the Chef server, which means that debugging efforts frequently start with analyzing the nginx service’s access.log file. This log contains every HTTP request made to the front-end machine and can be very useful when investigating request rates and usage patterns. The following is an example log entry:

175.185.9.6 - - [12/Jul/2013:15:56:54 +0000] "GET
/organizations/exampleorg/data/firewall/nova_api HTTP/1.1" 200
"0.850" 452 "-" "Chef Client/0.10.2 (ruby-1.8.7-p302; ohai-0.6.4;
x86_64-linux; +http://opscode.com)" "127.0.0.1:9460" "200"
"0.849" "0.10.2" "version=1.0" "some_node.example.com"
"2013-07-12T15:56:40Z" "2jmj7l5rSw0yVb/vlWAYkK/YBwk=" 985

where important fields in this log include:

  • The HTTP status code (200)
  • The IP address of the requesting client (175.185.9.6)
  • The timestamp ([12/Jul/2013:15:56:54 +0000])
  • The total request time ("0.850")
  • The request method (GET)
  • The request URL (/organizations/exampleorg/data/firewall/nova_api)

opscode-account

The opscode-account service’s current.log file contains a history of errors and successful requests related to API endpoints used by this service. The log formats for this service are similar to the log format for the opscode-chef service.

opscode-erchef, current

The opscode-erchef service’s current.log file contains a history of stack traces from major application crashes.

opscode-erchef, erchef

The opscode-erchef service’s erchef.log file contains a history of API requests that have been processed by Erchef. These logs can be rotated quickly, therefore it is generally best to sort them by date, and then find the most recently updated log file:

$ ls -lrt /var/log/opscode/opscode-erchef/erchef.log.*

The following is an example log entry:

2013-08-06T08:54:32Z erchef@127.0.0.1 INFO org_name=srwjedoqqoypgmvafmoi; req_id=g3IAA2QAEGVyY2hlZkAx

where important fields in this log include:

  • The HTTP method (POST)
  • The HTTP path (/organizations/srwjedoqqoypgmvafmoi/environments)
  • The message ({created,<<"_default">>})
  • The organization name (org_name=srwjedoqqoypgmvafmoi)
  • The timestamp (2013-08-06T08:54:32Z)
  • The name of the user and/or chef-client which made the request (pivotal)

In addition, the log file may contain additional entries that detail the amounts of time spent interacting with other services:

  • authz_time (the time spent talking to the opscode-authz service)
  • couchdb_time (the time spent talking to the couchdb service)
  • rdbms_time (the time spent talking to the postgresql service)
  • req_time (the request time)
  • solr_time (the time spent talking to the opscode-solr service)

opscode-webui

The opscode-webui service’s current.log file contains a history of all requests that were made to the Chef management console. Errors that may be found in this log are typically the cause of failed HTTP requests that were made to another service. The following is an example log entry for a successful request:

2013-08-06_16:03:53.50265  Started GET "/nodes" for 172.28.0.148 at 2013-08-06 16:03:53 +0000
2013-08-06_16:03:53.50462  Processing by NodesController#index as HTML
2013-08-06_16:03:54.29635  Rendered nodes/_navigation.html.haml (11.5ms)
2013-08-06_16:03:54.29741  Rendered nodes/index.html.haml within layouts/application (36.5ms)
2013-08-06_16:03:54.31334  Completed 200 OK in 808ms (Views: 59.0ms)
2013-08-06_16:03:54.31447  cache: [GET /nodes] miss

A failed request, on the other hand produces a large amount of output. The following is a partial example log entry for a failed request:

2013-08-06_16:05:15.45445  Started GET "/clients" for 172.28.0.148 at 2013-08-06 16:05:15 +0000
2013-08-06_16:05:15.54797  Processing by ClientsController#index as HTML
2013-08-06_16:05:15.76841  [Tue, 06 Aug 2013 16:05:15 +0000] ERROR: Server returned error for http://1
2013-08-06_16:05:19.77373  [Tue, 06 Aug 2013 16:05:19 +0000] ERROR: Server returned error for http://1
2013-08-06_16:05:26.77612  [Tue, 06 Aug 2013 16:05:26 +0000] ERROR: Server returned error for http://1
2013-08-06_16:05:37.77876  [Tue, 06 Aug 2013 16:05:37 +0000] ERROR: Server returned error for http://1
2013-08-06_16:05:57.78131  [Tue, 06 Aug 2013 16:05:57 +0000] ERROR: Server returned error for http://1

[..SNIP..]

2013-08-06_16:06:51.32997  A Net::HTTPFatalError occurred in clients#index:
2013-08-06_16:06:51.32997
2013-08-06_16:06:51.32997    502 "Bad Gateway"
2013-08-06_16:06:51.32997    /opt/opscode/embedded/lib/ruby/1.9.1/net/http.rb:2632:in 'error!'
2013-08-06_16:06:51.32998
2013-08-06_16:06:51.32998  -------------------------------
2013-08-06_16:06:51.32998  Request:
2013-08-06_16:06:51.32998  -------------------------------
2013-08-06_16:06:51.32999
2013-08-06_16:06:51.32999    * URL   : https://172.31.6.215/clients
2013-08-06_16:06:51.32999    * IP address: 127.0.0.1
2013-08-06_16:06:51.32999    * Parameters: {"action"=>"index", "controller"=>"clients"}
2013-08-06_16:06:51.33000    * Rails root: /opt/opscode/embedded/service/opscode-webui
2013-08-06_16:06:51.33000
2013-08-06_16:06:51.33000  -------------------------------
2013-08-06_16:06:51.33000  Session:
2013-08-06_16:06:51.33001  -------------------------------

Application Logs

Application logs are created by the services directly, and may require log rotation policies to be applied based on organizational goals and the platform(s) on which the services are running.

nginx

The nginx service creates both supervisor and administrator logs. The administrator logs contain both access and error logs for each virtual host utilized by Enterprise Chef. Each of the following logs require external log rotation.

Logs Description
/var/log/opscode/nginx/access.log The Web UI and API HTTP access logs.
/var/log/opscode/nginx/error.log The Web UI and API HTTP error logs.
/var/log/opscode/nginx/internal-account.access.log The opscode-account internal load-balancer access logs.
/var/log/opscode/nginx/internal-account.error.log The opscode-account internal load-balancer error logs.
/var/log/opscode/nginx/internal-authz.access.log The opscode-authz internal load-balancer access logs.
/var/log/opscode/nginx/internal-authz.error.log The opscode-authz internal load-balancer error logs.
/var/log/opscode/nginx/internal-chef.access.log The opscode-chef and opscode-erchef internal load-balancer access logs.
/var/log/opscode/nginx/internal-chef.error.log The opscode-chef and opscode-erchef internal load-balancer error logs.
/var/log/opscode/nginx/nagios.access.log The nagios access logs.
/var/log/opscode/nginx/nagios.error.log The nagios error logs.
/var/log/opscode/nginx/rewrite-port-80.log The rewrite logs for traffic that uses HTTP instead of HTTPS.

To follow the logs for the service:

$ chef-server-ctl tail nginx

Read nginx Logs

The nginx access log format is as follows:

log_format opscode '$remote_addr - $remote_user [$time_local]  '
  '"$request" $status "$request_time" $body_bytes_sent '
  '"$http_referrer" "$http_user_agent" "$upstream_addr" '
  '"$upstream_status" "$upstream_response_time" "$http_x_chef_version" '
  '"$http_x_ops_sign" "$http_x_ops_userid" "$http_x_ops_timestamp" '
   '"$http_x_ops_content_hash" $request_length';

A sample log line:

192.168.4.1 - - [17/Feb/2012:16:02:42 -0800]
  "GET /organizations/nginx/cookbooks HTTP/1.1" 200
  "0.346" 12 "-"
  "Chef Knife/0.10.4 (ruby-1.9.3-p0;
                      ohai-0.6.10;
                      x86_64-darwin11.2.0;
                      +http://opscode.com
                      )"
  "127.0.0.1:9460" "200" "0.339" "0.10.4"
  "version=1.0" "adam" "2012-02-18T00:02:42Z"
  "2jmj7l5rSw0yVb/vlWAYkK/YBwk=" 871

Field descriptions:

Field Description
$remote_addr The IP address of the client who made this request.
$remote_user The HTTP basic auth user name of this request.
$time_local The local time of the request.
$request The HTTP request.
$status The HTTP status code.
$request_time The time it took to service the request.
$body_bytes_sent The number of bytes in the HTTP response body.
$http_referrer The HTTP referrer.
$http_user_agent The user agent of the requesting client.
$upstream_addr The upstream reverse proxy used to service this request.
$upstream_status The upstream reverse proxy response status code.
$upstream_response_time The upstream reverse proxy response time.
$http_x_chef_version The version of Chef used to make this request.
$http_x_ops_sign The version of the authentication protocol.
$http_x_ops_userid The client name that was used to sign this request.
$http_x_ops_timestamp The timestamp from when this request was signed.
$http_x_ops_content_hash The hash of the contents of this request.
$request_length The length of this request.