Install and Setup ADFS 2.0 on a Windows Server

ADFS, Blog, Information Technology, Microsoft, Servers, Software, Windows

Note: This guide assumes you have a Windows server, it is joined to a domain and you have an account that is part of the domain admins group.

First we need to install ADFS 2.0

To install version 2.0, it first needs to be downloaded and installed. Click this link to download the installation file https://www.microsoft.com/en-au/download/details.aspx?id=10909
Choose the type of installation file based on your OS (I am running Server 2008 R2 so I will choose RTW\W2K8R2\amd64\AdfsSetup.exe). Save the installation file to your Desktop
Run the exe located on your desktop
On the Welcome screen, click Next
Accept the License Agreement.
I will be setting this ADFS server as a Federation server so I will leave the default of Federation Server selected and I will click Next. Note: you would select Federation server proxy if you were running your ADFS server in something like a DMZ.
Click Next on the prerequisite software screen. The install wizard will install these automatically if they are not already present on the system. If there are no errors, the software will be installed and the ADFS 2.0 management window will open

The next task is to create a certificate template to use with our SSL certificate

The first thing you need to do is open the Server Manager (Start > Administrative Tools > Server Manager)
Select Roles
Click Add Roles
Click Next to get past the Welcome Screen
Check off Active Directory Certificate Services and click Next
Click Next to get past the Welcome Screen
Click Next on the Role Services page
Leave the default of Enterprise selected and click Next. Note: if Enterprise is not an option, you must log out of the server and log back in using an account that is a member of the Domain Administrators group.
On the next page you need to decide between choosing a Root CA or a Subordinate CA. Select Root CA if you are OK with using a self-signed certificate for your ADFS instance. Select Subordinate CA if you would like to purchase a certificate from a 3^rd party certificate authority. I will be using a self-signed certificate so I will select Root CA and click Next.
Click Next on the Private Key page to create a new private key.
Choose the type of cryptography you would like to use on the next page. I will leave the default of SHA1 with a 2048 bit key character length, but you can choose whatever you want. Then click Next
Click Next on the CA Name page
Choose how long you want the certificate to be valid. I will leave the default of 5 years set. Click Next
Click Next on the Certificate Database page
Click Install to finish and then close the wizard when complete
Next open up the Certificate Authority (Start > Administrative Tools > Certificate Authority)
Expand your server on the left column, right click Certificate Templates and click Manage
Scroll down to the end of the list, right click Web Server and click Duplicate Template
Select Windows Server 2008 Enterprise and click OK. Note: you would select Windows Server 2003 only if you had 2003 servers on your network that needed to access the ADFS services.
Give the template a name (ex: ADFS SSL Certificate)
Next click on the Subject Name tab at the top.
Click the Build from this Active Directory information radio button, select Common Name from the subject name format dropdown, uncheck User principal name (UPN) and check DNS name
In order for the ADFS server to obtain a certificate, it needs to have the correct permission set up to do so. To do this, select the Security tab at the top.
To add the required permissions, click the Add
Click the Object Types… button, check Computers on the newly opened window and click OK.
Enter the name of the server that you previously installed ADFS on and click OK. The server will be added to the permission list.
Next we need to set Enroll the allow permission for this particular server. Check Enroll and click OK
Now we need to make this certificate template is available to be used to issue certificates. To do this, open up the Certificate Authority window again (Start > Administrative Tools > Certificate Authority)
Right click the Certificate Templates icon in the left column, hover over New and click Certificate Template to Issue
Select the name of the template you made previously (in my example I made it ADFS SSL Certificate) and click OK.

The next task is to create a SSL certificate on the server that is compatible with ADFS

Open up a management console window on your server (Start, type in mmc.exe and hit enter)
Click file and click Add or Remove Snap-ins
Highlight Certificates and click Add
Select Computer Account and click Next and then click Finish
Now that the options are configured, click OK
On the Console window, click View and then select Options
Select Certificate purpose and then click This will allow us to view the computer’s certificate by their purpose.
In the Intended Purposes view, right click Server Authentication, hover over All Tasks and click Request New Certificate
Click Next on the welcome screen.
Click Next on the Select Certificate Enrollment Policy
Select the template you created before (ex: ADFS SSL Certificate) by checking it and clicking Enroll.
Click Finish and close all of the open windows.

The next task will be to configure ADFS

Open up ADFS management (Start > Administrative tools > AD FS 2.0 Management)
On the ADFS 2.0 management window, click ADFS 2.0 Federation Server Configuration Wizard
Leave the default of Create a new Federation Service selected and click Next
On the Select Deployment Type page, select Stand-alone federation server and click Next
Make sure your SSL certificate is selected on the Federation Service Name page and click Next
Click Next on the summary screen and wait for the wizard to complete.
One the wizard is finished, click Close. Now ADFS is installed and ready to be used.

Send Cisco ASA Syslogs to Elasticsearch Using Logstash

Blog, ElasticSearch, Information Technology, Kibana, Logstash, Networking, Software

This guide is a continuation of this blog post here. The following assumes that you already have an Elasticsearch instance set up and ready to go. This post will walk you through installing and setting up logstash for sending Cisco ASA messages to an Elasticsearch index.

As of today (6/16/2015), version 1.5.1 is the latest stable release of Logstash so I will be using 1.5.1 in my setup guide.

OPTIONAL

Before we begin sending data to Elasticsearch, it is probably a good idea to set up a custom template in Elasticsearch. This will set it so that specific fields are set for the correct types of data.

For example: By default, all data passed to a new index in elasticsearch is treated as a string type. You wouldn’t want this for something like the bytes field in case you wanted to add up all the bytes for a specific time window search. So you create a custom template so that the bytes field is stored as type long. This is totally optional but it is probably in your benifit to do this before you start collecting your ASA syslogs. You can read more about custom elasticsearch templates here or you can just copy mine by doing this…

Copy the text below, change the IP address at the top to the IP of your elasticsearch server and save it as template.sh on your desktop.

#!/bin/sh
curl -XPUT http://10.0.0.112:9200/_template/logstash_per_index -d '
{
    "template" : "logstash*",
    "mappings" : {
      "cisco-fw" : {
         "properties": {
            "@timestamp":{"type":"date","format":"dateOptionalTime"},
            "@version":{"type":"string", "index" : "not_analyzed"},
	    "action":{"type":"string"},
	    "bytes":{"type":"long"},
	    "cisco_message":{"type":"string"},
	    "ciscotag":{"type":"string", "index" : "not_analyzed"},
	    "connection_count":{"type":"long"},
            "connection_count_max":{"type":"long"},
	    "connection_id":{"type":"string"},
            "direction":{"type":"string"},
            "dst_interface":{"type":"string"},
	    "dst_ip":{"type":"string"},
            "dst_mapped_ip":{"type":"ip"},
	    "dst_mapped_port":{"type":"long"},
            "dst_port":{"type":"long"},
            "duration":{"type":"string"},
	    "err_dst_interface":{"type":"string"},
	    "err_dst_ip":{"type":"ip"},
	    "err_icmp_code":{"type":"string"},
	    "err_icmp_type":{"type":"string"},
	    "err_protocol":{"type":"string"},
            "err_src_interface":{"type":"string"},
            "err_src_ip":{"type":"ip"},
            "geoip":{
               "properties":{
                  "area_code":{"type":"long"},
                  "asn":{"type":"string", "index":"not_analyzed"},
                  "city_name":{"type":"string", "index":"not_analyzed"},
                  "continent_code":{"type":"string"},
                  "country_code2":{"type":"string"},
                  "country_code3":{"type":"string"},
                  "country_name":{"type":"string", "index":"not_analyzed"},
                  "dma_code":{"type":"long"},
                  "ip":{"type":"ip"},
                  "latitude":{"type":"double"},
                  "location":{"type":"geo_point"},
                  "longitude":{"type":"double"},
                  "number":{"type":"string"},
                  "postal_code":{"type":"string"},
                  "real_region_name":{"type":"string", "index":"not_analyzed"},
                  "region_name":{"type":"string", "index":"not_analyzed"},
                  "timezone":{"type":"string"}
               }
            },
            "group":{"type":"string"},
 	    "hashcode1": {"type": "string"}, 
 	    "hashcode2": {"type": "string"}, 
            "host":{"type":"string"},
            "icmp_code":{"type":"string"},
            "icmp_code_xlated":{"type":"string"},
            "icmp_seq_num":{"type":"string"},
            "icmp_type":{"type":"string"},
            "interface":{"type":"string"},
            "is_local_natted":{"type":"string"},
            "is_remote_natted":{"type":"string"},
            "message":{"type":"string"},
            "orig_dst_ip":{"type":"ip"},
            "orig_dst_port":{"type":"long"},
            "orig_protocol":{"type":"string"},
            "orig_src_ip":{"type":"ip"},
            "orig_src_port":{"type":"long"},
            "policy_id":{"type":"string"},
            "protocol":{"type":"string"},
            "reason":{"type":"string"},
            "seq_num":{"type":"long"},
            "spi":{"type":"string"},
            "src_interface":{"type":"string"},
            "src_ip":{"type":"ip"},
            "src_mapped_ip":{"type":"ip"},
            "src_mapped_port":{"type":"long"},
            "src_port":{"type":"long"},
            "src_xlated_interface":{"type":"string"},
            "src_xlated_ip":{"type":"ip"},
            "syslog_facility":{"type":"string"},
            "syslog_facility_code":{"type":"long"},
            "syslog_pri":{"type":"string"},
            "syslog_severity":{"type":"string"},
            "syslog_severity_code":{"type":"long"},
            "tags":{"type":"string"},
            "tcp_flags":{"type":"string"},
            "timestamp":{"type":"string"},
            "tunnel_type":{"type":"string"},
            "type":{"type":"string"},
            "user":{"type":"string"},
            "xlate_type":{"type":"string"}
      }
    }
  }
}'

Save it and close. Now, open up a terminal window and change to the directory where the template.sh file is located. We need to set the script as executable so we can run it. Run the following commands in a terminal window…

cd Desktop
chmod +x template.sh
./template.sh

You should get back {“acknowledged”:true}

Now back to the tutorial…

Download and Install Java

If you are doing this on a fresh install of Ubuntu (like me), the first thing you’re going to need to do is install Java. Logstash requires at least Java 7 to function so let’s set that up. If you already have Java on your machine, you can skip to the next section. I will be using Java 8 in this example but you can run 7 or openjdk if you wish.

Open a terminal window (ctrl+shift+t) and type…

sudo apt-add-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer

Once you have accepted the license agreement, Java is ready to go.

Download and Install Logstash

Open a terminal window (ctrl+shift+t) and run these commands…

wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add -
echo 'deb http://packages.elasticsearch.org/logstash/1.5/debian stable main' | sudo tee /etc/apt/sources.list.d/logstash.list
sudo apt-get update
sudo apt-get install logstash

Logstash Configuration

Logstash is now installed so now we need to write a configuration file so that we can do things like specify listening port, patterns, the IP of the Elasticsearch server etc.

A Logstash configuration file is basically built of 3 parts: The input (network protocol, listening port, data type etc.), the filter (patterns, grok filters, syslog severity etc.) and the output (IP address of the elasticsearch server logstash is shipping the modified data to etc.).

First and foremost, let’s create a blank logstash configuration file. Open up your terminal window and type…

sudo vi /etc/logstash/conf.d/logstash.conf

or if you don’t know how to use VI

sudo gedit /etc/logstash/conf.d/logstash.conf

I am going to post my logstash configuration file below and then I will explain each part of the file and what it is doing. So, copy my code, paste it into your logstash.conf file and follow along below making changes to your file as you see fit.

logstash.conf

input {
  	udp { 
    		port => 5544
    		type => "cisco-fw"
  	}
}

filter {
	grok {
    		match => ["message", "%{CISCO_TAGGED_SYSLOG} %{GREEDYDATA:cisco_message}"]
  	}

  	# Extract fields from the each of the detailed message types
  	# The patterns provided below are included in core of LogStash 1.4.2.
	grok {
		match => [
  			"cisco_message", "%{CISCOFW106001}",
  			"cisco_message", "%{CISCOFW106006_106007_106010}",
  			"cisco_message", "%{CISCOFW106014}",
  			"cisco_message", "%{CISCOFW106015}",
  			"cisco_message", "%{CISCOFW106021}",
  			"cisco_message", "%{CISCOFW106023}",
  			"cisco_message", "%{CISCOFW106100}",
  			"cisco_message", "%{CISCOFW110002}",
  			"cisco_message", "%{CISCOFW302010}",
  			"cisco_message", "%{CISCOFW302013_302014_302015_302016}",
  			"cisco_message", "%{CISCOFW302020_302021}",
  			"cisco_message", "%{CISCOFW305011}",
  			"cisco_message", "%{CISCOFW313001_313004_313008}",
  			"cisco_message", "%{CISCOFW313005}",
  			"cisco_message", "%{CISCOFW402117}",
  			"cisco_message", "%{CISCOFW402119}",
  			"cisco_message", "%{CISCOFW419001}",
  			"cisco_message", "%{CISCOFW419002}",
  			"cisco_message", "%{CISCOFW500004}",
  			"cisco_message", "%{CISCOFW602303_602304}",
  			"cisco_message", "%{CISCOFW710001_710002_710003_710005_710006}",
  			"cisco_message", "%{CISCOFW713172}",
  			"cisco_message", "%{CISCOFW733100}"
		]
	}

  	# Parse the syslog severity and facility
  	syslog_pri { }

	geoip {
      		add_tag => [ "GeoIP" ]
      		database => "/opt/logstash/databases/GeoLiteCity.dat"
      		source => "src_ip"
    	}

	if [geoip][city_name]      == "" { mutate { remove_field => "[geoip][city_name]" } }
    	if [geoip][continent_code] == "" { mutate { remove_field => "[geoip][continent_code]" } }
    	if [geoip][country_code2]  == "" { mutate { remove_field => "[geoip][country_code2]" } }
    	if [geoip][country_code3]  == "" { mutate { remove_field => "[geoip][country_code3]" } }
    	if [geoip][country_name]   == "" { mutate { remove_field => "[geoip][country_name]" } }
    	if [geoip][latitude]       == "" { mutate { remove_field => "[geoip][latitude]" } }
    	if [geoip][longitude]      == "" { mutate { remove_field => "[geoip][longitude]" } }
    	if [geoip][postal_code]    == "" { mutate { remove_field => "[geoip][postal_code]" } }
    	if [geoip][region_name]    == "" { mutate { remove_field => "[geoip][region_name]" } }
    	if [geoip][time_zone]      == "" { mutate { remove_field => "[geoip][time_zone]" } }

	# Gets the source IP whois information from the GeoIPASNum.dat flat file database
	geoip {
      		add_tag => [ "Whois" ]
      		database => "/opt/logstash/databases/GeoIPASNum.dat"
      		source => "src_ip"
    	}

 	# Parse the date
  	date {
    		match => ["timestamp",
      			"MMM dd HH:mm:ss",
      			"MMM  d HH:mm:ss",
      			"MMM dd yyyy HH:mm:ss",
      			"MMM  d yyyy HH:mm:ss"
    		]
  	}
}

output {
	stdout { 
		codec => json
	}

	elasticsearch {
    		host => "10.0.0.133"
    		flush_size => 1
	}
}

End of logstash.conf

Don’t freak out! I will walk you through my configuration file and explain what each section is doing. Like I said before, a logstash configuration file is made up of 3 parts: the input, the filter and the output. So let’s walk through it, shall we?

Note: All of my examples will be using this as an example ASA syslog message.

<182>May 07 2015 13:26:42: %ASA-6-302014: Teardown TCP connection 48809467 for outside:124.35.68.19/46505 to inside:10.10.10.32/443 duration 0:00:00 bytes 300 TCP FINs

The Input

input {
  udp { 
    port => 5544
    type => "cisco-fw"
  }
}

This tells logstash the protocol (UDP) and what port to listen (5544). You can make it any port you want but you just need to set it in your ASA firewall like so…

or for you CLI people…

logging enable
logging timestamp
logging buffer-size 40960
logging trap informational
logging facility 22
logging host inside 10.0.0.133 17/5544

You can learn more on how to set this up by checking out my ASA syslog tutorial here

The Filter

This is the filter section. This is where most of the work will be done in logstash. The is where you can do things like use grok patterns to split data into fields based off the message and other neat little features for manipulating your data. My filter section has 5 different parts: grok, syslog_pri, geoip, mutate and date. Let’s look at them individually and show what exactly they do.

Grok

filter {
	grok {
    		match => ["message", "%{CISCO_TAGGED_SYSLOG} %{GREEDYDATA:cisco_message}"]
  	...
}

This section is where you use patterns to match various elements of your log messages and extract them into fields. This specific pattern is using the CISCO_TAGGED_SYSLOG definition in the firewalls pattern built into logstash and splitting it into sections.

The CISCO_TAGGED_SYSLOG looks like this…

CISCO_TAGGED_SYSLOG ^<%{POSINT:syslog_pri}>%{CISCOTIMESTAMP:timestamp}( %{SYSLOGHOST:sysloghost})?: %%{CISCOTAG:ciscotag}:

and, based off our example ASA syslog May 07 2015 13:26:42: %ASA-6-302014: Teardown TCP connection 48809467 for outside:124.35.68.19/46505 to inside:10.10.10.32/443 duration 0:00:00 bytes 300 TCP FINs, it pulls out this into these fields…

syslog_pri = 182
timestamp = May 07 2015 13:26:42
sysloghost =
ciscotag = ASA-6-302014

and the rest of the message of Teardown TCP connection 48809467 for outside:124.35.68.19/46505 to inside:10.10.10.32/443 duration 0:00:00 bytes 300 TCP FINs is just is just set as GREEDYDATA:cisco_message, which will be used later in the next grok filter below.

grok {
    match => [
    	"cisco_message", "%{CISCOFW106001}",
    	....
    	"cisco_message", "%{CISCOFW733100}"
    ]
}

Now, using the GREEDYDATA:cisco_message from the previous grok filter, we are going to use the same firewall patterns file built into logstash and match the message type based off the message. The filter goes through all the patterns until it finds a match and then splits the contents of the message into fields. So, based off our GREEDYDATA:cisco_message of Teardown TCP connection 48809467 for outside:124.35.68.19/46505 to inside:10.10.10.32/443 duration 0:00:00 bytes 300 TCP FINs from our example syslog, it will try to match it against the patterns file. Our specific example matches the CISCOFW302013_302014_302015_302016 line in the patterns file, which reads…

CISCOFW302013_302014_302015_302016 %{CISCO_ACTION:action}(?: %{CISCO_DIRECTION:direction})? %{WORD:protocol} connection %{INT:connection_id} for %{DATA:src_interface}:%{IP:src_ip}/%{INT:src_port}( \(%{IP:src_mapped_ip}/%{INT:src_mapped_port}\))?(\(%{DATA:src_fwuser}\))? to %{DATA:dst_interface}:%{IP:dst_ip}/%{INT:dst_port}( \(%{IP:dst_mapped_ip}/%{INT:dst_mapped_port}\))?(\(%{DATA:dst_fwuser}\))?( duration %{TIME:duration} bytes %{INT:bytes})?(?: %{CISCO_REASON:reason})?( \(%{DATA:user}\))?

…and puts the data into these fields…

action = Teardown
protocol = TCP
connection_id = 48809467
src_interface = outside
src_ip = 124.35.68.19
src_port = 46505
dst_interface = inside
dst_ip = 10.10.10.32
dst_port = 443
duration = 0 =00 =00
bytes = 300
reason = TCP FINs

Syslog_pri

syslog_pri { }

This section takes the POSINT syslog_pri from the first grok filter and gets the facility and severity level of the syslog message. Our example had a syslog_pri number of 182 and logstash can determine that the message is an informational based message from the local6 facility. I was able to get that by referencing this chart and finding which column and row 182 fell under.

             emergency   alert   critical   error   warning   notice   info   debug
 kernel              0       1          2       3         4        5      6       7
 user                8       9         10      11        12       13     14      15
 mail               16      17         18      19        20       21     22      23
 system             24      25         26      27        28       29     30      31
 security           32      33         34      35        36       37     38      39
 syslog             40      41         42      43        44       45     46      47
 lpd                48      49         50      51        52       53     54      55
 nntp               56      57         58      59        60       61     62      63
 uucp               64      65         66      67        68       69     70      71
 time               72      73         74      75        76       77     78      79
 security           80      81         82      83        84       85     86      87
 ftpd               88      89         90      91        92       93     94      95
 ntpd               96      97         98      99       100      101    102     103
 logaudit          104     105        106     107       108      109    110     111
 logalert          112     113        114     115       116      117    118     119
 clock             120     121        122     123       124      125    126     127
 local0            128     129        130     131       132      133    134     135
 local1            136     137        138     139       140      141    142     143
 local2            144     145        146     147       148      149    150     151
 local3            152     153        154     155       156      157    158     159
 local4            160     161        162     163       164      165    166     167
 local5            168     169        170     171       172      173    174     175
 local6            176     177        178     179       180      181    182     183
 local7            184     185        186     187       188      189    190     191

Pretty cool stuff.

OPTIONAL: GeoIP

geoip {
    	add_tag => [ "GeoIP" ]
    	database => "/opt/logstash/databases/GeoLiteCity.dat"
    	source => "src_ip"
}

This part is completely optional and up to you if you want to set it up. What it does is it uses the source ip address in the syslog and it gets location data based of a flat file database. This flat file database will be used by logstash to get the location of the IP addresses hitting the firewall so you can get turn the source IP address of 124.35.68.19 into this…

"geoip": {
      "ip": "124.35.68.19",
      "country_code2": "US",
      "country_code3": "USA",
      "country_name": "United States",
      "continent_code": "NA",
      "region_name": "NJ",
      "city_name": "Edison",
      "postal_code": "08820",
      "latitude": 40.57669999999999,
      "longitude": -74.3674,
      "dma_code": 501,
      "area_code": 732,
      "timezone": "America/New_York",
      "real_region_name": "New Jersey",
      "location": [
        -74.3674,
        40.57669999999999
      ]
}

If this doesn’t interest you, remove the geoip section from the config file and skip to the next section. If this does interest you, follow the steps below.

Open a terminal window and type…

cd ~
wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
sudo mkdir /opt/logstash/databases
gunzip GeoLiteCity.dat.gz
sudo mv ~/GeoLiteCity.dat /opt/logstash/databases/

and that is it for the first GeoIP tag for location data.

This is another optional GeoIP filter…

geoip {
    	add_tag => [ "Whois" ]
    	database => "/opt/logstash/databases/GeoIPASNum.dat"
    	source => "src_ip"
}

Just like the GeoIP location database, the GeoIP ASN database uses the source IP address, but instead of returning location information, it returns the ASN information. So essentially it turns 124.35.68.19 into…

"number": "AS6128",
"asn": "Cablevision Systems Corp."

So you can see ISP names in your elasticsearch searches. Also very cool and highly recommended for your setup. If this does not interest you for your setup, remove the geoip section from your configuration file and skip to the next section. If this does interest you, follow the steps below…

Open a terminal window and type…

cd ~
wget http://download.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz
gunzip GeoIPASNum.dat.gz
sudo mv ~/GeoIPASNum.dat /opt/logstash/databases/

and that’s it with the GeoIP section.
Mutate

if [geoip][city_name]      == "" { mutate { remove_field => "[geoip][city_name]" } }

This part is fairly straight forward. Basically what is happening is that if the entry in the GeoIPASNum.dat file for a particular IP address has a country but does not have a city, it will remove that field before it is inserted into elasticsearch. That is basically it with the mutate section of my config file.

Date

date {
    match => ["timestamp",
    	"MMM dd HH:mm:ss",
    	"MMM  d HH:mm:ss",
    	"MMM dd yyyy HH:mm:ss",
    	"MMM  d yyyy HH:mm:ss"
    ]
}

This part is also pretty straight forward. It takes the timestamp value from the first grok filter and sets it as the timestamp when putting it into elasticsearch. So basically, instead of the timestamp being set as the time when logstash received the message, the timestamp is set as when the event was triggered on the firewall based off the firewalls clock and hopefully the firewall is configured to use NTP so that all your devices clocks are synchronized.

Output

output {
	stdout { 
		codec => json
	}

	elasticsearch {
		host => "10.0.0.133"
		flush_size => 1
	}
}

This section is how the final result is displayed/sent to various things. My output section has 2 parts: stdout and elasticsearch.

Stdout

stdout { 
	codec => json
}

Stdout is optional but I have it in there so I can see if everything is working properly through the terminal window like so

Obviously you do not need this part to run but I like to have it in there for debugging purposes.

Elasticsearch

elasticsearch {
    	host => "10.0.0.133"
    	flush_size => 1
}

Elasticsearch is where you specify the IP address of your elasticsearch server and that is pretty much it.

So now that you have your logstash.conf file set up, you can now run logstash for your ASA firewall. Save your config file and type in this into a terminal window…

/opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf

That’s it, you’re finished.

Note: If you have not turned on syslog messages on your ASA firewall, read my other blog post here.

Install Kibana 4 and Elasticsearch on Ubuntu

Blog, ElasticSearch, Information Technology, Kibana, Software

Great news, everyone! Kibana 4 is no longer in beta and has been released to the masses.

In this guide, I will show you how to get started installing Kibana 4 and Elasticsearch in a linux environment. I will be using Ubuntu Server 14.04 in my examples.

For those who do not know, Kibana is an interface to data exploration through graphical visualizations. Kibana 4 is the latest release from the people over at Elastic. Kibana 4 offers a much better interface for sorting through your data.

Download and Install Java

If you are doing this on a fresh install of Ubuntu (like me), the first thing you’re going to need to do is install Java. Elasticsearch requires at least Java 7 to run so let’s set that up. I will be using Java 8 in this example but you can run 7 or openjdk if you wish.

Open a terminal window (ctrl+shift+t) and type…

sudo apt-add-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer

Once you have accepted the license agreement, Java is ready to go.

Download and Install Elasticsearch

Open another terminal window (ctrl+alt+t) and type…

cd ~
wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add -
echo 'deb http://packages.elasticsearch.org/elasticsearch/1.5/debian stable main' | sudo tee /etc/apt/sources.list.d/elasticsearch.list
sudo apt-get update
sudo apt-get install elasticsearch

That commands above will install Elasticsearch on your machine.

The next thing we need to do is to give our Elasticsearch cluster a name. To do this, type in this commands…

sudo sed -i -e 's|#cluster.name: elasticsearch|cluster.name: kibana|' /etc/elasticsearch/elasticsearch.yml

Now you need to tell your machine to run Elasticsearch every time the machine boots up. To do that, type…

sudo update-rc.d elasticsearch defaults 95 10
sudo /etc/init.d/elasticsearch start

And now you have Elasticsearch running on your machine. You can check by typing “sudo service elasticsearch status” in the terminal window, or by opening up a browser and going to http://127.0.0.1:9200 and seeing a result similar to this…

{
  "status" : 200,
  "name" : "Karen Page",
  "cluster_name" : "kibana",
  "version" : {
    "number" : "1.5.2",
    "build_hash" : "62ff9868b4c8a0c45860bebb259e21980778ab1c",
    "build_timestamp" : "2015-04-27T09:21:06Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.4"
  },
  "tagline" : "You Know, for Search"
}

Download and Install Kibana 4

Now that Elasticsearch is installed on our machine, we can now go ahead and setup Kibana 4. Unlike previous versions of Kibana, Kibana 4 does not use apache to run.

Open up another terminal window (ctrl+shift+t) and type in these commands…

cd ~
wget https://download.elastic.co/kibana/kibana/kibana-4.1.0-linux-x64.tar.gz
tar xvf kibana-4.1.0-linux-x64.tar.gz
sudo mv ~/kibana-4.1.0-linux-x64 /opt/kibana

Now we need to have Kibana 4 start up when the machine boots so we need to have it run as a service. To do that, type in these commands in a terminal window…

sudo wget --output-document="/etc/init.d/kibana4" https://raw.githubusercontent.com/akabdog/scripts/master/kibana4_init
sudo chmod +x /etc/init.d/kibana4
sudo update-rc.d kibana4 defaults 96 9
sudo service kibana4 start

You can check to see if Kibana is working properly by goint to http://127.0.0.1:5601/ in your browser of choice. And that is in in terms of installing Kibana 4 and Elasticsearch. Part 2 of this blog post on setting up Logstash for your Cisco ASA syslogs is over here.

Create a Custom Elasticsearch Template

Blog, ElasticSearch, Information Technology, Networking, Software

This post will show you how to create a custom index template for Elasticsearch.

Why would you need to create a custom template Lets say you are storing ASN data in your Elasticsearch index.
For example…

Google Inc. – 15 messages
Facebook Inc. – 25 messages
Linkedin Inc. – 33 messages

When you query the index for ASN fields, you are going to get 15 hits for Google, 25 hits for Facebook, 33 hits for Linkedin and 73 hits for Inc. This is because, by default, ElasticSearch does automatic index creation which analyzes each field and splits strings at spaces when indexing. So what if you want to just have the whole field item return as a result so that something like “Google Inc” will show up as 15 hits? Well, you have to create an ElasticSearch index template. Note: creating a template will not magically modify old indexes, that data has already been indexed. The template will only work for newly created indices in ElasticSearch after you add the template.

First thing you need to do is figure out the naming scheme for your indices. Knowing the name pattern for new indices will make it so that the template you are about to create only applies to that index and not other indices in ElasticSearch. I use logstash to ship everything to ElasticSearch and the default index naming pattern is logstash-YYYY-MM-DD so, iny my template, I will have logstash* with the asterisk acting as a wildcard. If you’re not using logstash and are unsure of the naming, go to /var/lib/elasticsearch and look in the indices folder to see the names of your current indices. Remember this for when we create the template.

Next you want to find the name inside of a current index so the template will only match the types you want it to match. Open your browser and type http://localhost:9200/_all/_mapping?pretty=1 in the URL bar and hit enter.

For example: I see

"logstash-2014.09.30": {
        "cisco-fw": {
            "properties": {

Because I have my logstash config file set to specify anything coming in on a certain port as type cisco-fw and because of that, the type in ElasticSearch is cisco-fw but yours might be default or something else. Remember this for when we create the template

Next thing is open up a notepad so we can start creating the template.

Here is the template for your template

curl -XPUT http://localhost:9200/_template/logstash_per_index -d '
{
    "template" : "logstash*",
    "mappings" : {
      "cisco-fw" : {
      }
    }
}'

Where logstash_per_index is the name you want to give the template, logstash* is the index naming scheme and cisco-fw is the type.

Now, under properties, you are going to set the field type and options based on field name. For my example, I am doing ASN values, so, under properties, I would write

"asn":{"type":"string", "index":"not_analyzed"}

The ASN type is a string (obviously) and index is set to not_analyzed. Not_analyzed means that it is still searchable with a query, but it does not go through any analysis process and is not broken down into tokens. This will allow us to see Google Inc as one result when querying ElasticSearch. Do this for all the fields in your index. For example, here is my completed template…

#!/bin/sh
curl -XPUT http://localhost:9200/_template/logstash_per_index -d '
{
    "template" : "logstash*",
    "mappings" : {
      "cisco-fw" : {
         "properties": {
            "@timestamp":{"type":"date","format":"dateOptionalTime"},
            "@version":{"type":"string", "index" : "not_analyzed"},
            "action":{"type":"string"},
            "bytes":{"type":"long"},
            "cisco_message":{"type":"string"},
            "ciscotag":{"type":"string", "index" : "not_analyzed"},
            "connection_count":{"type":"long"},
            "connection_count_max":{"type":"long"},
            "connection_id":{"type":"string"},
            "direction":{"type":"string"},
            "dst_interface":{"type":"string"},
            "dst_ip":{"type":"string"},
            "dst_mapped_ip":{"type":"ip"},
            "dst_mapped_port":{"type":"long"},
            "dst_port":{"type":"long"},
            "duration":{"type":"string"},
            "err_dst_interface":{"type":"string"},
            "err_dst_ip":{"type":"ip"},
            "err_icmp_code":{"type":"string"},
            "err_icmp_type":{"type":"string"},
            "err_protocol":{"type":"string"},
            "err_src_interface":{"type":"string"},
            "err_src_ip":{"type":"ip"},
            "geoip":{
               "properties":{
                  "area_code":{"type":"long"},
                  "asn":{"type":"string", "index":"not_analyzed"},
                  "city_name":{"type":"string", "index":"not_analyzed"},
                  "continent_code":{"type":"string"},
                  "country_code2":{"type":"string"},
                  "country_code3":{"type":"string"},
                  "country_name":{"type":"string", "index":"not_analyzed"},
                  "dma_code":{"type":"long"},
                  "ip":{"type":"ip"},
                  "latitude":{"type":"double"},
                  "location":{"type":"geo_point"},
                  "longitude":{"type":"double"},
                  "number":{"type":"string"},
                  "postal_code":{"type":"string"},
                  "real_region_name":{"type":"string", "index":"not_analyzed"},
                  "region_name":{"type":"string", "index":"not_analyzed"},
                  "timezone":{"type":"string"}
               }
            },
            "group":{"type":"string"},
            "hashcode1": {"type": "string"},
            "hashcode2": {"type": "string"},
            "host":{"type":"string"},
            "icmp_code":{"type":"string"},
            "icmp_code_xlated":{"type":"string"},
            "icmp_seq_num":{"type":"string"},
            "icmp_type":{"type":"string"},
            "interface":{"type":"string"},
            "is_local_natted":{"type":"string"},
            "is_remote_natted":{"type":"string"},
            "message":{"type":"string"},
            "orig_dst_ip":{"type":"ip"},
            "orig_dst_port":{"type":"long"},
            "orig_protocol":{"type":"string"},
            "orig_src_ip":{"type":"ip"},
            "orig_src_port":{"type":"long"},
            "policy_id":{"type":"string"},
            "protocol":{"type":"string"},
            "reason":{"type":"string"},
            "seq_num":{"type":"long"},
            "spi":{"type":"string"},
            "src_interface":{"type":"string"},
            "src_ip":{"type":"string"},
            "src_mapped_ip":{"type":"ip"},
            "src_mapped_port":{"type":"long"},
            "src_port":{"type":"long"},
            "src_xlated_interface":{"type":"string"},
            "src_xlated_ip":{"type":"ip"},
            "syslog_facility":{"type":"string"},
            "syslog_facility_code":{"type":"long"},
            "syslog_pri":{"type":"string"},
            "syslog_severity":{"type":"string"},
            "syslog_severity_code":{"type":"long"},
            "tags":{"type":"string"},
            "tcp_flags":{"type":"string"},
            "timestamp":{"type":"string"},
            "tunnel_type":{"type":"string"},
            "type":{"type":"string"},
            "user":{"type":"string"},
            "xlate_type":{"type":"string"}
      }
    }
  }
}'

Once you change all the types you need, it is now time to add the template to ElasticSearch. You can either save you file in notepad, make it a script and run that through a terminal window or you can copy the text from notepad and enter into a terminal window. You should see a {“ok”:true,”acknowledged”:true} response if everything was formatted properly.
ACK

And that is it. You will only see the fruits of your labor when a new index is created and matches the parameters set in your template file (i.e logstash*). Because of my naming scheme with Logstash, new indices are only created at the start of the day (logstash-YYYY-MM-DD) so I had to wait until the next day to see if my template was working properly. If you are impatient, cannot wait to see if it worked or not and don’t care about losing data in your current index then you can delete it from ElasticSearch by issuing the following CURL command in a terminal window

curl -XDELETE localhost:9200/index_name

where index_name is the name of your index (ex. logstash-2014-12-11)

Helpful tip: if you start seeing data not show up in the index, it is very possible that you may have messed up one of the field types in you template file. I am writing this because I ran into this issue and could not figure out why there was no data in my index. To solve this, go to /var/log/elasticsearch and see the log file for the date where data was not properly going into the ElasticSearch index (it should be a lot bigger in file size compared to the other log files). In the log file, I was seeing this error multiple times

org.elasticsearch.index.mapper.MapperParsingException: failed to parse [protocol]

What happened was that, thinking protocol meant port number protocol (ex. 25, 80, 443 etc), I set the protocol field as type LONG. To my surprise, protocol was either TCP or UDP so it should have been set as type string. ElasticSearch was expecting a long to index based off my template but instead was getting strings so the application freaked out. Instead of modifying the template file on the server, I decided to delete it from ElasticSearch, make my changes to the protocol field and then re-upload the template back to ElasticSearch. To do that, I opened a terminal and typed

curl -XDELETE http://localhost:9200/_template/logstash_per_index

where logstash_per_index is the name of the template. That command will delete the template off of your server. Make your changes to your template in notepad and then add the template back to ElasticSearch.

Since the template only applies to newly created indices and your index did not have any data inside of it because of the incorrect template, you can go ahead and just delete that index and create a new one that will work with the newly modified template.

curl -XDELETE localhost:9200/index_name

where index_name is the name of your index (ex. logstash-2014-12-11).

And that is it. Leave a comment down below if you found this information helpful or if you have any questions for me. Good luck!

Enable Remote Desktop on a Remote Windows Server

Blog, Information Technology, Microsoft, Servers, Software, Windows

Scenario: you just finished installing some brand new servers in your server racks at the data center. You plugged them into your network, updated them with the latest windows updates and figured that you can finish the rest of the configuration while sitting at your desk at the office. Unfortunately you forgot to enable remote desktop and there is no way to get into the new machines.

Instead of driving all the way back to the data center, you can use these steps to enable remote desktop… remotely.

Steps

First you need to install PsExec from the Microsoft website http://technet.microsoft.com/en-us/sysinternals/bb897553

Extract the zip file, open a command window and cd to the newly extracted directory. (Ex. I extracted to a folder on my desktop called PSTools. Start>Run>Cmd>”cd C:\Users\Developer\Desktop\PSTools”)

Next you need to run the command listed below. Mofidy the command by changing remoteserver to the IP of the remote server.

psexec \\remoteserver reg add "hklm\system\currentcontrolset\control\terminal server" /f /v fDenyTSConnections /t REG_DWORD /d 0

This sets the registry value on the remote machine to enable and disable remote desktop connections.

Now we need to add the firewall exceptions to allow us remote access. Again, modify the commands below by changing remoteserver to the IP of the machine.

psexec \\remoteserver netsh firewall set service remoteadmin enable
psexec \\remoteserver netsh firewall set service remotedesktop enable

After running those two commands, you should be able to remote desktop into that machine.

Jack's Website

Whatever is on my mind

Software