Let’s say you have a bunch of domains along with their registration date in a text file that looks like this:
2017-08-01,01.dbqsktxy.ru
2017-07-23,1.dbqsktxy.ru
2017-07-23,2.dbqsktxy.ru
2017-07-29,adfbfjhy.homefirstelement.ru
2017-06-23,aizfh.xn--80ahia7byai4bu9bb.xn--p1ai
2017-06-18,bestcaredeal.su
2017-07-22,bestnaturalshop.ru
2017-07-25,bestonlinesale.su
2017-07-22,bestonlinevalue.su
etc...and you want a way to view that information in a way that quickly makes sense.
The following technique will explain how to put it all on a timeline in the web browser with some Python, Javascript and HTML.
The end results are informative, but if your list of domains and dates are too long, it can take a while to load in a browser and may require zooming out to view everything. Regardless, it’s a fun project and might give some kind of insight into a list of domains when combined with other information (as we’ll see at the end of this).
The Javascript library we’ll be using is called vis-timeline. It’s part of a larger library called visjs.
The timeline we’re going to create has a default look like this (image taken from here):
When using a javascript library, you can call out to the .js file on the internet, but I like to download the script locally. I’ll post a link to download the needed files with the script at the end.
Let’s get started:
Declare your input and output files:
inputfile = 'domains_and_reg.txt'
outputfile = 'domains_and_reg.html'This next part is a function I use in a lot of scripts to write/append to a file:
def write_append(filename, line):
writefile = open(filename,'a')
writefile.write(line)
writefile.write('\n')
writefile.close()For this script, I’m hard-coding the HTML that will be written to the output file:
htmltop = """
<!DOCTYPE HTML>
<html>
<head>
<title>Timeline basic demo</title>
<script src="vis.min.js"></script>
<link href="vis.min.css" rel="stylesheet" type="text/css" />
<style type="text/css">
body, html {
font-family: sans-serif;
}
</style>
</head>
<body>
<div id="visualization"></div>
<script type="text/javascript">
var container = document.getElementById('visualization');
var data = ["""
htmlbottom = """];
var options = {};
var timeline = new vis.Timeline(container, data, options);
</script>
</body>
</html>"""Our data from the domain and date list needs to look like this for vis.js:
{id: 1, content: 'item 1', start: '2013-04-20'}So we’ll create a list called ‘data’ that will hold everything after it’s processed, open the input file, convert each line into a list that contains two items (the date and domain) and then we’ll put that into a dictionary item (while incrementing the id field) which will then be converted to a string.
data = []
idnumber = 0
with open(inputfile,'r') as f:
for line in f.read().split('\n'):
l = line.split(',')
try:
dt = l[0]
domain = l[1]
except:
continue
idnumber +=1
dataline = {'id':idnumber, 'content':domain,'start':dt}
dataline = str(dataline)After converting to a string, there are some extra characters (' in the strings). I have to go about removing those with:
cleaneddataline = dataline.replace("'content'","content").replace("'start'","start").replace("'id'","id")Then I can add the line to the big list that contains all the lines with:
data.append(cleaneddataline)And for the end of the script, we open the output file for writing, write the top part of the HTML, write each line as it needs to be for vis.js to understand it, and then close with the bottom part of the HTML.
writefile = open(outputfile,'w')
write_append(outputfile,htmltop)
for i in data:
write_append(outputfile,i + ",")
write_append(outputfile,htmlbottom)Wherever the script was run, we end up with a file called ‘domains_and_reg.html’.
Opening it, we can see the domains placed on a timeline by the date they were registered:
If you want to view the actual file that was created from a small list of domains, you can do so here, but keep in mind you’ll probably have to zoom out in your browser to see the full results like in the screenshot above.
Some other ideas:
1: Put malware hashes on a timeline, so if you have something that was seen on a date and you want to know what that looks like on a timeline, you have dates and a malware hash in your file, with each line like this:
2017-10-29 11:04:58, be5bee2088a8d46f74d787ca59abbe9ade56f9bbad11b6e34f77ff219ea8fe8d(that’s a super old Locky Ransomware sample)
You’d have to adjust the date/time data to fit the right format.
2: Another cool thing I’ve done is to take a phishing email and the domains in the email and put the following on a timeline:
- Date the email was received
- Date of domain registrations.
That looks like this:
It’s a bit hard to see all zoomed out, but here’s what I got out of it:
The data for this one is from multiple Emotet phishing emails that were linking to a malicious URL. They had some other URL’s providing supporting files to the emails as well. I put the dates the emails were delivered along with the word ’email’ (so it looked like ‘date,email’ in the input file) and the first time the domains were seen via DNS data from a third party that has access to a lot of DNS data (Cisco Umbrella…where I work).
Looking at the timeline, I can quickly notice that the URL it links to was first seen just before the date the emails came in. The other URLs were first seen many months before the phishing emails. While I didn’t put the registration date, I was still able to assume from this timeline that the domain in the active phishing URL was compromised while the other ones were probably set up by bad actors a while ago.
The full script is below.
Download the javascript and css files you’ll need here: vis.min.js, and vis.min.css
inputfile = 'domains_and_reg.txt'
outputfile = 'domains_and_reg.html'
def write_append(filename, line):
writefile = open(filename,'a')
writefile.write(line)
writefile.write('\n')
writefile.close()
htmltop = """
<!DOCTYPE HTML>
<html>
<head>
<title>Timeline basic demo</title>
<script src="vis.min.js"></script>
<link href="vis.min.css" rel="stylesheet" type="text/css" />
<style type="text/css">
body, html {
font-family: sans-serif;
}
</style>
</head>
<body>
<div id="visualization"></div>
<script type="text/javascript">
var container = document.getElementById('visualization');
var data = ["""
htmlbottom = """];
var options = {};
var timeline = new vis.Timeline(container, data, options);
</script>
</body>
</html>"""
data = []
idnumber = 0
with open(inputfile,'r') as f:
for line in f.read().split('\n'):
l = line.split(',')
try:
dt = l[0]
domain = l[1]
except:
continue
idnumber +=1
dataline = {'id':idnumber, 'content':domain,'start':dt} # data must look like this: {id: 1, content: 'item 1', start: '2013-04-20'},
dataline = str(dataline)
cleaneddataline = dataline.replace("'content'","content").replace("'start'","start").replace("'id'","id")
data.append(cleaneddataline)
writefile = open(outputfile,'w')
write_append(outputfile,htmltop)
for i in data:
write_append(outputfile,i + ",")
write_append(outputfile,htmlbottom)