CRITs: Collaborative Research Into Threats
The project has been in development for the last 3.5 years and available for the last 2.5 under a limited-release license. I am amazingly proud at how far the project has come. With 100’s of organizations across government, private, and public sectors using the platform already, we’ve seen a drastic improvement in how organizations are able to research threats, share threat data, and implement protective measures across their networks. We decided to open source CRITs and give back to the security community because it is important to have freely available tools in an industry where every organization, big or small, needs to do all they can to protect themselves against threats. Being able to communicate and share threats between organizations large and small, government and public, opens up the industry to a more collaborative effort towards intelligence-based active threat defense. I wanted to give some background about how CRITs came to be. I’ve been there since the beginning as the lead developer and I continue to stay in that role in my personal time while taking on the role of Project Manager.
Towards the end of 2010 I was asked to tackle a code-rewrite of a small project at work. The project was a malware database written in Python using Django and MySQL. It was growing long in the tooth and was in dire need of a more scalable design. Around this same time I was becoming frustrated with what I will call the Data Rediscovery Problem.
Most of us while we work tend to put data where it is most convenient at the time. If we’re working on a malicious binary we’ll drop it onto a host with the tools necessary to handle it. If we’re writing quick scripts we’ll develop them on whichever box winds up being the first terminal we find available to us. This is all to save time and effort to get the job done quickly and easily. Even worse, what do we name those scripts? Usually things like foo.py, foo2.py, etc. What happens in a few weeks when we find ourselves in need of that data or those scripts again? We have to remember where we put them. A lot of times we’ll eventually rediscover them. Other times we’ll forget and wind up re-generating the data or re-writing the script. Even worse is when someone else is duplicating the work we did because our stuff wasn’t readily available. This is a huge waste of time and resources. It also makes it frustratingly difficult for new hires and teammates to find data they need to do their job.
While I was rewriting the malware database, I thought it would be neat if it could be extended to solve the Data Rediscovery issue. I wanted to provide a way for people to quickly upload content and make it as efficient as possible to save time and be a reference point for all of their data. That would solve half of the rediscovery issue, which is a good start.
The first thing I did was extend past storing Samples (malicious binaries) and allow for the storage of PCAPs. Network traffic is such a huge help when trying to dig towards “ground truth”. I always had problems finding the network traffic I needed so adding PCAP support was an easy decision. I was approached by an analyst and asked if I could extend the project to support Emails. Seemed like a reasonable request so I added Email support. Tracking Domains and IPs seemed like the next logical step given their presence in binaries, PCAPs, and emails, so those were added.
Then I was asked about supporting IoCs (Indicators of Compromise). This was sort of a game-changer. IoCs are amazingly important for an organization looking to defend its network. They need to be handled carefully. Even though they might be simple in concept, the response to them can sometimes be extremely complex. We spent some time thinking about what handling IoCs in the project meant but eventually settled in on a design and got it implemented.
At this time it became obvious we needed a name for this project. It started out being a critical tool for our Malware Analysts but was now being used heavily by our Intel Analysts as well as our Incident Response team. They all began collaborating in a way that they were never able to before. Sharing data was as simple as uploading it to the same place everyone else was. This made analysis and response a much more efficient and effective process. It moved the groups to a more intelligence-based active threat defense. With all of us researching these threats, I quickly settled on the name Collaborative Research Into Threats.
Over the last three years, the project has grown support for Campaigns, Certificates, Events, Raw Data, and Targets (with more in the works!). Through word-of-mouth hundreds of organizations asked to get their hands on CRITs so they can make use of it to improve their security posture. Many developers have jumped on board and contributed hundreds of features for managing and working with threat data. One of the biggest additions was the Services Framework. This allowed people to develop extensions to the project and enhance functionality without the need to modify core code. To date we have 25 services that have been contributed back to the project giving Intel and Malware analysts a way to work with threat data and not have to download it, integrate CRITs with third-party and homegrown software, as well as visualize and discover threat data in ways that were previously unavailable. This solved the second half of the Data Rediscovery Issue. Now someone can write a tool once, create a service out of it, and make it available for all of their users and the entire community if they wish. The ability to work with threat data without always needing to download it to another machine has proven time and time again to be invaluable.
Another major enhancement to the project was the inclusion of support for Structured Data Exchange Formats (CybOX, STIX, and TAXII). Being able to collaborate with groups internally is a huge win. But being able to collaborate with other trusted organizations in real time changes the security industry completely. Where organizations used to keep this information to themselves, they are now able to use these exchange formats to communicate threat data and position other organizations to protect their network well in advance of the threat making its way to them. By incorporating these formats into CRITs we’ve given all of the users of the platform the capability to share threat data quickly when time matters the most.
Like any other tool in the industry, CRITs isn’t the end-all and be-all solution to defending your network. It is simply an enabler. It gives analysts a better way of storing, enriching, and discovering threat data necessary to protect their organization. It doesn’t replace analysts. If anything it speaks to the necessity for all organizations to have smart, capable analysts who can leverage a platform like CRITs to put good data in, and get good data out. Too long has our industry worked in isolation to solve the same problem. It’s time we all work together and change our mindsets to a more open, intelligence-based active threat defense.