Current topics in automatic incident handling for CERTs IFAS HKCERT , IFAS and use-cases IHAP project ContactDB project Current R&D
Information Feed Analysis System Information Feed Analysis System
National CSIRTs need visibility on network in their economy National CSIRTs need visibility on network in their economy However, many national CSIRTs don’t operate networks themselves, and normally don’t have global (or any) direct visibility How does the CSIRT know what’s going on in their country?
Luckily, there are a lot of network operators, research teams, vendors, and other CSIRTs out there that collect information, and will share it with national CSIRTs. Luckily, there are a lot of network operators, research teams, vendors, and other CSIRTs out there that collect information, and will share it with national CSIRTs. And here comes the “but”...
There are many feeds, all with their own data formats and mediums: There are many feeds, all with their own data formats and mediums: - Formats: CSV, JSON, XML, STIX, IODEF
- Mediums: HTML, RSS, email, HTTP APIs
While there are efforts to standardise data formats, this will take a long time, and will likely never cover 100% of feeds We can’t change the format of remote feeds - we can only change what we do with the data.
Different feeds use many terms to mean the same thing: Different feeds use many terms to mean the same thing: - ip, source_ip, src_ip, endpoint, attacker_ip, cnc_ip...
If we receive events from many feeds, we need to normalise so we can compare them together.
As a national CSIRT, we’re concerned with the health of national networks: which means measurement. As a national CSIRT, we’re concerned with the health of national networks: which means measurement. We can only measure longterm if we store events, enabling us to analyse them. We also want to search through events, like: - C&C servers in domestic networks in last week
- Bots infected with Trojan.abc on BigISP
- Defaced web sites targeting gov.zz
There’s way too much network event data out there to manually process There’s way too much network event data out there to manually process Options: - a) use lots of analyst time doing tedious log processing
- b) write lots of small, independent scripts
- c) ignore inbound logs completely
- d) use an automated processing system
We need something which automatically: We need something which automatically: - Gathers many different types of feeds
- Normalises the data in those feeds
- Stores that data somewhere
- Allows search and performs statistical analysis
IFAS = Information Feed Analysis System IFAS = Information Feed Analysis System Project sponsored by HKCERT and developed by HKCERT and CSIRT Foundry An integration of open source tools, released as open source for CSIRTs
Abusehelper: gather, process, and enrich feeds, generate events Abusehelper: gather, process, and enrich feeds, generate events Logstash: process and normalise feeds Elasticsearch: store events in schema-free index server Kibana: search through events IFAS Reporter: get overall statistics, build realtime dashboards
Open source under Apache 2.0 License Open source under Apache 2.0 License Only possible with the hard work released under open source licenses from Abusehelper and Elasticsearch teams Contributions, bug reports, feature requests most welcome!
Production: 8-16GB memory machine Production: 8-16GB memory machine Dev: 4GB possible Multi-core machine (4+ ideal) Runs in a VM no problem
Currently under closed pilot to trusted CSIRTs Currently under closed pilot to trusted CSIRTs - Eventually public release
Please contact contact@ifas.io for details
IFAS = Information Feed Analysis System IFAS = Information Feed Analysis System Project sponsored by HKCERT and developed by HKCERT and CSIRT Foundry An integration of open source tools, released as open source for CSIRTs
Abusehelper: gather, process, and enrich feeds, generate events Abusehelper: gather, process, and enrich feeds, generate events Logstash: process and normalise feeds Elasticsearch: store events in schema-free index server Kibana: search through events IFAS Reporter: get overall statistics, build realtime dashboards; charts based on D3.js
Collaborate of global intelligences Collaborate of global intelligences - No sensor installation required
- Assimilate different input formats / standards
- Normalize field names
- Collect and normalize information 24x7
Situational Awareness - Alerts
- Public Awareness
- Tracking Trends
Actionable Actionable Engagement Tool - Engage top ISPs to cooperate
- Cooperate on large scale clean up (bots)
Standardization - Standardize as measurement
Total governance on data Total governance on data Built on open source tools that has good community - Abusehelper, Elasticsearch, D3.js
Low hurdle to use and customize - Open source under Apache 2.0 License
- Free of charge
- UTF Support
- Support JSON/CSV import & export
- Modular
- add plug-ins for new feeds
- add output chart plug-ins using D3.js
Machine with 8GB+ & Multi-core CPU Machine with 8GB+ & Multi-core CPU Ubuntu Server (can run in VM) Bitbucket Account (code repository) 30 minutes of installation and configuration
Powerful search on all the information collected Powerful search on all the information collected
Statistical analysis-Trends & Distributions Statistical analysis-Trends & Distributions
Set tracking criteria – get notify ASAP Set tracking criteria – get notify ASAP - domain: *.gov.hk
- Alert lists : educational institutions (hkedu), NGOs (hkorg)
Correlate Cryptolocker 2013-Oct with Zeus Correlate Cryptolocker 2013-Oct with Zeus
Data do help HKCERT engaging ISPs (their sales team) Data do help HKCERT engaging ISPs (their sales team) Data do help a server hosting SP understand their customers’ security problems
Defacement Defacement Phishing Export to CSV for batch processing, with some other scripts Malware hosting – a bit difficult Large volume of incidents – need prioritisation
All you can use All you can use All you can contribute - Add input filters for new feeds
- Add new plug-in modules
- Add new chart and visualization
- Integrate with other systems, e.g. RTIR
- …
Standard language: STIX, taxonomy of ENISA
An ongoing project that turn security events into Actionable Data - Set Priority, Choose Monitors, Consolidate Results
Incident handling automation project Incident handling automation project
Very similar to IFAS, developed in parallel by CERT.pt, CERT.at Very similar to IFAS, developed in parallel by CERT.pt, CERT.at Also uses Logstash, Elastic Search and Abusehelper Less work on the Webinterface, more work on Ontology, „Data harmonisation document“
Discussions about CERT.AT developments/documents Discussions about CERT.AT developments/documents Discussions about cooperation between CERTs ENISA support
Open Source Open Source Maintainable Flexible and Modular - must be possible to integrate existing software and modules (Pastemon, AbuseHelper, etc..) Reusable Easily Extendable - should require little knowledge and basic programming skills Easily Deployable Easily Updatable – easy to share new developments with other CERTs and update the system with that new code Easily Configurable - config files that can be easily modified to fit CERT‘s needs Documented - must be well documented
http://www.enisa.europa.eu/activities/cert/support/incident-handling-automation
https://bitbucket.org/clarifiednetworks/abusehelper/wiki/Data%20Harmonization%20Ontology https://bitbucket.org/clarifiednetworks/abusehelper/wiki/Data%20Harmonization%20Ontology A standard set of well defined field names within Abusehelper (AH) Allows CERTs to: - Write bots which are interoperable within AH
- Measure in identical ways
- Easier to parse different feeds („generic santizer bot“) : you just have to define the mappings
abuse@ lookups suck (IRT object not in use, no standard; Just now RIPE DB is changing with abuse-c:) abuse@ lookups suck (IRT object not in use, no standard; Just now RIPE DB is changing with abuse-c:) Getting the right lookup is non-trivial, complex Many (national) CERTs create their own abuse contact lookup DBs. National CERT DB, TI directory, FIRST data can not be looked up automatically via scripts.
A caching contact database with more specific internal data A caching contact database with more specific internal data Some of this data (tel nos, etc) will never be in the public whois Unify with TI, FIRST etc data Make it query-able by scripts
What databases exist? What can we query? What databases exist? What can we query?
Public code repo ;-) Whois server (thx Mauro) RESTful API (Mauro, Rafiot) Some scripts to import TI data (Aaron, David) Still some bugs ;-)
Document (WIP): Document (WIP): https://github.com/certtools/contactdb/blob/master/doc/contact-databases-for-abuse-handling.mkd Codebase: https://github.com/certtools/contactdb (thx Rafiot, David, Mauro!)
AH Architecture showed the way… but … it’s complex AH Architecture showed the way… but … it’s complex Current experiments underway with RabbitMQ, Redis, logstash + Elastic Search Promising results so far, easy to use, more testing needed Problem: re-implement all parsers??
The CERT community has limited ressources for development The CERT community has limited ressources for development We re-implement the same thing all the time Let‘s share code or at least exchange ideas on how to automate incident handling! Let‘s share on how to measure success Thanks HKCERT, ENISA, CERT.at, CERT.pt, CIRCL, etc.. Mailinglist: https://tiss.trusted-introducer.org/mailman/listinfo/ihap
Dostları ilə paylaş: |