Releasing sources on GitHub

Signed-off-by: Sébastien Mériot <[email protected]>

Author: PandiPanda69

.gitignore

@@ -0,0 +1,5 @@
+*.pyc
+.*.sw*
+doc/build
+doc/.*
+.coverage

CHANGELOG.md

@@ -0,0 +1,4 @@
+# Changelog
+
+## 1.0.0 (2016-02-08)
+- Publishing sources on GitHub.

CONTRIBUTING.md

@@ -0,0 +1,47 @@
+Every contribution are welcomed as long as few rules are followed.
+
+# TL;DR
+
+- Branch off `devel`.
+- Only one feature per commit.
+- In case of changes request, amend your commit to avoid multiple commits.
+- Run `make lint` before and after coding and take care not to lower the rating.
+- Write unit tests as much as possible so we can easily check your code.
+- Run unit tests using `make test` before commiting.
+
+# How to contribute
+
+- If you're thinking about a new feature, see if there's already an issue open
+about it or please open one otherwise.
+- One commit per feature.
+- Branch off the `devel` branch.
+- Test your code with unit tests and use `make test` to run them all.
+- Run `make lint` before and after coding and take care not to lower the rating.
+It worth mentioning you can freely improve the rating !
+- If we ask for a few changes, please amend your commit rather than creting new
+commits.
+- Remember to be `Python 2.7` compliant.
+
+# How to start coding ?
+
+First thing to do is to install dev requirements by running `make install-dev-deps`.
+Everything you need to start coding should be setup.
+To be sure everything went ok, run unit tests using command `make test`.
+
+If needed, edit the `config/settings.py` so you can change logger level to
+debug (`LOGGER['level']` and `API['debug']` properties).
+
+# Branches
+
+- `master` still contains a stable version of the code. Releases are tagged
+from this branch.
+- `devel` contains all changes in the current development version. Don't use
+this code in production since it might break it.
+
+# Licensing for new files
+
+`ip-reputation-monitoring` is licensed under GNU GPL v3 license. Anything
+contributed to `ip-reputation-monitoring` must be released under this license.
+
+When introducing a new file into the project, please make sur it has a
+copyright header making clear under which licenses it's being released.

LICENSE

@@ -0,0 +1,38 @@
+CD=cd
+FIND=find
+DEL=rm
+PYTHON=python
+PIP=pip
+COVERAGE=coverage
+PYLINT=pylint
+SPHINX=sphinx-build
+
+MODULE_DIR=reputation
+DOC_DIR=doc/build
+
+clean: clean-doc
+	$(FIND) . -name *.pyc | xargs $(DEL)
+
+clean-doc:
+	$(DEL) -rf $(DOC_DIR)
+
+install-deps:
+	$(PIP) install -r requirements.txt
+
+install-dev-deps:
+	$(PIP) install -r requirements/dev.txt
+
+test: 
+	cd $(MODULE_DIR) && \
+	$(COVERAGE) > /dev/null 2>&1 && \
+	$(COVERAGE) run --source='.' -m unittest discover && \
+	$(COVERAGE) report \
+		|| \
+	$(PYTHON) -m unittest discover
+
+lint:
+	cd $(MODULE_DIR) && \
+	$(PYLINT) --disable=W0141 --max-line-length=150 adapters/ api/ archive/ config/ default/ factory/ main.py mongo/ parsing/ reporting/ run_api.py tests/ spamhaus_monitor.py  tools/ utils/
+
+doc: clean-doc
+	$(SPHINX) -b html doc/source doc/build

Makefile

@@ -0,0 +1,281 @@
+# ip-reputation-monitoring #
+
+## Summary ##
+
+This tool aims to monitor the reputation of a network in order
+to easily detect blacklisted IPs or IPs sending suspicious e-mails.
+Every day an e-mail is sent telling you the 10th IPs having the
+worst reputation among the network.
+
+The monitoring is based over both RBL and FBL as described below.
+
+An API is provided in order to query the reputation of a single
+IP using RBL and FBL aggregation. The API also offers an endpoint
+that checks DNS black list.
+
+### RBL ###
+
+A Realtime Blackhole List is composed of IP addresses considered as
+spammers. Several organizations are maintaining such list and this tool
+supports some of them: Microsoft SNDS, BlockList, CleanTalk, StopForumSpam.
+
+Generally, those lists are released under CSV format. Unfortunately,
+CleanTalk does only release this list in HTML, so a HTML to CSV
+parser has been written to do so.
+
+### FBL ###
+
+A feedback loop is a realtime feedback about mails sent from observed
+network. Those mails can be sent by e-mail hosting providers or by
+foundations fighting against spams in order to give a feedback about
+suspicious mails issued from the network.
+The tool supports Abuse Reporting Format (ARF) which is a standard
+to report suspicious mails used by AOL and SignalSpam feebacks.
+
+SpamCop feebacks are also supported by the tool, even if their mails
+do not follows ARF guidelines.
+
+### DNS BL ###
+
+The DNS-based blackhole list is a subset of RBL and, instead of being a
+CSV file containing an IP list, is a software mechanism allowing anyone
+to query the database using the DNS protocol whether an IP is blacklist.
+
+Supported DNS BL are the following:
+ - Abuse combined
+ - Abuseat
+ - Msrbl
+ - Sorbs
+ - Spamhaus ZEN
+ - SpamCop
+ - UCE Protect
+ - Wpbl
+
+### Spamhaus BL ###
+
+Spamhaus is an organization tracking spam over Internet. Their website
+provides an easy way to keep an eye over active issues on a network giving
+the domain responsible of an IP set.
+
+This tool provides a script able to parse the issues raised by Spamhaus to
+track network reputation.
+
+## Installation ##
+
+### Requirements ###
+
+To setup this little tool, you'll need:
+
+ * A Linux environment (it might work under Windows but you'll need to rewrite
+ shell scripts)
+ * Python 2.7+
+ * Packages `python-dev` and `python-pip`
+ * A MongoDB database (2.6.x or greater with TLS support*)
+ * A MX supporting IMAPS
+ * A scheduler (cron, supervisor, ...)
+
+*TLS support can be disabled by editing `settings/config.py`.
+
+### Step by step ###
+
+When all of these requirements are met, you can install the tool:
+
+ 1. Download the zipfile or checkout the sources.
+ 2. Install python dependencies (apt-get install python-dev python-pip)
+ 3. Run `make install-deps`.
+
+
+## Configuration ##
+
+### General settings ###
+
+For security purpose, most of setting values must be defined as VARENV.
+So, to configure the tool, you just have to export following environment
+variables:
+
+ * `AS_NUMBER`: your AS number, i.e.: OVH one is 16276
+ * `EMAIL_HOST`: ip or domain of your MX
+ * `FBL_USER`: username used to poll incoming e-mails
+ * `FBL_PASSWORD`: password for previous username
+ * `FBL_PARTNER_HEADER`: Name of the header defining e-mail source (see next
+ section)
+ * `MONGO_HOST`: ip or domain of your MongoDB host
+ * `MONGO_PORT`: port to reach MongoDB
+ * `MONGO_DB`: name of the MongoDB database
+ * `MONGO_USER`: MongoDB user
+ * `MONGO_PASSWORD`: MongoDB password
+ * `REPORTING_TARGET`: e-mail address where a daily report about ips with the
+ worst reputation must be sent
+ * `REPORTING_SENDER`: `From:` header value of the dail report e-mail
+ * `SNDS_KEY`: your personal SNDS key
+ * `SPAMHAUS_DOMAIN_NAME`: the domain your IPs are attached to, i.e.: OVH one
+ is `ovh.net`
+
+### Tagging incoming e-mails ###
+
+As you can see, a required varenv is called `FBL_PARTNER_HEADER`.
+Since everybody is able to send an email pretending being Microsoft SNDS,
+NSA or whatever, it's important to check the identity of the mail send with
+a well-kept secret. The easier way to do so is to assign a single e-mail
+address per organization you keep secret.
+
+Using the MX features, the mail must be validated and tagged with the previous
+ `FBL_PARTNER_HEADER` header. We recommend to keep this header name secret too.
+The value of this header is the name of the FBL. (currently supported values:
+AOL, SignalSpam, SpamCop)
+
+An example should help you to understand:
+
+    OVH wants to received spam report from AOL.
+
+    OVH creates a new e-mail address for AOL: [email protected].
+    AOL uses it to send all spam report.
+    OVH adds a new rule attached to this address: when receiving a new e-mail
+    from  <[email protected]>, add header "X-PARTNER-HEADER: AOL" and forward this
+    e-mail to the FBL_USER varenv mailbox.
+
+    The monitoring tool is polling MX the FBL_USER varenv mailbox on EMAIL_HOST
+    server and its "FBL_PARTNER_HEADER" has been set to "X-PARTNER-HEADER`.
+
+
+### Network IPs ###
+
+At last, you need add your network addresses CIDRs (only one per line) in the
+`config/ips.list` file.
+
+### Implementing its own RBL storage class ###
+
+As you can see in the `config.py`, there is a way to customize the tool by
+providing its own implementation to store RBL parsed documents.
+By default, a basic implementation is provided and store everything on
+the filesystem, using the property `RBL_STORAGE_CONTEXT` to determine the
+root path to use.
+
+You can code your own RBL storage service by implementing
+`adapters.service.storage.StorageServiceBase` and then, tell the tool to
+use this implementation by editing the property `CUSTOM_IMPLEMENTATIONS`.
+
+### Implementing new parsers ###
+
+Implementing new parsers is painless and you'll only have to implement an
+interface. If your parser is valid enough, it should be automatically
+registered and enabled as long as daemons are restarted.
+
+**1. Implementing a new CSV parser**
+
+For a brand new CSV parser, you'll need to implement
+`parser.csv.csvparser.CSVParser`. Here are the explanations of methods that
+must be implemented:
+
+    # -*- encoding: utf-8 -*-
+
+    from datetime import datetime
+    from parser.csv.csvparser import CSVParser
+
+    class MyNewParser(CSVParser):
+
+        def __init__(self, path):
+            # Consider the delimiter is a comma.
+            CSVParser.__init__(self, path, ',')
+
+        def compute_weight(self, data):
+            # The weight of this entry. All the weights are then summed to rank IPs.
+            # Notice data is an array containing the splitted line.
+            return 1
+
+        def get_date(self, data):
+            # Date to use for this entry
+            return datetime.now()
+
+        def get_source(self, data):
+            # Source name
+            return 'My new parser'
+
+        def get_ip(self, data):
+            # IP the entry is talking about
+            return data[0]
+
+        def get_description():
+            # Mandatory method to be automatically registered !
+            return {
+                'name': 'My new parser',
+                'shortened': 'MNP'  # Shortened name, optionnal
+            }
+
+        get_description # staticmethod(get_description)
+
+**2. Implementing a new mail reader**
+
+Unlike CSV parsers, mail reader are not automatically registered.
+That's why if you only want  to add a new provider using ARF,
+you'll have to edit `parsing.mails.mailfactory.MailReaderFactory` and add the
+new source name. (This code can be greatly improved, your PR are welcomed !)
+
+For a brand new mail reader, you'll need to edit this file too to add your
+source. And, you'll have to implement an abstract class to be able to read
+mails. Here is the default format:
+
+    # -*- encoding: utf-8 -*-
+
+    from datetime import datetime
+    from parsing.mails.mailreader import AbstractMailReader
+
+    class MyNewReader(MailReader):
+
+        def __init__(self, raw):
+            AbstractMailReader.__init__(self)
+            # Raw is the received e-mail.
+            self._data # raw
+
+        def compute_weight(self):
+            # The weight of this entry. All the weights are then summed to rank IPs.
+            return 1
+
+        def get_date(self):
+            # Date to use for this entry.
+            return datetime.now()
+
+        def get_source(self):
+            # Source name
+            return 'My new mail reader'
+
+        def get_ip(self):
+            # IP the mail is talking about
+            return "1.2.3.4"
+
+### Extending DNS BL support ###
+
+If you want to add new DNS BL that are not supported by default, you just have
+to edit the file `config/dnsbl.py` and add a new dictionnary providing needed
+information about the DNS BL.
+
+Note that the shortened name is mandatory.
+
+
+## Running ##
+
+You can now insert theses entries in your favorite scheduler:
+
+ * Schedule `reputation-rbl.sh` to be run once a day.
+ * Schedule `spamhaus-bl.sh` to be run every hour.
+ * Schedule `reputation-fbl.sh` and `api.sh` to be run as a daemon.
+
+## API ##
+
+Once everything is running, you can start using the API. By default, it's
+listening to the port 5000.
+Here are the few available endpoints:
+
+ * `GET /reputation/(ip)`: Query reputation of an IP for each registered source.
+ * `GET /reputation/(ip)/details/(source)`: Query the detailed reputation of an
+ IP for a given source. You must use the shortened name of the source (or its
+   name if no shortened one has been provided). Default available source: AOL,
+   BLCK, CTALK SCOP SFS, SGS, SNDS.
+ * `GET /blacklist/(ip)`: Query DNS BL to know whether this IP is black listed
+ or not.
+ * `GET /spamhaus/active`: Query recorded Spamhaus active issues.
+ * `GET /spamhaus/resolved`: Query recorded Spamhaus resolved issues.
+
+
+
+Enjoy