Module Chaining
Taking all of the above into consideration, this section will cover how
module chaining is supposed to work from a module authors perspective.
Primary Modules
Modules that ingest the core types ‘tcp’, ‘udp’, and ‘ip’ can return an
instance of ChopProtocol to be consumed by secondary modules. Before
use, ChopProtocol must be imported by doing:
from ChopProtocol import ChopProtocol
To instantiate an instance of ChopProtocol you can do something like:
myhttpinstance = ChopProtocol('http')
The argument passed to ChopProtcol is the ‘type’ of the data being
passed, in the above example, the data is of type ‘http’.
After instantiating an object based on ChopProtocol you have access to
the following functions:
setAddr - Set the quadtuple containing source ip/port and destination
ip/port – this will be auto set by the framework if you do not
setTimestamp - Set variable that contains the timestamp of the protocol
– this will be autoset to the timestamp of whatever packet you return
data on if you do not set it
setClientData - Set the arbitrary data structure for the data coming
from the client
setServerData - Set the arbitrary python data structure for the data
coming from the server
setTeardown - (ChopLib 4.3+) Indicate this data should be forwarded to
downstream module’s teardown functions.
Note that the format of ChopProtocol is not meant to be restrictive. You
can and should override or ignore some functionality if it doesn’t fit
your model of how data should be handled (e.g., creating a ‘data’
element instead of having client and server elements). Before returning
an instance of ChopProtocol it is recommended you familarize yourself
with internal structure of the class. It is also extremely important
that you thoroughly document the format and organization of the object
you return from your module.
_clone function
ChopLib requires the ability to create copies of ChopProtocol to provide
modules with their own unique copy. By default ChopProtocol contains a _clone
function that uses copy’s ‘deepcopy’ function. If your data (e.g., clientData
and serverData) are complex enough, this might not be enough to copy your data.
In these instances you should create an inherited class based on ChopProtocol
and redefine the _clone function.
Secondary Modules
If you want to write a decoder for a protcol that runs on top of another
protocol, such as http, normally you would first parse the http traffic
out and then proceed to parse the protocol that you were actually trying
to decode. With module chaining, you can pass the data through a primary
module that takes tcp and turns it into http and then focus on only the
protocol you care about
As documented above, secondary modules have one function they must
define to handle data:
handleProtocol(protocol) – Protocol data, partially defined by primary
module
Starting with ChopLib 4.3, you can optionally define the following to
handled ‘teardown’ data:
teardownProtocol(protocol) – Protocol data, partially defined by
primary module
Secondary modules can further return data to be used by other,
downstream secondary modules by the same procedure as primary modules
for returning custom types.
Note that module authors writing secondary modules should refer to
documentation for primary modules since the organization, data, etc in
what is returned by a primary module many not conform to the default
ChopProtocol syntax.
The “chop” library
ChopShop provides the “chop” library for module usage to interact with
the outside world. This allows the module writer to worry less about how
to output data. The chop library provides output “channels” to allow you
to very easily output data to the location of the module invoker’s
choosing. The following output channels are supported:
chop.prnt - Function that works similar to print, supports output to a gui, stdout, or a file depending on the users command line arguments
chop.debug - Debug function that outputs to a gui, stderr, or a file depending on the users command line arguments
chop.json - Outputs a json string based on an object passed to it, enabled if JSON output is enabled by the user
chop also provides the following other related functions:
chop.tsprnt - same as chop.prnt but prepends the packet timestamp to the string
chop.prettyprnt - same as chop.prnt but the first argument is a color string, e.g., "RED"
chop.tsprettyprnt - same as chop.tsprnt but the first argument is a color string, e.g., "CYAN"
chop.set_custom_json_encoder - given a reference to a function will attempt to use it as a custom json encoder for all calls to chop.json
chop.set_ts_format_short - accepts a boolean that enables short time format '[H:M:S]' (default is '[Y-M-D H:M:S TZ]')
DO NOT use python’s regular “print” command.
The following colors are currently supported with chop.prettyprnt and
chop.tsprettyprnt:
"YELLOW" - Yellow on a Black Background
"CYAN" - Cyan on a Black Background
"MAGENTA" - MAGENTA on a Black Background
"RED" - Red on a Black Background
"GREEN" - Green on a Black Background
"BLUE" - Blue on a Black Background
"BLACK" - Black on a White Background
"WHITE" - White on a Black Background
Note that if a gui is not available or colors are not supported in the
terminal running ChopShop, chop.prettyprnt’s functionality is equivalent
to chop.prnt.
Examples
Using the chop library is pretty straightforward, if you want to output
regular text data just type:
chop.prnt("foo")
chop.prnt("foo", "bar", "hello")
chop.prnt("The answer is: %s" % data)
If you would like to mirror the functionality of python’s print’s
ability to supress the trailing ‘’ added to output, you can do the
following:
chop.prnt("foo", None)
To color the data (for gui purposes) just type:
chop.prettyprnt("RED", "foo")
chop.prettyprnt("MAGENTA", "bar")
chop.prettyprnt("YELLOW", "bah", None)
If you would like to support outputting json data, you can utilize
chop.json to do so:
myobj = {'foo': ['bar', 'bah']}
chop.json(myobj)
If you feel the need to make your own custom json encoder, you can use
“chop.set_custom_json_encoder(encoder_function)” to customize how
the json will be output.
Note that the default json encoder does not support any non standard
types
File Saving
ChopShop provides a simple API for saving files using the chop.*file()
family of functions. There are three functions in this family:
chop.savefile
chop.appendfile
chop.finalizefile
The definition of chop.savefile() looks like:
def savefile(filename, data, finalize = True)
To use chop.savefile() you provide the filename and the data. The
optional third argument (a boolean) is used to determine if the file
object should be kept open or closed. This allows you to do
(pseudo-code):
while (chunk_of_data = decode_some_data_from_pcap):
if on_last_chunk:
finalize = True
else:
finalize = False
chop.savefile('foo', chunk_of_data, finalize)
If not given, the default behavior is to close the file object. Since
each file object is opened in write mode module authors need to be aware
of this behavior as it will overwrite any existing files with the same
name.
Similar to chop.savefile(), chop.appendfile() has the following
definition:
def appendfile(filename, data, finalize = False)
To use chop.appendfile() you provide the filename and the data. The
optional third argument (a boolean) is used to determine if the file
object should be kept open or closed. If not given, the default behavior
is to leave the file object open. Note, that unlike savefile, appendfile
opens files in “append” mode, so it will not overwrite any file that
already exists.
The last function in the file family is chop.finalizefile() – as the
name implies, it allows you to finalize (or close) a file once you are
done with it. It has the following definition:
def finalizefile(filename)
If the filename given is not open, finalizefile will do nothing. Also
note that you can use savefile or appendfile to the same affect by
calling them with an empty string as the data and finalize set to True.
E.g.:
chop.appendfile(filename, "", True)
chop.savefile(filename, "", True)
finalizefile gives you a shorter, quicker way to close the file that is
easier to see in code.
Note that as a module author you only provide the filename, not the full
path to the file you want created on disk. The full path is handled by
the -s argument to chopshop. For example:
chopshop -f foo.pcap -s "/tmp/%N" "gh0st_decode -s; awesome_carver -s"
This will make sure each carved file from gh0st_decode go into
/tmp/gh0st_decode and files from awesome_carver will go in
/tmp/awesome_carver. The other supported format string is “%T” which
will be translated into the current UNIX timestamp (/tmp/%N/%T would put
files in /tmp/module_name/timestamp).