Datasources

Extending Kismet: Creating Capture Sources #

Kismet supports additional capture types via the KisDatasource interface. Data sources run in an independent process and can be written in any language, however they require a C++ component which functions as a Kismet driver to allow it to communicate with the datasource binary.

Datasources can report packets or complex records - if your datasource needs to pass parsed information about a device event, that’s possible!

Capture via IPC and Network #

Kismet datasources communicate from the capture binary to the Kismet server via an IPC channel or TCP connection. This channel passes commands, data, and other objects via an extension of the Kismet External API protocol; written using the Google Protobuf library, this protocol is extensible and parsers can be generated for nearly any language.

The datasource IPC channel is via inherited file descriptors: Prior to launching the capture binary, the Kismet server makes a pipe(2) pair and will pass the read (incoming data to the capture binary) and write (outgoing data from the capture binary) file descriptor numbers on the command line of the capture binary.

Operating as a completely separate binary allows the capture code to use increased permissions via suid, operate independently of the Kismet main loop, allowing the use of alternate main loop methods or other processor-intensive operations which could stall the main Kismet packet loop, or even using other languages to define the capture binary, such as a Python capture system which utilizes Python radio libraries.

The network protocol is an encapsulation of the same protocol over a TCP channel, with some additional setup frames. The network protocol will be more fully defined in future revisions of this document.

The External Datasource Protocol #

The datasource capture protocol acts as additional commands within the Kismet External API; it is defined in protobuf_definitions/datasource.proto.

Datasource commands are in the KismetDatasource namespace, and their string equivalents in the helper API are prefixed with KDS. Reply-only frames are suffixed with RESPONSE.

KDS Commands #

KDSCLOSEDATASOURCE (KismetDatasource.CloseDatasource) Kismet -> Datasource #

Closes the active datasource; this is called during the shutdown process when a source is closed or Kismet exits.

Content #

None

Reply #

None

KDSCONFIGURE (KismetDatasource.Configure) Kismet -> Datasource #

Configure the behavior of a running source.

Content #

Response #

KDSCONFIGUREREPORT KismetDatasource.ConfigureReport

KDSCONFIGUREREPORT (KismetDatasource.ConfigureReport) Datasource -> Kismet #

Report configuration status and success to Kismet; This report must contain the sequence number of the KDSCONFIGURE request in the success field.

Content #

Response #

None

KDSDATAREPORT (KismetDatasource.DataReport) Datasource -> Kismet #

Datasources uses KDSDATAREPORT to send packets, signal data, and GPS data to Kismet. The packet payload is mapped to the Kismet datasource, and sent to the packet processing subsystem.

Content #

Response #

None

KDSERRORREPORT (KismetDatasource.ErrorReport) Datasource -> Kismet #

Fatal error condition which should initiate a datasource shutdown.

Content #

Response #

None

KDSLISTINTERFACES (KismetDatasource.ListInterfaces) Kismet -> Datasource #

Request a list of supported interfaces; Kismet uses this to populate the Data Sources display where a user can activate available sources.

Content #

None

Response #

KDSINTERFACESREPORT KismetDatasource.InterfacesReport

KDSINTERFACESREPORT (KismetDatasource.InterfacesReport) Datasource -> Kismet #

Returns a list of supported interfaces, if the datasource is capable of listing potential sources.

Content #

Response #

None

KDSNEWSOURCE (KismetDatasource.NewSource) Datasource -> Kismet #

Remote captures are multiplexed over a single TCP port; to associate a remote capture with the proper driver code in Kismet, the datasource must send a KDSNEWSOURCE.

Content #

Response #

After receiving and successfully mapping a KDSNEWSOURCE to a datasource driver, Kismet will send a KDSOPENSOURCE command to being configuration.

KDSOPENSOURCE (KismetDatasource.OpenSource) Kismet -> Datasource #

Kismet will start a datasource by sending a KDSOPENSOURCE; this will be sent for an IPC source or a remote capture source which has completed the initial handshake.

Content #

Response #

KDSOPENSOURCEREPORT KismetDatasource.OpenSourceReport

KDSOPENSOURCEREPORT (KismetDatasource.OpenSourceReport) Datasource -> Kismet #

A KDSOPENSOURCEREPORT carries all the information about a new datasource.

Content #

Response #

None

KDSPROBESOURCE (KismetDatasource.ProbeSource) Kismet -> Datasource #

Kismet will attempt to match a source to a datasource driver by asking each datasource to probe the source definition.

Content #

Response #

KDSPROBESOURCEREPORT KismetDatasource.ProbeSourceReport

KDSPROBESOURCEREPORT (KismetDatasource.ProbeSourceReport) Datasource -> Kismet #

Content #

Response #

None

KDSWARNINGREPORT (KismetDatasource.WarningReport) Datasource -> Kismet #

KDSWARNINGREPORT can be used by the datasource to set a non-fatal warning condition; this will be shown in the datasource details.

Content #

Response #

None

KDS Subcomponents #

When the same data is used in multiple packets, it is typically placed in a KismetDatasource.Sub... message which is included in the top-level command.

Many Sub... blocks contain only a single field; these may be expanded in the future to contain multiple fields, depending on the requirements of the protocol.

KismetDatasource.SubBuffer #

Some datasources may need to send a complete protobuf record for data; this can be accomplished with the SubBuffer structure, which contains simply:

KismetDatasource.SubChannels #

Basic list of channels. Channels are reported as strings, as they can represent complex tuning options (such as 6HT40+ for Wi-Fi).

KismetDatasource.SubChannel #

Basic channel. Channels are reported as strings, as they can represent complex tuning options.

KismetDatasource.SubChanhop #

KismetDatasource.SubGps #

KisDatasource.SubJson #

KisDatasource.SubInterface #

KisDatasource.SubPacket #

Raw packet data is injected into the Kismet Packetchain system. Datasources which send data with a DLT handled by Kismet will be automatically processed; datasources sending a new DLT will have to provide a parser for that link type.

KisDatasource.SubSignal #

Some packet formats include signal-level data as part of the packet headers (for example, Radiotap); for other packets, this data may be available as an external set of data.

KisDatasource.SubSpecSet #

For data sources which support raw spectrum capture, the SubSpecSet configuration block will be sent to configure the ranges.

KisDatasource.SubSpectrum #

Data sources which support raw spectrum capture return the spectrum record in a SubSpectrum.

KisDatasource.SubSuccess #

Response messages include a SubSuccess; this is used to indicate command completion.

Defining the driver: Deriving from KisDatasource #

The datasource driver is the C++ component which brokers interactions between the capture binary and Kismet.

All datasources are derived from KisDatasource. A KisDatasource is based on a tracker_component to provide easy export of capture status.

The amount of customization required when writing a KisDatasource driver depends on the amount of custom data being passed over the IPC channel. For packet-based data sources, there should be little additional customization required, however sources which pass complex pre-parsed objects will need to customize the protocol-handling methods.

KisDatasource instances are used in two ways:

1. Maintenance instances are used as factories to create new instances. A maintenance instance is used to enumerate supported capture types, initiate probes to find a type automatically, and to build a capture instance.
2. Capture instances are bound to an IPC process for the duration of capture and are used to process the full capture protocol.

At a minimum, new datasources must implement the following from KisDatasource:

probe_type(…) is called to find out if this datasource supports a known type. A datasource should return true for each type name supported.

virtual bool probe_type(string in_type) {
    if (StrLower(in_type) == "customfoo")
        return true;

    return false;
}

build_data_source(…) is the factory method used for returning an instance of the KisDatasource. A datasource should simply return a new shared ptr instance of its custom type.

virtual std::shared_ptr<kis_datasource> 
    build_datasource(std::shared_ptr<kis_datasource_builder> in_shared_builder) { 
        return std::make_shared<custom_kis_datasource>();

    };

A datasource which operates by passing packets should be able to function with no further customization: Packet data passed via the PACKET record will be decapsulated and inserted into the packetchain with the proper DLT.

Handling incoming packets #

Generally, nothing needs to be done to handle incoming packets, the parent datasource definition will take care of it. If, however, you receive packets which need modification before they are injected into the packetchain, you can override handle_rx_datalayer(...):

void kis_datasource::handle_rx_datalayer(std::shared_ptr<kis_packet> packet,
        const KismetDatasource::SubPacket& report) {

    auto datachunk = packetchain->new_packet_component<kis_datachunk>();

    if (clobber_timestamp && get_source_remote()) {
        gettimeofday(&(packet->ts), NULL);
    } else {
        packet->ts.tv_sec = report.time_sec();
        packet->ts.tv_usec = report.time_usec();
    }

    if (get_source_override_linktype()) {
        datachunk->dlt = get_source_override_linktype();
    } else {
        datachunk->dlt = report.dlt();
    }

    packet->set_data(report.data());
    datachunk->set_data(packet->data);

    get_source_packet_size_rrd()->add_sample(report.data().length(), time(0));

    packet->insert(pack_comp_linkframe, datachunk);
}

Any implementation of handle_rx_datalayer(...) MUST implement the above framework, but may manipulate the DLT, timestamp, and contents of the kis_datachunk record in any way needed.

Similarly, handle_rx_jsonlayer(...) can be overridden to rewrite JSON non-packet records.

Handling the PHY #

Kismet defines phy_handler objects to handle different physical layer types - for example there are phyhandlers for IEEE802.11, Bluetooth, and so on.

A phy handler is responsible for defining any custom data structures specific to that phy, converting phy-specific data to the common interface so that Kismet can make generic devices for it, providing any additional javascript and web resources, and similar tasks.

Defining the PHY #

Phy handlers are derived from the base kis_phy_handler class.

At a minumum a new phy must provide (and override):

• The basic C++ contructor and destructor implementations
• The create function to build an actual instance of the phy handler
• A common classifier stage to create common info from the custom packet info
• A storage loader function to attach any custom data when a device is loaded from storage

Loading from storage and custom data types #

A new phy will almost certainly define a custom tracked data type - dot11_tracked_device and bluetooth_tracked_device for instance. As part of defining this custom type, the phy must provide a storage loader function to import stored data into a proper object.

In addition, there are some specific pitfalls when loading custom objects - be sure to check the “Restoring vector and map objects” section of of the tracked_component docs!

Currently storage code is not used, but may be leveraged in the future.

Handling the DLT #

A datasource which is packet-based but does not conform to an existing DLT defined in Kismet will often need to provide its own DLT handler.

Do I need a custom DLT handler? #

If data records are entirely parsed by the classifier (see below for more information), then a separate DLT handler may not be necessary, however if your DLT embeds signal, location, or other information which needs to be made available to other Kismet data handlers, it should be decoded by your DLT handler.

Capture sources implementing alternate capture methods for known DLTs (for instance, support for 802.11 on other operating systems, etc.) do not need to implement a new DLT handler.

Deriving the DLT #

Kismet DLT handlers are derived from kis_dlt_handler from kis_dlt.h. A DLT handler needs to override the constructor and the handle_packet(...) functions:

class dlt_example : public kis_dlt_handler {
public:
    dlt_example();

    virtual int handle_packet(std::shared_ptr<kis_packet> in_pack) override;
};

dlt_example::dlt_example() :
    kis_dlt_handler() {

    // Packet components and insertion into the packetchain is handled
    // automatically by the Kis_DLT_Handler constructor.  All that needs
    // to happen here is setting the name and DLT type 
    dlt_name = "Example DLT";

    // DLT type is set in tcpdump.h 
    dlt = DLT_SOME_EXAMPLE;

    // Optionally, announce that we're loaded
    _MSG_INFO("Registering support for DLT_SOME_EXAMPLE");
}

// handle_packet(...) is called by the packet chain with the packet data
// as reported by the datasource.  This may already include GPS and signal
// information, as well as the actual link data frame.

// HandlePacket is responsible for decapsulating the DLT, creating any
// additional kis_packet records, and prepping the data for the classifier
// stage.

int dlt_example::handle_packet(std::shared_ptr<kis_packet> in_pack) {
    // Example sanity check - do we already have packet data
    // decapsulated?  For a type like radiotap or PPI that encodes another
    // DLT, this encapsulated chunk might be handled differently 
    auto decapchunk = in_pack->fetch_as<kis_datachunk>(pack_comp_decap);

    if (decapchunk != nullptr) {
        return 1;
    }

    // Get the linklayer data record 
    auto linkdata = in_pack->fetch_as<kis_datachunk>(pack_comp_linkframe);

    // Sanity check - do we even have a link chunk? 
    if (linkdata == nullptr) {
        return 1;
    }

    // Sanity check - does the DLT match? 
    if (linkdata->dlt != dlt) {
        return 1;
    }

    // Other code goes here 
}

Manipulating packet data #

It is important to minimize copying of packet data whenever possible. While sometimes unavoidable, many manipulations of packet data are simply removing headers and trailing data from the record.

Kismet uses a multi-C++ version of stringview via the nonstd::stringview library. This should be used to manipulate the windowed views of packet data, and in general, data records in a packet are all views of the same initial packet data from the IPC protocol. For example, the following code is used in the Radiotap DLT:

    auto decapchunk = packetchain->new_packet_component<kis_datachunk>();
    ...
	decapchunk->dlt = KDLT_IEEE802_11;
    ...
    decapchunk->set_data(linkchunk->substr(offset, linkchunk->length() - offset - fcs_cut));
    ...
	in_pack->insert(pack_comp_decap, decapchunk);

Using the stringview substr function keeps us from copying data unnecessarily.

Minimizing memory allocations #

The packet handling loop is an extremely commonly used loop, where small changes can have large impacts. To minimize thrashing the memory allocation system, Kismet uses object pools for the packet components. Notice the following code:

    auto decapchunk = packetchain->new_packet_component<kis_datachunk>();

This uses the packetchain object pool for packet components. If there is a returned object in the pool, it will be reset and returned; if the pool is exhausted, a new object will be created and returned. Objects created this way are automatically returned to the pool at the end of their lifetime.

Handling Non-Packet Data #

Non-packet data can be decapsulated by extending the KisDataSource::handle_packet method. By default this method handles defined packet types; an extended version should first call the parent instance.

void some_data_source::handle_packet(string in_type, KVmap in_kvmap) {
    KisDataSource::handle_packet(in_type, in_kvmap);

    string ltype = StrLower(in_type);

    if (ltype == "customtype") {
        handle_packet_custom(in_kvmap);
    }
}

Extended information can be added to a packet as a custom record and transmitted via the Kismet packetchain, or can be injected directly into the tracker for the new phy type (See the tracked components docs for more information). Injecting into the packet chain allows existing Kismet code to track signal levels, location, etc., automatically.

If the incoming data is directly injected into the data tracking system for the new phy type, then special care must be taken to create pseudo-packet records for the core device-tracking system. Ultimately, a pseudo-packet event must be created, either when processing the custom IPC packet or in the device classifier. Generally, it is recommended that a datasource attach the custom record to a packet object and process it via the packetchain as documented in the tracked components docs.

When processing a custom frame, existing KV pair handlers can be used. For example:

void some_data_source::handle_packet_custom(KVmap in_kvpairs) {
    KVmap::iterator i;

    // We inject into the packetchain so we need to make a packet
    kis_packet *packet = NULL;

    // We accept signal and gps info in our custom IPC packet
    kis_layer1_packinfo *siginfo = NULL;
    kis_gps_packinfo *gpsinfo = NULL;

    // We accept messages, so process them using the existin message KV
    // handler
    if ((i = in_kvpairs.find("message")) != in_kvpairs.end()) {
        handle_kv_message(i->second);
    }

    // Generate a packet using the packetchain
    packet = packetchain->GeneratePacket();

    // Gather signal data if we have any
    if ((i = in_kvpairs.find("signal")) != in_kvpairs.end()) {
        siginfo = handle_kv_signal(i->second);
    }

    // Gather GPS data if we have any
    if ((i = in_kvpairs.find("gps")) != in_kvpairs.end()) {
        gpsinfo = handle_kv_gps(i->second);
    }

    // Add them to the packet
    if (siginfo != NULL) {
        packet->insert(pack_comp_l1info, siginfo);
    }

    if (gpsinfo != NULL) {
        packet->insert(pack_comp_gps, gpsinfo);
    }

    // Gather whatever custom data we have and add it to the packet
    if ((i = in_kvpairs.find("customfoo")) != in_kvpairs.end()) {
        handle_kv_customfoo(i->second, packet);
    }

    // Update the last valid report time
    inc_num_reports(1);
    set_last_report_time(globalreg->timestamp.tv_sec);

    // Inject the packet into the packet chain, this will clean up
    // the packet when it's done with it automatically.
    packetchain->processPacket(packet);
}