piper

piper contains three commands for piping json to Azure EventGrid, AWS EventBridge and Splunk

Piper, Piper, the red, red worm

woke last night to the sound of the storm

her words were words I sailed upon

— Phish (from the song Piper)

eventbridge-pipe

Install

go get

go get github.com/willabides/piper/cmd/eventbridge-pipe

bindown

Add a bindown dependency:

$ bindown template-source add piper https://github.com/WillAbides/piper/releases/latest/download/bindown-templates.yml
$ bindown dependency add eventbridge-pipe piper#eventbridge-pipe

Usage

Usage: eventbridge-pipe --type=STRING --source=STRING
eventbridge-pipe posts events to AWS EventBridge.
example:
    $ AWS_ACCESS_KEY='AKIA****************'
    $ AWS_SECRET_KEY='shhh_this_is_a_secret'
    $ data="$(cat <<"EOF"
        {"action": "obj.add", "@timestamp": 1604953432032, "el_name": "foo", "doc_id": "asdf"}
        {"action": "obj.rem", "@timestamp": 1604953732032, "el_name": "bar", "doc_id": "fdsa"}
      EOF
      )"
    $ echo "$data" | \
      eventbridge-pipe -s 'test-source' -t 'jp:action' -b 'my-bus' -T 'jp:"@timestamp"' \
      -r 'jp:"el_name"'
Learn about JMESPath syntax at https://jmespath.org
Flags:
  -h, --help                     Show context-sensitive help.
      --region="us-east-1"       The aws region to publish events to.
  -t, --type=STRING              Value for the DetailType field. JMESPath
                                 expressions allowed with "jp:" prefix.
  -b, --event-bus=STRING         Value for the "EventBusName" field.
  -r, --resource=RESOURCE,...    An element for the list in the "Resources"
                                 array. JMESPath expressions allowed with "jp:"
                                 prefix.
  -s, --source=STRING            Value for the "Source" field. JMESPath
                                 expressions allowed with "jp:" prefix.
  -T, --timestamp=STRING         Value for the "Time" field converted from epoch
                                 milliseconds. JMESPath expressions allowed with
                                 "jp:" prefix.
      --batch-size=10            Number of events to send in a batch.
      --flush-interval=2000      Time in milliseconds to wait before sending a
                                 partial batch. Set to 0 to never send a partial
                                 batch.

eventgrid-pipe

Install

go get

go get github.com/willabides/piper/cmd/eventgrid-pipe

bindown

Add a bindown dependency:

$ bindown template-source add piper https://github.com/WillAbides/piper/releases/latest/download/bindown-templates.yml
$ bindown dependency add eventgrid-pipe piper#eventgrid-pipe

Usage

Usage: eventgrid-pipe --subject=STRING --type=STRING <topic-endpoint>
eventgrid-pipe posts events to Azure Event Grid.
example:
    $ topic_endpoint='https://mytopicendpoint.westus2-1.eventgrid.azure.net'
    $ topic_key='shhh_secret_topic_key'
    $ data="$(cat <<"EOF"
        {"action": "obj.add", "@timestamp": 1604953432032, "el_name": "foo", "doc_id": "asdf"}
        {"action": "obj.rem", "@timestamp": 1604953732032, "el_name": "bar", "doc_id": "fdsa"}
      EOF
      )"
    $ echo "$data" | \
      eventgrid-pipe "$topic_endpoint" \
      -H "aeg-sas-key: $topic_key" \
      -T 'jp:"@timestamp"' \
      -t 'audit-log' \
      -s 'jp:action' \
      -i 'jp:doc_id'
Learn about JMESPath syntax at https://jmespath.org
Arguments:
  <topic-endpoint>    Endpoint for posting events
Flags:
  -h, --help                   Show context-sensitive help.
  -i, --id=STRING              Value for the "id" field. If unset, a uuid will
                               be generated for each event. JMESPath expressions
                               allowed with "jp:" prefix.
  -s, --subject=STRING         Value for the "subject" field. JMESPath
                               expressions allowed with "jp:" prefix.
  -t, --type=STRING            Value for the "eventType" field. JMESPath
                               expressions allowed with "jp:" prefix.
  -T, --timestamp="now"        Value for the "eventTime" field converted from
                               epoch milliseconds. If unset, the current system
                               time will be used.JMESPath expressions allowed
                               with "jp:" prefix.
  -H, --header=HEADER,...      Header to sent with the request in the same
                               format as curl. e.g. '-H "aeg-sas-key: $EVKEY"'
      --data-version="1.0"     Value for the "dataVersion" field. JMESPath
                               expressions allowed with "jp:" prefix.
      --batch-size=10          Number of events to send in a batch.
      --flush-interval=2000    Time in milliseconds to wait before sending a
                               partial batch. Set to 0 to never send a partial
                               batch.

splunk-pipe

Install

go get

go get github.com/willabides/piper/cmd/splunk-pipe

bindown

Add a bindown dependency:

$ bindown template-source add piper https://github.com/WillAbides/piper/releases/latest/download/bindown-templates.yml
$ bindown dependency add splunk-pipe piper#splunk-pipe

Usage

Usage: splunk-pipe <endpoint>
splunk-pipe posts events to splunk.
example:
    $ splunk_endpoint="http://localhost:8080"
    $ splunk_hec_token="shhh_secret_token"
    $ data="$(cat <<"EOF"
        {"action": "obj.add", "@timestamp": 1604953432032, "el_name": "foo", "doc_id": "asdf"}
        {"action": "obj.rem", "@timestamp": 1604953732032, "el_name": "bar", "doc_id": "fdsa"}
      EOF
      )"
    $ echo "$data" | \
      splunk-pipe "$splunk_endpoint" \
      -H "Authorization: Splunk $splunk_hec_token" \
      -T 'jp:"@timestamp"'
Learn about JMESPath syntax at https://jmespath.org
Arguments:
  <endpoint>    Endpoint for posting events
Flags:
  -h, --help                   Show context-sensitive help.
  -t, --sourcetype=STRING      Value for the "sourcetype" field. JMESPath
                               expressions allowed with "jp:" prefix.
  -s, --source=STRING          Value for the "source" field. JMESPath
                               expressions allowed with "jp:" prefix.
  -T, --timestamp=STRING       Value for the "eventTime" field converted from
                               epoch milliseconds. JMESPath expressions allowed
                               with "jp:" prefix.
  -H, --header=HEADER,...      Header to sent with the request in the same
                               format as curl. e.g. '-H "Authorization: Splunk
                               $HEC_KEY"'
  -h, --host=STRING            Value for the "host" field. JMESPath expressions
                               allowed with "jp:" prefix.
      --index=STRING           Value for the "index" field. JMESPath expressions
                               allowed with "jp:" prefix.
      --batch-size=10          Number of events to send in a batch.
      --flush-interval=2000    Time in milliseconds to wait before sending a
                               partial batch. Set to 0 to never send a partial
                               batch.