Firehose to Athena
Firehose to Athena
In order to have audit data for your app, you can use the prebuilt audit class. There is no additional setup required to use this.
from loggers import Audit
# example task
def _add(x, y): return x + y
# global or session defined config
config = dict(
app='audit',
email='richardfernandeznyc@gmail.com',
tags=dict(
environment='non-prod',
team='NTA',
)
)
# logger can be initialized elsewhere
logger = Audit.configure('firehose', **config)
# runtime execution happens here
audit = logger.log(
task='_add',
task_args=dict(x=1, y=2),
ticket_id='nta-1000',
)
result = _add(1, 2)
# Optionally, log your output results.
audit.log_output(output=result)
If you need your own brand new data stream:
Add a schema to schemas folder in this project’s root.
Run the following script to create your table in Athena:
zappa invoke 'src/firehose_to_athena/table_utils.create_table'
zappa invoke 'src/firehose_to_athena/firehose_utils.create_stream'
(Optional) Manually running previous step in AWS console:
Test your implementation:
python src/loggers/audit.py
Modify this example in your code for your stream.
```python
from logger import ValidatedLog
def _add(x, y): return x + y
config = dict(
app=’audit’,
email=’rich@namely.com’,
tags=dict(
environment=’non-prod’,
team=’NTA’,
)
)
logger = ValidatedLog.configure(‘firehose’, **config)
writing to.
audit = logger.log(
‘Audit’,
task=’_add’,
task_args=dict(x=1, y=2),
ticket_id=’nta-1000’,
)
…
```