Data Stash — Event API Client
Data Stash can ingest data from different data sources, transform them, and then send JSON output via HTTP to the Openbridge Events API. You can also store the outputs into other formats such as CSV.
What Is Logstash?
Logstash is an open source data collection engine with real-time pipelining capabilities. Logstash can dynamically unify data from disparate sources and normalize the data into destinations of your choice. Cleanse and democratize all your data for diverse advanced downstream analytics and visualization use cases.
Why Data Stash?
Data Stash can perform some magic by automatically processing, cleaning, encoding and streaming contents of one or more CSVs directly to our API. Once it arrives at our API we automatically route all the data to a destination table in your data warehouse. Since CSV files can be a bit messy we have pre-packaged processing configurations that turn those old files into first class data sources. Here are a few of the standard operations we have defined:
- Exclude columns resident in a CSV (e.g., remove/drop the userID, email address and social security columns) from the output
- Replace non-ASCII characters with an ASCII approximation, or if none exists, a replacement character defaults
- Remove extraneous white space from records in target columns
- Strip backslashes, question marks, equals, hashes, minuses or other characters from the target columns
- Set the desired data type of a given column and have it transform records to meet that type
- Set everything to lowercase
- The proper UTF-8 encoding of the data
- Mask sensitive data with security “hashes” for one or more fields.
- Add new fields, such as IDs or concatenations of other columns, which can replace the contents of a column or store the results in a new field that is appended to the CSV
Data Stash is neatly packaged into a Docker image so you can run this on your local laptop or deploy it to a server. The first step is to build or pull the image via docker build -t openbridge/ob_datastash or simply pull it from Docker Hub:docker pull openbridge/ob_datastash:latest
Once you have your image you are ready to get started!