/hsredshift

📊 Data ingestion and ETL using Amazon Redshift.
  1. Python 100.0%
Python
Switch branches/tags
Nothing to show
Find file
Clone or download

Clone with HTTPS

Use Git or checkout with SVN using the web URL.

Download ZIP

Launching GitHub Desktop...

If nothing happens, download GitHub Desktop and try again.

Launching GitHub Desktop...

If nothing happens, download GitHub Desktop and try again.

Launching Xcode...

If nothing happens, download Xcode and try again.

Launching Visual Studio...

If nothing happens, download the GitHub extension for Visual Studio and try again.

Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
hsredshift Update card_played_popularity_report for Taverns of Time Jun 11, 2018
ideas Add script for ladder climbing streaks Apr 7, 2018
loaders/emr Clean up imports with isort Dec 4, 2017
scripts Move annotate_replay functionality to hsreplay.utils Apr 20, 2018
tests Fix initial_dbf_id for transforming CHOOSE_ONE cards Jun 6, 2018
udfs Add other class support to archetype name UDF May 14, 2018
.editorconfig Add .editorconfig May 29, 2017
.gitignore Update .gitignore Jan 5, 2018
.travis.yml Remove empty `env:` key to fix build Apr 20, 2018
README.md Update Travis badge in README.md Jun 12, 2017
requirements_extra.txt Add a deploy stage to travis.yml Dec 4, 2017
setup.cfg Require sqlalchemy-redshift >= 0.7.1 Jan 27, 2018
setup.py Update setup metadata to use setup.cfg May 15, 2017
tox.ini tox: Use flake8-isort Apr 13, 2018

README.md

Redshift Data Warehouse Libraries for HSReplay.net

Build Status

Libraries for ETL and analysis of HSReplay.xml files via Redshift.

Running MRJobs

This section assumes that your working directory is the repository root.

To run a job locally:

$ python <JOB_NAME>.py <INPUT_FILE.TXT>

To generate an inputs.txt data set:

$ PYTHONPATH=$PYTHONPATH:. python ./loaders/emr/generate_inputs.py

By default the file will go to loaders/emr/data/ this will also stage the inputs.txt file on S3 to make launching the job faster. To bootstrap a cluster:

$ mrjob create-cluster --conf-path ./loaders/emr/mrjob.conf
<CLUSTER-ID>

Then to run the job on the cluster:

$ python my_job.py -r emr --conf-path mrjob.conf --cluster-id <CLUSTER-ID> <INPUT_S3_PATH> --no-output --connection="<upload-db connection string>"

A concrete example might look like:

PYTHONPATH=$PYTHONPATH:. python ./loaders/emr/load_redshift.py -r emr --conf-path ./loaders/emr/mrjob.conf --cluster-id j-2YYNNMRVT35MN s3://hearthsim-mrjob/data/2017-01-05_00-00-00_TO_2017-01-05_00-10-00_inputs.txt --no-output

License

Copyright © HearthSim - All Rights Reserved

Community

This is a HearthSim project. All development happens on our IRC channel #hearthsim on Freenode.