EdData

Overview

The insights from the data lake can be downloaded as a CSV file using EdData data export utility or using AWS Athena Client / GCP Bigquery Client. This utility connects to the Data lake and downloads the data specific to each customer. This utility requires customer specific credentials to download the data, currently supported Cloud Providers :

• AWS
• GCP

Terms & Conditions

By Agreeing,

• I hereby confirm that I will not include the credentials/JSONS/Keys in any Public Repository. If found I will be the sole responsible person and any action that might follow
• I hereby confirm that I will not disclose the credentials/JSONS/Keys to other internal/external people

Authentication and Configuration Details

Authentication and Configuration details will be provided by EdCast support team which should be utilized as parameters in the utility.

EdData Export - AWS

Installation

This python based utility can be installed and run from a Unix or Windows environment. Below are the steps to install and run the utility.

1. Install python3
2. Download data export utility from the eddata-sdk github repository
3. This repo includes below 2 files
   • requirements.txt - This files contains the python libraries required by the utility
   • edc_bulk_download.py - This is the data export utility that connects to the data lake and
     downloads the file as CSV
4. Install the python pre-req libraries using below command
   • pip3 install -r requirements.txt

Command to run

python3 edc_data_export.py --region us-west-2 --query "select * from user_card_performance_reporting_i_v where day='2020-07-04'" --aws_access_key_id <<sample_access_key>> --aws_secret_access_key <<sample_secret_key>> --filename download_data.csv --org_id 300032 --env prod --s3bucket edc-analytics-us-west-2-prod-02

Above command runs and stores the extracted data in CSV format in the download_data.csv.

Sample Queries

Some sample queries that can be used in the utility

• select * from user_card_performance_reporting_i_v where day=’2020-04-01’
• select * from user_card_performance_reporting_i_v where day between ‘2020-04-01’ and ‘2020-04- 04’
• select * from user_card_performance_reporting_i_v where day between ‘2020-04-01’ and ‘2020-04-04’ and user_email=’[email protected]’
• select * from user_card_performance_reporting_i_v where day between ‘2020-04-01’ and ‘2020-04-04’ and user_first_name like ’admin%
• select * from user_card_performance_reporting_i_v where day between ‘2020-04-01’ and ‘2020-04-04’ and card_tile like ‘admin%’
• select * from user_assignments_performance_i_v where day between ‘2020-04-01’ and ‘2020-04-04’ and assignment_state=’completed’

There is no output to the utility call, a file will be downloaded locally with the given name.

EdData Export - GCP

Installation

This python based utility can be installed and run from a Unix or Windows environment. Below are the steps to install and run the utility.

1. Install python3
2. Download data export utility from the eddata-sdk github repository
3. Download below 2 files
   a. gcp_requirements.txt - This files contains the python libraries required by the utility
   b. edc_data_export.py - This is the data export utility that connects to the data lake and downloads the file as CSV
4. Install the python pre-req libraries using below command
   a. pip3 install -r gcp_requirements.txt

Command to run

python3 edc_data_export.py \
	    --provider gcp \ 
	    --cred_file $SERVICE_ACCOUNT_JSON \
	    --query "select * from edc_prod_analytics_customer_$ORG_ID.$TABLE_NAME" \ 
	    --filename output.csv

The output file will be downloaded locally with the given name provided through --filename argument.

Sample Queries for GCP

Some sample queries that can be referred.

• select * from edcprod_analytics_customer$ORG_ID.user_card_performance_reporting_i_v where day=’2020-04-01’
• select * from edcprod_analytics_customer$ORG_ID.user_card_performance_reporting_i_v where day between ‘2020-04-01’ and ‘2020-04-04’
• select * from edcprod_analytics_customer$ORG_ID.user_card_performance_reporting_i_v where day between ‘2020-04-01’ and ‘2020-04-04’ and user_email=’[email protected]’
• select * from edcprod_analytics_customer$ORG_ID.user_card_performance_reporting_i_v where day between ‘2020-04-01’ and ‘2020-04-04’ and user_first_name like ’admin%
• select * from edcprod_analytics_customer$ORG_ID.user_card_performance_reporting_i_v where day between ‘2020-04-01’ and ‘2020-04-04’ and card_tile like ‘admin%’
• select * from edcprod_analytics_customer$ORG_ID.user_assignments_performance_i_v where day between ‘2020-04-01’ and ‘2020-04-04’ and assignment_state=’completed

Best Practices

• To improve the performance, try breaking the queries into multiple chunks using date filters.
• The utility supports --help parameter to understand what parameters are used with examples.

Warning and Errors

• The parameters should match what was given by the Edcast customer support team and are case sensitive, any mismatch would result in error.
• Utility aborts with error if the query attempts to download more than 1GB of data.

Utility Limitations

• Only csv format is supported for the output file.
• Any query which is scanning more than 1GB of data will be aborted - the users have to break the query into multiple chunks.

EdData Platform Events

EdCast data analytics platform process and stores below events in the EdData data lake.

EdData Data Sets

• user_card_performance_reporting_i_v
• group_performance_reporting_i_v
• channel_performance_reporting_i_v
• group_assignments_performance_i_v
• user_assignments_performance_i_v
• searches_fact_i_v

Users and Content

(user_card_performance_reporting_i_v)

User and content activity data based on user’s interactions on platform cards tracked as events (all events starting with card_) and each event capturing the following attributes.

Groups

(group_performance_reporting_i_v)

Group activities data based on group events (starting with group_) triggered by users, cards, or system.

Assignments (Users)

(user_assignments_performance_i_v)

Assignment data based on card assignment events related to individual assignments triggered by users or system (external LMS)

Assignments (Groups)

(group_assignments_performance_i_v)

Assignment data based on card assignment events related to group assignments triggered by users or system (external LMS)

Channels

(channel_performance_reporting_i_v)

Channels activities data based on channel events (starting with channel_) triggered by users, cards, or system

Searches

(searches_fact_i_v)

Search activities