Overview
This engine template predicts the probability of an user will convert (conversion event by user) in the current session.
Usage
Event Data Requirements
By default, the template requires the following events to be collected:
- 'page view' events with session ID
- the first page view event can optionally provide the browser and referrer ID
- user 'buy' event with session ID
The landing page ID, referrer ID, browser information and user's buy event will be used to train the model.
Input Query
- Landing page ID
- Referrer ID
- Browser
Output PredictedResult
- score
1. Install and Run PredictionIO
First you need to install PredictionIO 0.14.0 (if you haven't done it).
Let's say you have installed PredictionIO at /home/yourname/PredictionIO/
. For convenience, add PredictionIO's binary command path to your PATH
, i.e. /home/yourname/PredictionIO/bin
:
1 | $ PATH=$PATH:/home/yourname/PredictionIO/bin; export PATH |
Once you have completed the installation process, please make sure all the components (PredictionIO Event Server, Elasticsearch, and HBase) are up and running.
If you are using PostgreSQL or MySQL, run the following to start PredictionIO Event Server:
1 | $ pio eventserver &
|
If instead you are running HBase and Elasticsearch, run the following to start all PredictionIO Event Server, HBase, and Elasticsearch:
1 | $ pio-start-all
|
You can check the status by running:
1 | $ pio status
|
If everything is OK, you should see the following outputs:
1 2 3 4 | ... (sleeping 5 seconds for all messages to show up...) Your system is all ready to go. |
2. Create a new Engine from an Engine Template
Now let's create a new engine called MyLeadScoring by downloading the Lead Scoring Engine Template. Go to a directory where you want to put your engine and run the following:
1 2 | $ git clone https://github.com/PredictionIO/template-scala-parallel-leadscoring.git MyLeadScoring $ cd MyLeadScoring |
A new directory MyLeadScoring is created, where you can find the downloaded engine template.
3. Generate an App ID and Access Key
You will need to create a new App in PredictionIO to store all the data of your app. The data collected will be used for machine learning modeling.
Let's assume you want to use this engine in an application named "MyApp1". Run the following to create a new app "MyApp1":
1 | $ pio app new MyApp1
|
You should find the following in the console output:
1 2 3 4 5 6 | ... [INFO] [App$] Initialized Event Store for this app ID: 1. [INFO] [App$] Created new app: [INFO] [App$] Name: MyApp1 [INFO] [App$] ID: 1 [INFO] [App$] Access Key: 3mZWDzci2D5YsqAnqNnXH9SB6Rg3dsTBs8iHkK6X2i54IQsIZI1eEeQQyMfs7b3F |
Note that App ID, **Access Key* are created for this App "MyApp1". You will need the Access Key when you collect data with EventServer for this App.
You can list all of the apps created its corresponding ID and Access Key by running the following command:
1 | $ pio app list
|
You should see a list of apps created. For example:
1 2 3 4 | [INFO] [App$] Name | ID | Access Key | Allowed Event(s) [INFO] [App$] MyApp1 | 1 | 3mZWDzci2D5YsqAnqNnXH9SB6Rg3dsTBs8iHkK6X2i54IQsIZI1eEeQQyMfs7b3F | (all) [INFO] [App$] MyApp2 | 2 | io5lz6Eg4m3Xe4JZTBFE13GMAf1dhFl6ZteuJfrO84XpdOz9wRCrDU44EUaYuXq5 | (all) [INFO] [App$] Finished listing 2 app(s). |
4. Collecting Data
Next, let's collect training data for this Engine. By default, Lead Scoring Engine Template supports the following entities: user, page, and item. An user views a page, and buys an item.
Note that a "sessionId" property is required to indicate these events happen in the same session. In the first visit of a user, you should specify the optional "referrral ID" and "browser" information. These are used to determine where the user comes from and the browser information.
In summary, this template requires user-view-page event and user-buy-item events with the session ID, referrer ID and browser properties.
You can send these events to PredictionIO Event Server in real-time easily by making a HTTP request or through the provided SDK. Please see App Integration Overview for more details how to integrate your app with SDK.
Let's try sending events to EventServer with the following curl
commands (The corresponding SDK code is showed in other tabs).
Replace <ACCCESS_KEY>
by the Access Key generated in above steps. Note that localhost:7070
is the default URL of the Event Server.
For convenience, set your access key to the shell variable, run:
$ ACCESS_KEY=<ACCESS_KEY>
For example, when an user with ID u0 views a URL page "example.com/page0" on time 2014-11-02T09:39:45.618-08:00
, with session ID "akdj230fj8ass" (current time will be used if eventTime is not specified) you can send the event to Event Server. Run the following curl
command:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | $ curl -i -X POST http://localhost:7070/events.json?accessKey=$ACCESS_KEY \ -H "Content-Type: application/json" \ -d '{ "event" : "view", "entityType" : "user", "entityId" : "u0", "targetEntityType" : "page", "targetEntityId" : "example.com/page0", "properties" : { "sessionId" : "akdj230fj8ass", "referrerId" : "referrer0.com", "browser" : "Firefox" } "eventTime" : "2014-11-02T09:39:45.618-08:00" }' |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | import predictionio client = predictionio.EventClient( access_key=<ACCESS KEY>, url=<URL OF EVENTSERVER>, threads=5, qsize=500 ) # A user views a page client.create_event( event="view", entity_type="user", entity_id=<USER ID>, target_entity_type="page", target_entity_id=<PAGE ID>, properties = { "sessionId": <SESSION ID>, # required "referrerId": <REFERRER ID>, # optinal. but should specify this if you have this information when user views the landing page "browser": <BROWSER> # optinal. but should specify this if you have this information when user views the landing page } ) |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | <?php require_once("vendor/autoload.php"); use predictionio\EventClient; $client = new EventClient(<ACCESS KEY>, <URL OF EVENTSERVER>); // A user views a page $client->createEvent(array( 'event' => 'view', 'entityType' => 'user', 'entityId' => <USER ID>, 'targetEntityType' => 'page', 'targetEntityId' => <PAGE ID>, 'properties' => array( 'sessionId' => <SESSION ID>, 'referrerId' => <REFERRER ID>, 'browser' => <BROWSER> ) )); ?> |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | # Create a client object. client = PredictionIO::EventClient.new(<ACCESS KEY>, <URL OF EVENTSERVER>) # A user views a page. client.create_event( 'view', 'user', <USER ID>, { 'targetEntityType' => 'page', 'targetEntityId' => <PAGE ID>, 'properties' => { 'sessionId' => <SESSION ID>, 'referrerId' => <REFERRER ID>, 'browser' => <BROWSER> } } ) |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | import org.apache.predictionio.Event; import org.apache.predictionio.EventClient; import com.google.common.collect.ImmutableList; EventClient client = new EventClient(<ACCESS KEY>, <URL OF EVENTSERVER>); // A user views a page Event viewEvent = new Event() .event("view") .entityType("user") .entityId(<USER_ID>) .targetEntityType("page") .targetEntityId(<PAGE_ID>); .property("sessionId", "<SESSION ID>") .property("referrerId", "<REFERRER ID>") .property("browser", "<BROWSER>"); client.createEvent(viewEvent); |
In the same browsing session "akdj230fj8ass", the user with ID u0 buys an item i0 on time 2014-11-02T09:42:00.123-08:00
(current time will be used if eventTime is not specified), you can send the following buy event. Run the following curl
command:
1 2 3 4 5 6 7 8 9 10 11 12 13 | $ curl -i -X POST http://localhost:7070/events.json?accessKey=$ACCESS_KEY \ -H "Content-Type: application/json" \ -d '{ "event" : "buy", "entityType" : "user", "entityId" : "u0", "targetEntityType" : "item", "targetEntityId" : "i0", "properties" : { "sessionId" : "akdj230fj8ass" } "eventTime" : "2014-11-02T09:42:00.123-08:00" }' |
1 2 3 4 5 6 7 8 9 10 11 12 | # A user buys an item client.create_event( event="buy", entity_type="user", entity_id=<USER ID>, target_entity_type="item", target_entity_id=<ITEM ID>, properties = { "sessionId": <SESSION ID>, # required } ) |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | <?php // A user buys an item $client->createEvent(array( 'event' => 'buy', 'entityType' => 'user', 'entityId' => <USER ID>, 'targetEntityType' => 'item', 'targetEntityId' => <ITEM ID>, 'properties' => array( 'sessionId' => <SESSION ID> ) )); ?> |
1 2 3 4 5 6 7 8 9 10 11 12 | # A user buys an item. client.create_event( 'buy', 'user', <USER ID>, { 'targetEntityType' => 'item', 'targetEntityId' => <ITEM ID>, 'properties' => { 'sessionId' => <SESSION ID> } } ) |
1 2 3 4 5 6 7 8 9 | // A user buys an item Event buyEvent = new Event() .event("buy") .entityType("user") .entityId(<USER_ID>) .targetEntityType("item") .targetEntityId(<ITEM_ID>) .property("sessionId", "<SESSION ID>"); client.createEvent(buyEvent); |
Query Event Server
Now let's query the EventServer and see if these events are imported successfully.
Go to following URL with your browser:
http://localhost:7070/events.json?accessKey=<YOUR_ACCESS_KEY>
or run the following command in terminal:
1 | $ curl -i -X GET "http://localhost:7070/events.json?accessKey=$ACCESS_KEY" |
It should return the imported events in JSON format. You can refer to Event Server Debugging Recipes for more different ways to query Event Server.
Import More Sample Data
This engine requires more data in order to train a useful model. Instead of sending more events one by one in real time, for quickstart demonstration purpose, we are going to use a script to import more events in batch.
A Python import script import_eventserver.py
is provided to import sample data. The sample data includes 50 sessions of events. In each session, a randomly selected user (with user ID "u1" to "u10") lands on a page (randomly selected from example.com/page1 to example.com/page20) with referrerId (randomly selected from referrer1.com to referrer10.com) and browser information. The user may view more pages, and may or may not buy an item (with item ID "i1" to "i50").
First, you will need to install Python SDK in order to run the sample data import script. To install Python SDK, run:
1 | $ pip install predictionio
|
or
1 | $ easy_install predictionio
|
Make sure you are under the MyLeadScoring
directory. Execute the following to import the data:
1 2 | $ cd MyLeadScoring $ python data/import_eventserver.py --access_key $ACCESS_KEY |
You should see the following output:
1 2 3 4 5 6 7 8 | ... User u8 buys item i13 session c347980abdf711e4b135b8e8560679ba User u5 lands on page example.com/page11 referrer referrer4.com browser Firefox User u5 views page example.com/page8 User u5 views page example.com/page17 User u5 buys item i5 166 events are imported. |
You can query the event server again as described previously to check the imported events.
5. Deploy the Engine as a Service
Now you can build, train, and deploy the engine. First, make sure you are under the MyLeadScoring
directory.
1 | $ cd MyLeadScoring |
Engine.json
Under the directory, you should find an engine.json
file; this is where you specify parameters for the engine.
1 2 3 4 5 6 7 | ... "datasource": { "params" : { "appName": "MyApp1" } }, ... |
Building
Start with building your MyLeadScoring engine. Run the following command:
1 | $ pio build --verbose
|
This command should take few minutes for the first time; all subsequent builds should be less than a minute. You can also run it without --verbose
if you don't want to see all the log messages.
Upon successful build, you should see a console message similar to the following.
1 | [INFO] [Console$] Your engine is ready for training. |
Training the Predictive Model
To train your engine, run the following command:
1 | $ pio train
|
When your engine is trained successfully, you should see a console message similar to the following.
1 | [INFO] [CoreWorkflow$] Training completed successfully. |
Deploying the Engine
Now your engine is ready to deploy. Run:
1 | $ pio deploy
|
When the engine is deployed successfully and running, you should see a console message similar to the following:
1 2 | [INFO] [HttpListener] Bound to /0.0.0.0:8000 [INFO] [MasterActor] Bind successful. Ready to serve. |
Do not kill the deployed engine process.
By default, the deployed engine binds to http://localhost:8000. You can visit that page in your web browser to check its status.
6. Use the Engine
Now, You can retrieve the results. When a user lands on your page "example.com/page9", with referrer "referrer10.com" and browser "Firefox", you can get the predicted lead score by sending this JSON '{ "landingPageId" : "example.com/page9", "referrerId" : "referrer10.com", "browser": "Firefox" }' to the deployed engine. The engine will return a JSON with the score.
Simply send a query by making a HTTP request or through the EngineClient
of an SDK.
With the deployed engine running, open another terminal and run the following curl
command or use SDK to send the query:
1 2 3 4 5 6 7 | $ curl -H "Content-Type: application/json" \ -d '{ "landingPageId" : "example.com/page9", "referrerId" : "referrer10.com", "browser": "Firefox" }' \ http://localhost:8000/queries.json |
1 2 3 4 5 6 7 | import predictionio engine_client = predictionio.EngineClient(url="http://localhost:8000") print engine_client.send_query({ "landingPageId" : "example.com/page9", "referrerId" : "referrer10.com", "browser": "Firefox" }) |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | <?php require_once("vendor/autoload.php"); use predictionio\EngineClient; $client = new EngineClient('http://localhost:8000'); $response = $client->sendQuery(array( 'landingPageId' => 'example.com/page9', 'referrerId' => 'referrer10.com', 'browser' => 'Firefox' )); print_r($response); ?> |
1 2 3 4 5 6 7 8 9 10 11 | # Create client object. client = PredictionIO::EngineClient.new('http://localhost:8000') # Query PredictionIO. response = client.send_query( 'landingPageId' => 'example.com/page9', 'referrerId' => 'referrer10.com', 'browser' => 'Firefox' ) puts response |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | import com.google.common.collect.ImmutableMap; import com.google.common.collect.ImmutableList; import com.google.gson.JsonObject; import org.apache.predictionio.EngineClient; // create client object EngineClient engineClient = new EngineClient("http://localhost:8000"); // query JsonObject response = engineClient.sendQuery(ImmutableMap.<String, Object>of( "landingPageId", "example.com/page9", "referrerId", "referrer10.com", "browser", "Firefox" )); |
The following is sample JSON response:
1 | {"score":0.7466666666666667} |
MyLeadScoring is now running.