This is a requested blog post by the module coordinators, it will cover the tasks for week 8:

  • The Recommender System
  • iOS Twitter client App
  • Database selection
  • User-Evaluation

The Recommender System

Filtering of tweets – Refining the filtering and weighing of tweets was the next step. The Minimum Viable Product (MVP) used the ‘Bag of words’ approach where ‘Bag of words’ is a set of words that are extracted from the user timeline (user’s own tweets). To filter tweets each tweet from the home timeline is matched with the ‘Bag of words’ and the tweet with most matches with the words in the ‘Bag of words’ gets the higher weight. To refine this now the Recommender system creates Term Frequency (TF) documents which contains the same words as the ‘Bag of words’ but also has the number of occurrences of that word. So the filtering of the home timeline is with the tweets that has words present in the TF document as well as the occurrences of those words. The tweet with more matches and more frequency gets a higher weight.

iOS Twitter client App

Collecting user’s feedback and their useful interactions: an app to collect user’s interaction to help the Recommender system to further refine the filtering and recommendation. Those user interactions are:

  • Swipe the tweet left and right to indicate a like and dislike.

  • ‘Like’ swipe further shows up an icon on the tweet when tapped shows a list if the user wants to see more tweets from the

     i. Author

 ii. Content of the tweet

  • ‘Dislike’ swipe further shows up an icon on the tweet when tapped shows a list if the user wants to reduce tweets from the

    i.Author

    ii. Irrelevant

    iii. Too old

  • Highlighting the URL’s in the tweets content, so that the user can tap on the links to see more information which indicates they are interested in the tweet.

All these interactions are recorded and will be sent to the Recommender System.

Database Selection

Selecting an appropriate database was an important step after the MVP had been completed. We require to store tweets due to the Twitter API rate limits and also to record the user’s interactions with the Mobile client to help the Recommender System with personalising the tweets.

The tweets from the Twitter API have a complicated JSON structure. Instead of using a SQL Database which requires a specified stucture, we have chosen to go with the Document/NoSQL database to store the JSON of tweets. For the User-Interaction data we could use the SQL database as the structure of the table is almost fixed.

We have reviewed some databases but as of now we have RethinkDB or PostgreSQL or a combination of both considered. We are currently in the testing phase of both. Our next blog will cover in greater details our selection process and the final descision.

User-Evaluation

We wanted to start the user evaluation as we have only 6 weeks to complete the project and also it gives us the record of how the Recommender System is improving the quality of personalised tweets over time. To perform user evaluations we will create a customised iOS app. It will be a seperate project but in the same Xcode workspace so we can re-use our Swift code.

The User Evaluation App throws a shuffled set of tweets from the filtered home timeline from the Recommender System and Twitter’s home timeline, each tweet being presented with a ‘like’ , ‘dislike’ and ‘neither’ actions to it. Finally the result is collected and stored in the database. This way we hope to be able to see objective results if users tend to like tweets more from our version of the feed or not.