Twitter can be a great channel for you to speak with your community, in fact I’ve blogged about using Twitter before. For the basics on a company Twitter account see here or if you want to use TweetDeck to manage your Twitter accounts you can check out my blog here.
One thing that Twitter has going for it is hashtags (#). They are a great way to link subjects in Twitter. One subject with many different tweets can be linked and discovered because they all share the same hashtag.
Using a hashtag is a great way to track your own content on Twitter.
If there’s one particular to your business or brand, you could even use it to track conversations across many users. We’ve toyed with #optimalbi and #optimalorange.
If you haven’t guessed by now I’m not a data analyst like Shaun McGirr, who wrote this, I’m a community manager. But I can’t go bugging the professionals every time I want some data looked at, so I decided to have a go myself at finding the data required to decide whether to use #optimalbi or #optimalorange. This is how I did it.
The first thing to do is to define what your objective is. This will have two parts; outcome and output.
First, the outcome objective is what change you want to effect. For this project I wanted to write better tweets for OptimalBI.
Secondly, the output is what you will produce at the end of the test. In this case I will produce data on whether the hashtag #optimalbi or #optimalorange gets a better response from our Twitter audience.
Define the test
In order to test which hashtag was more successful I decided to spend a week sending every tweet twice, one day apart, with #optimalbi on one day and #optimalorange on day two. The Tweets would have exactly the same content and be sent at the same time of day.
In order to capture ‘more successful’ I decided to measure two dimensions; reach and engagement.
Reach would be measured by the total number of impressions the tweets with each hashtag would get over the period of the test.
Engagement would be measured by the total number of engagements (replies, favourites, retweets) Tweets with each hashtag receive.
Run the test
So every day I would send a tweet as usual with #optimalbi, then copy the tweet and use TweetDeck to schedule the same tweet with the #optimalorange hashtag for the next day at the same time. In the end I could only do this from Monday to Thursday because sending Tweets on Friday with #optimalbi would have meant sending Tweets on Saturday with #optimalorange, this is something I’ll come back to later.
Extract the data
One of the best things about Twitter is Twitter Analytics. Twitter Analytics has the usual dashboards you would expect with the analysis for any social media platform, such as LinkedIn, but Twitter has one very important distinction.
This wonderful button allows you to export your Twitter data as a CSV file.
So, I selected the appropriate date range and hit export.
Prepare the data
When you download your Twitter data you get the whole lot in one big CSV. CSV files have their uses but as flat files they lack a lot of useful features. So, the first thing I did was convert this CSV file into an Excel file.
That file is big because Twitter has the ability to measure many, many things therefore has information which wasn’t really useful to me in this case such as permanent ID and whether a tweet was promoted. The second thing I did was remove unnecessary data from the spread sheet.
Then all I wanted was the tweets with #optimalbi and #optimalorange so I filtered on those and created two new pages in the Excel workbook for each of them.
The OptimalBI twitter account is a working Twitter account so seeing that I had on occasion tweeted with one hashtag but not the other is actually understandable. So I matched the tweets on both spreadsheets so they had the same tweets.
This left me with three spreadsheets;
- The complete set of all the Twitter data from my test period
- One set of tweets with #optimalbi
- One set of the same tweet with #optimalorange
With this my two sets of Twitter data now prepared it was easy to total the number of impressions and engagements;
#optimalbi – 849 impressions and 21 engagements
#optimalorange – 827 impressions and 14 engagements
So, by the measures of this test #optimalbi is the more successful hashtag. I checked with a data analyst though, to see whether this difference is statistically significant. They said the world is full of noise, and Twitter especially so, making 849 and 829 not so different after all.
I have met my outcome goal of providing data from which to make a decision of which hashtag to use.
What’s wrong with this picture?
This test achieved it’s objective, but nothing is perfect.
Here are some of the things which could have improved this test:
- One principle of testing is that you only test one thing. In this case sending tweets out 24 hours apart meant they were going out on different days. As I mentioned earlier no tweets for the test went out on Friday because the day after is Saturday. It is known that different days get different responses.
- The sample size for this test was small, in the time period covered, number of tweets, number of impressions and number of engagements. Partly, that is just how it is for a company of our size. If we were to do a test like this for someone famous then we would have a lot more data. Partly, it’s because I needed a decision quickly. I could run the same test again with #optimalorange on day one and #optimalbi on day two, I could run the test over a month with week about for which hashtag goes first. There are infinite variations I could run, given the time.
- The tweet sent first always had the hashtag #optimalbi, #optimalorange was always sent second. While they were sent 24 hours apart it is very possible that if two almost identical tweets were sent the first would be more successful than the second.
Results vs Conclusions
So, having conducted this trial I have a result. By the metrics that I choose #optimalbi received more impressions and engagement than #optimalorange by 22 impressions and 7 engagements.
What I don’t have is the substantive conclusion that using #optimalbi is better than using #optimalorange because the numbers are too similar.
I have satisfied my second goal of having some data, but not my first goal of finding a way to write better tweets.
I was a bit down about this so I talked to our in house expert Shaun McGirr. He agreed that I could run a better test with a larger sample over a longer time. I could switch which tweet was sent first to so that one tweet was not constantly being sent second. But he also explained that in all likelihood it wouldn’t make a difference. The factors that go into writing a successful tweet are so many and varied that changing a hashtag isn’t likely to have a material effect on how successful a tweet is no matter how you define success or how rigorous your testing method is.
Shaun went on to explain that this is usually the case when you test a hypothesis. The results don’t support a substantive conclusion. The best thing you can say is that you have eliminated something that doesn’t have a significant effect, in this case which hashtag is used.
For me this is actually a positive, #optimalbi has less characters than #optimalorange, by four. While four isn’t a large number in a tweet with a maximum of 160 characters another 4 can be very useful.
My conclusion is that I can use #optimalbi, because it is shorter, without worrying that #optimalorange is far better received.
I’ll be running further trials related to my works here as Community Manager to make our content and communication with you as good as it can be. I’ll probably be asking Shaun to help out in future.
Success is preparation meets opportunity – Jack