The Epic conclusions: XBOT vs. Archmage in the era of rebellion
Over the past two seasons, I've been running an experiment pitting the two prominent bot services for Wild Ranked in Splinterlands: XBOT in ArchMage. The goal is to see which is better bang for my buck ("my" is important, as the results below will be somewhat specific to my account. not to say this has nothing to say for the general splinterlands player, but the results should be taken with a grain of salt).
Interested readers can see the original layout of the experiment and the halfway report after I finished the first season with XBOT.
The experiment
For a quick recap, the experiment consisted of letting the bot pilot my account for a full season with the following constraints:
- No changes to my deck
- Keeping my staked SPS near-constant
- Keeping track of the length of the season
The outcomes of interest
As a quick recap, the outcomes of interest were:
- Primary: Net rewards per day
- Max rating
- Number of battles using Rebellion cards
- Number of wins using Rebellion cards
Calculating reward value
- SPS is valued at $0.0233 (its value on 1/30)
- Champion chests receive value from SPS and potions. Everything else is "valueless"
- Use the EV for chests rather than actual rewards, so all that really matters is the number of chests (actual rewards are too high variance)
- Diamond chests have an EV of $0.18
- Champion chests have an EV of $0.48
The results
Below shows the results for the two seasons.
The above images are plots of my rating over time in the two seasons. The first season, run by XBOT, you can see it sat in Diamond III for a few days before jumping up to Champ. This led to more diamond chests during the season. The second image, showing the plot for Archmage, shows that it jumped into Champion quicker, although it did have a drop shortly thereafter back into diamond. This led to more champion chests from Archmage over the course of the season.
These trajectories led to the following rewards:
| Metric | XBOT | ArchMage |
|---|---|---|
| Length of season | 15 days | 16 days |
| Daily diamond chest | 30 | 18 |
| Daily champion chests | 45 | 60 |
| Season champion chests | 58 | 61 |
| Battles with Rebellion | 53 | 151 |
| Wins with Rebellion | 36 | 90 |
| Win % with Rebellion | 68% | 60% |
| Total rewards | $64.70 | $71.3 |
| Total fees | $0.76 | $2.42 |
| Net rewards | $63.94 | $68.95 |
| Net rewards per day | $4.26 | $4.31 |
Overall, the final net rewards per day were remarkably close: within 1% (which is about what I guessed mid-way after seeing XBOT in action for the first time). Archmage was a higher earner (mainly from getting up to Champion II slightly faster than XBOT so getting more chests in that range), but this was offset by the tripled fee.
My advice to each bot
Both bots have their strengths and weaknesses that I saw in watching my account over the past month.
For XBOT: Incentivize the bot to use more Rebellion. I know the set is perceived as a bit weaker, and I imagine making use of some of the positioning is harder on the bot, but I've had great success when I've use Rebellion manually. And Archmage got better at using Rebellion even in the two week time frame I measured it. With how much rebellion people are holding for conflicts, using it more will be a boon to the service.
For Archmage: Figure out how to avoid failing to submit a team. I can't emphasize enough how demoralizing it is to see a battle with no team submitted. The bot needs to have some sort of fallback fail-safe team to submit. Losing is whatever. Not submitting a team is really frustrating, and the psychological aspect shouldn't be overlooked.
What bot am I using going forward?
Asking the hard questions I see. With my primary outcome basically a wash, it boils down to preference. Because the primary outcome was a wash, and the other outcomes revealed pluses and minuses, I had to find a differentiating factor. One thing I also checked was what was the value solely from SPS. In this metric, XBOT came out ahead, solely because of the lower fee (the amount of SPS earned were similar, mainly driven by the bulk of chest value coming from potions). I don't like paying 3x the fee for a similar service. And selfishly, if I ever hit a jackpot chest with 1000+ SPS, I'd like to keep and extra 100+ of it for myself.
For these reasons, and because I need to choose one, and because I want to get a little more experience with XBOT, and because I like XBOT's interface a little more, I will be using XBOT in the near future. I might revisit this if ArchMage adds new features, or, crucially, eliminates forfeited matches (I cannot stress how annoying those are).
Do I have any advice to other players from this?
Wow, another hard hitting question from my headlines. The not really helpful advice is: do what I did. Run the experiment, keep track of the rewards, and decide for yourself.
But what if you don't want to do the work and want me to tell you what to do? That's harder. Assuming you don't own either token to begin with, you need to factor in the cost. Currently, XBOT regular and Archmage Alpha are similar in price (~$50). I do think XBOT has a bit of an edge if you're gunning for leaderboard slots (but if that describes you, you probably aren't reading this article, as you don't need my advice). I think the two are pretty comparable for most accounts. I don't see either service closing shop anytime soon. So I think, unfortunately, it boils down to preference.
Referral
Interested in getting in on Splinterlands? Interested in using a bot? Want some free cards? Use my referral code below, and I'll get you started in the game with some cards.
Hey @badrag, thank you for your feedback!
I just wanted to mention a couple of things. The season you tested with Archmage only included about 1/3 of the time before we made the changes we've been working on. So, a significant part of your test results may not accurately reflect our current optimized code.
The failed submits were due to a lingering issue from a database problem that we were actively resolving while also implementing new code. We are closely monitoring this issue, and it seems to be mostly fixed. Of course, there might still be some new bugs, but we're addressing them promptly.
It might be worth giving it another go for another season or two, especially because our results should keep improving based on how we've built it.
Let me know if you have any more questions or concerns, but either way, thanks for the fair look.
All in all, it looks like Archmage still helped you earn more, even when accounting for fees. With the failed submits being fixed... that should be even better ;)
Question though - are you factoring the rewards with EV or actual rewards?(Edit: Reread and it was answered already)I just reread - not sure how I missed that you DID mention the EV portion :) That's what I get for reading quickly I guess. @badrag
Thanks for the reply!
Yeah, the updated code was something I had hoped wouldn't be an issue. My logic for going ahead was that the issue happened with the Rebellion release, which was like 6 weeks ago. So I didn't feel too bad penalizing a bot that wasn't able to fix problems in that time frame. And from the messages in the discord, it seemed like the biggest changes were to the lower leagues. AM actually performed pretty well for me early on; it got up to Champ faster than XBOT did.
I do think I'll rerun the test in a season or two and see how much the update did.
And yeah, the final numbers were interesting. If the services were the same fee, I'd go with AM hands down. But 3x the fee, kept it more similar. And importantly, that fee is SPS, the only truly fungible reward we're getting right now. If I could pay the 15% fee in potions I would also choose AM hands down.
@badrag Thanks again for sharing your thoughts on comparing Archmage and XBot. It's really helpful to look at them side by side.
The more I've thought about this, the more I think there's a missing piece in your assessment though.
Let's dive a bit deeper into this, especially about how choosing one over the other could affect our future earnings.
You pointed out that Archmage has higher rewards but also higher fees, and XBot has lower rewards and fees. That means your total "sellable" earnings with your current account are about break even. But there's more to think about, like how our choices today can help us earn more in the future.
Here's the main point: Getting Cards is crucial for long-term earnings. Think of Cards not just as rewards, but as tools that continually boost your earning power. It's like constantly upgrading a machine – every new Card you add is like giving your earnings a power boost.
Many key cards are ONLY earned and improved through winning battles. You get Rewards Cards by earning Loot Chests and Gladius Cards by earning Merits for Gladius Cases. This is where automation services shine – they help ensure your account is not only earning sellable rewards but also collecting those special Cards that enhance your future earning ability. Every new Card is an upgrade, making your account stronger and more capable of earning.
Now, let's look at how Archmage and XBot help us do this. Here's a simple breakdown using your numbers, showing how each service helps us make our Card collection better and boost our future earnings:
I created a spreadsheet that runs the math for distribution of all rewards based on chest type to come up with these numbers and then divided seasons totals by days in the season to try and make the comparison more fair as well - All numbers are based on expected averages on a per chest basis to compare on the whole what you'd expect
Key Takeaway:
If net earnings from sellable rewards is almost the same with both services, these numbers show that Archmage helps improve your collection about 18% faster each day. This means that even if you spend the same amount each season, not using Archmage might mean missing out on a chance to make your account better for free, at a rate of 18% every day.
In short, choosing Archmage, based on your numbers, could mean you'll grow your earnings faster over time.
Thanks for the review. I'm sure this took a lot of work!
It's nice to see an independent side-by-side comparison but there's a huge amount of randomness involved and the sample size is tiny. It's 1 account for 1 season only.
Looking at your account's card value, it's almost 50% Rebellion cards, which is very uncommon in my experience. Most of our users have few or no Rebellion cards due to them being new & less cost-effective. That makes you account's collection unique and not representative for the average player.
I think the best advice to give undecided players is to try out both services. The good news is that XBOT has a free 7 day trial, so you have nothing to lose by giving it a shot.
Thanks for the reply! Yeah, this was a fun project and took a decent bit of work.
A few responses:
The sample is much bigger than that. The level of observation isn't really my account, it's each battle (using my account's deck). So it's more like 350+ samples per bot. Which is a really nice sample size for this kind of work.
Yeah, I have more value in Rebellion than most accounts, but not more cards. I only have about 30 Rebellion cards. The value is there because they are GF cards for conflicts. And a large part of doing this experiment now was to see how quickly the bots adapted to new cards being available. I did slightly handicap XBOT by having it first and giving it less time to learn. Restarting XBOT yesterday I'm already seeing more Rebellion usage.
Yup, that's exactly the advice I gave - do this for your own account (because you're right that even though the sample size is large, it's only large for my account). So hopefully people get that takeaway.
It looks like the number of Champion chests earned with XBOT changed from 18 to 45 in your report.
I've ran the numbers using average chest value from Splex, and accounting for the difference in Season length:
Based on this XBOT comes out slightly ahead in earnings.
Yeah, I fat-fingered the number and reused 18 instead of doing the 45 correct the first time. I've spoken with AM about it, and it changes the difference they showed in card earnings from ~18% to ~3%.
I'll have to check how Splex calculates the value and where those numbers come from ($6.2 in fees to AM feels higher than I'd expect). I was happy with how I calculated the numbers which was what I had stated from the start. I didn't want to change any of the calculations after specifying the experiment for exactly this reason: if I change it and the results change, it would look like I was favoring one bot over the other.
Regardless, I think the point stands that the two services are pretty much neck and neck.
And I find it a bit amusing both services are trying to spin the results here as a big win for them. The answer is, and I know things that aren't black-and-white are hard, but in this case it's the truth: the two services are really comparable and people should decide for themselves.
If there was like a 10%+ difference in net earnings, I'd happily endorse one service over the other. But there just isn't. And while 350+ is a really good sample size, 1-2% is certainly within the margin of error. I'm not worried about maybe missing out on $0.05/day if I choose the wrong service. At that point it comes down to other, smaller things like I mentioned at the bottom. In this case, I really like paying fewer fees, as I keep more SPS, which is the most important type of reward to me right now. So XBOT wins that regardless of whether it's netting 1% more or less than AM might be.
Haha yeah, both sides are trying to interpret the data in their favour.
What I will say though, and I promise I'll stop after this, battle performance is very dependent on the collection. Some collections will do better with XBOT, some better with AM. In this case AM was slightly stronger, but that's certainly not the case on average. Also, you didn't use XBOT's boost mode, which is designed to improve battle performance.
In cases where AM performs better, you're paying more in fees, which negates the difference. (Like in your example)
In cases where XBOT performs better, the difference in net earnings is huge. (Because of more earnings AND less fees)
Also if you care about extra features, like automatic SB combining, card rentals, SPS rentals, in-depth analysis, liquid rewards, XBOT beats AM by a landslide.
I love the xtra features of xbot! That's part of what made me go that route at the end of the day. I like the interface better. I like the charts and analysis. I like the daily check in rewards which I can use to rent sps.
And yeah, the "bots perform differently with different accounts" was basically the impetus for doing this at all, since all I can publicly obviously see is how they perform for the leaderboard. If my account ever gets a solid enough deck to fight for leaderboard spots, I'll have to rerun the experiment, where I imagine XBOT will come up nicely ahead.
As a final thought, I am a bit pleased with myself that this experiment and the write-up yielded well-thought-out responses from the folks on both bots. I'm pretty sure like 3 people read my articles on Hive, so it's nice to have some legit engagement. So thanks :)
Thanks for sharing! - @mango-juice
nice analysis
Congratulations @badrag! You have completed the following achievement on the Hive blockchain And have been rewarded with New badge(s)
Your next target is to reach 4500 upvotes.
You can view your badges on your board and compare yourself to others in the Ranking
If you no longer want to receive notifications, reply to this comment with the word
STOPCheck out our last posts: